Late to the party, but two points:
There is a specific premise to the question: Part of MathOverflow's "brand", in contrast to, say, Math Stack Exchange, is that we're not about all mathematics questions, but specifically, "research-level questions" -- whatever that means. Personally, I think it would be productive to try to reformulate the adjective "research-level" in a better way. After all, the idea that mathematics is a "linearly-ordered" subject, progressing from grade-school concepts to high school concepts to undergraduate concepts to graduate concepts to research concepts, is a widespread misconception. I don't think I'm alone in wincing every time I tell someone I'm a mathematician and get the response "Oh, I never got past calculus". So this whole business is problematic, but I think that has been at least partially addressed in other responses.
The issue of being perceived as "unwelcoming" is not unique to Math Overflow -- it's a widespread phenomenon on the Stack Exchange network, particularly on the flagship site Stack Overflow, as documented in the company blog, meta, meta, news articles, and memes. And among "curated knowledge" sites, it's not unique to the Stack Exchange network -- for example wikipedia struggles with analogous issues. I can't say I've carefully considered the issue from this broader perspective, but I do think that any attempt to grapple with these issues would do well to take a look at other places where similar problems have been encountered and (hopefully) solutions have been formulated.
Of course, there are other broad contexts in which to situate the issue, such as the "welcomingness" of mathematics as a discipline, academia as a whole, and so forth. But I think the peculiarities of the dynamics of Stack Exchange and similar sites are particularly relevant, and underrepresented in the discussion here.
EDIT: To expand on (2), I'd like to meditate on a recent experience on a different SE site. A few months ago I asked this question on Physics SE, a site I have occasionally visited, but with which I'm not all that familiar. The question attracted a good answer, but was also attracting votes to close for being "opnion-based". I made some edits to remove any language I thought might be interpretable as "opinion-based", but to no avail -- the question was still closed as "opinion-based". With no further feedback, I thought I'd done all I could and gave up, figuring that Physics SE had their own inscrutable criteria for acceptable questions on their site, and that it was not worth trying to get to the bottom of things at the time.
A few days ago, I decided to give it another crack. My only lead was the close reason -- it was "opinion-based". So I scoured the question for anything which might whiff of an opinion, and excised it. I speculated that the given close reason was not the true close reason -- probably the last few votes to close, made after my edits, had simply chosen to give the same reason as the earlier votes for simplicity, even though the reasons had changed. I recorded these speculations in the comments, but I didn't attempt to rectify any of these hypothetical issues.
I submitted the question to be reopened. A helpful moderator came along and improved my tags. While still closed, the question attracted another good answer in the comments. Eventually it was reopened, before being immediately closed for a new reason -- it lacked focus. I was confused again, but this time the same moderator left a helpful comment explaining that my question needed to have a unique answer to be acceptable.
Aha! It was indeed a community difference. I'm pretty certain that my question as currently phrased would be the sort of focused, limited list question we'd embrace at MO (maybe in community wiki format, but honestly we probably wouldn't even bother with that), but over there any list question was a non-starter. This wasn't in their FAQs, but there was some old discussion on meta on the topic. The helpful moderator helpfully listened when I raised these points in the comments, and eventually helpfully reversed the second closure despite personally feeling that the "list" objection was strong. We'll see where it goes.
I'm drawing a few lessons from this:
To a new user, our criteria for deciding whether a question is acceptable are doubtless completely inscrutable. Even an experienced user on another SE site who has some reasonable idea of what "research mathematics" is will likely find it extremely frustrating trying to meet these opaque standards. For a user who doesn't have experience on another SE site, it will be unclear that there even are standards -- it will appear that their questions are being categorically rejected for no reason at all.
At the same time to our users, many of these criteria seem self-evident. We may have internal debate over what exactly "research mathematics" is, but we probably don't even realize that there are subtle conventions of phrasing and so forth which have never been made explicit that we pick up on and use to determine acceptability of questions.
If we don't even consciously know some of the reasons we find questions unacceptable, it's extremely difficult for these reasons to be communicated to new users.
Things like official close reasons are very blunt tools for communicating what's wrong with a question. Expanding in a comment is better, but there's still going to be a communication gap.