I think you ask a quite reasonable question. One certainly could have a more condensed
description of what is on-topic on MathOverflow, and put this in a prominent place where people see it before asking their first question. In particular one could make more clear what is
the difference to Math.SE. As a draft of such description, one could take e.g. the following:
A mathematical question is on-topic and appreciated on MathOverflow if it is a question
which professional mathematicians may ask each other. If you are not a mathematician yourself,
you should rather ask your question on Math.SE instead -- unless a mathematician has
encouraged you to ask it here.
Apart from this, the main rules are the following:
Questions should be brief, clear, self-contained, well-formulated and allow
well-defined, short (at most about 1-2 pages) answers.
Don't expect anyone to spend time on answering a question which is vague, overly long,
full of language mistakes, or which looks like it has taken you only 5 minutes to ask.
Questions which are or look like homework or contest problems are off-topic,
i.e. not welcome on this site.
MathOverflow is not the right place to announce your results or to ask people to
check your proofs ("Is my proof of the Riemann hypothesis / Goldbach conjecture / ...
correct?" etc.); doing that anyway is likely to cause harsh reactions of the community.
Avoid asking questions which can easily be answered via a Google search.
In particular avoid duplicates, i.e. questions which have already been asked
before on this site.
If you know the answer to your question, then don't ask it here --
it would waste other people's time.
To avoid duplicating efforts, ask a question either on Math.SE or here,
but not on both sites. If you would like to have an exception from this rule
for a question which remained unanswered on Math.SE for a long time,
enquire on Meta.
Answering your question should not require reading particular books, papers etc..
This also means that questions about particular preprints are off-topic.
Questions which do not fulfil the above conditions usually get downvoted, closed and removed.
However note that as everywhere, it is accepted that respected members of the community
sometimes do not follow some of the rules. As a new -- possibly even pseudonymous --
user unknown to the community, don't make the mistake to copy such behavior!
For a longer and more detailed explanation of what is on-topic on MathOverflow, see here.