Right now, there are 4 ways to vote to close a question as off-topic.
"This question does not appear to be about research level mathematics within the scope defined in the help center."
"MathOverflow is for mathematicians to ask each other questions about their research. See Math.StackExchange to ask general questions in mathematics."
This belongs on another StackExchange site, say Math.StackExchange.com "Q&A for people studying math at any level and professionals in related fields"
Custom reason
I spend some time trying to figure out how to vote when any of the first $3$ reasons could be used, and when the question is actually closed, I often see that there is a split among the votes to close, sometimes $2$ chose one way, $2$ chose another, $1$ chose the third. I don't think this is helpful. Even if all of these are obviously correct, it suggests there is disagreement about the question. So, sometimes I just agree with earlier votes to close.
Could these reasons to close be clarified so there is less overlap, or combined as redundant, or combined as not worth distinguishing?
I presume the difference between $5$ people voting to close as "See Math.StackExchange to ask general questions" and "this belongs on Math.StackExchange" is that the latter migrates the question while the former doesn't. If there were no such distinction, they would be completely redundant. I guess the reason to vote for the second and not the third is that the question has some flaw to be fixed, or maybe we anticipate that the question would be closed on MSE and automatically bounced back. However, I don't think this is stated, so closing with the second option doesn't tell the OP whether/how to fix the question before reposting it.
See also Revamping the closure reasons.