What kind of questions should I not ask here?
There are several broad categories of questions that should not be asked on MathOverflow. If a question is closed, it's probably because it fell into one of these categories. If your question is closed, please don't take this to mean that you are not welcome on the site, or think that this will be held against you in the future. A lot of active users have had questions closed at one point or another.
MathOverflow is not for homework help. Try to make sure your question is of interest to at least one other mathematician. Though there are no hard and fast rules about who may post here, the intended audience is professional mathematicians, mathematics graduate students, and advanced undergraduates. If your question is closed as "off topic," it might be because it was too elementary. This doesn't mean that your question is "bad," just that MathOverflow isn't the right place for it. But there are lots of other math Q&A sites where your question might fit right in, and we have a list of such sites. If your question was closed as "too localized," it was probably an explicit homework problem (or looked like one). MathOverflow is not the right place for such questions; they send the message that you want somebody else to do your work for you, and you're not even willing to ask it in a general form.
MathOverflow is not a discussion forum. As a side-effect of being very good for to-the-point questions and answers, the Stack Exchange software is bad for discussions and designed to minimize them. There's a place for discussion about mathematics, but it isn't MathOverflow. Blogs and threaded discussion forums are a more appropriate place for discussions. For more information on this, see the relevant tea.mathoverflow.net threads (e.g. this one).
MathOverflow is not an encyclopedia. MO is a site for questions that have answers. MathOverflow visitors should know how to learn new things and do mathematics on their own, but we all get stuck sometimes, and this is where MO saves the day. When you're stuck, you can come to MathOverflow and say "I'm trying to do X. How can I do that? Does this work? Does anybody have a reference?" The idea being that for an expert, it should take very little effort to understand your confusion and set you on the right path. Or maybe a non-expert has come across the same sticking point and can explain how she resolved it. MathOverflow is not the appropriate place to ask somebody to write an expository article for you. If you want somebody to write an article about some subject, you should make a stub on Wikipedia, make a query block on nLab, or make a request on PlanetMath.
MathOverflow is not for questions about MathOverflow. Mathematicians don't come here to learn about the intricacies of this site; they come here to get answers to their questions. Let's try to help them out by not cluttering up the system with navelgazing meta-discussion. If you want to report a bug, suggest a feature, or discuss the dynamics of MathOverflow, visit our meta-discussion forum, meta.mathoverflow.net. If you're just really interested in how the underlying Stack Exchange software works, consider visiting meta.stackoverflow or meta.stackexchange, but please don't ask MathOverflow-specific there; it will annoy them.
Have a look at the how to ask a good question page. If your question can't be made to follow the guidelines laid out there, it's likely not a very good question for this site.