Right now, there is a question posted on MO that rightly does not belong to MO: The equivalence of the mathematical induction and the principle of the smallest. Of course, such questions comes and goes every-MO-days. More often than not, if they are lucky, they get one or two polite comments that MO is not right place for asking such questions. What if they also get a comment that also adding that "look at that reference" or "maybe you could find an answer in this (other) site", or even a very short hint as an answer. Indeed, I was about to put such a comment for the question linked above. Then I said to myself, maybe it is wrong doing. Is there any general policy for putting comments on such questions? Or better to ask: Should there be a general policy for putting comments on such questions?
My feeling is that one should not give answers (or hints) to questions that are not appropriate for MO, even in the comments. Doing so encourages people to post inappropriate questions in the hope that they'll get an answer even though the question is closed.
I am on the opposite end from Andy. My (humble, as usual) opinion is that (barring complete trivialities where you intend to send the person the direct message that he is either too stupid or too lazy to waste your time on or that his question has absolutely nothing to do with mathematics), you should demonstrate that the problem is easily solvable before casting the first vote to close. So, my current policy is to give out enough information for a reasonably clever reader to finish the solution without too much effort (and even then I sometimes abstain from voting to close myself). I understand what Andy is afraid of, but I am much more reluctant to make the error of the opposite type and drive someone away without a really good reason. The inappropriate questions that hang for a day or two can always be deleted automatically after closure but the reopening of a decent question may easily happen too late, when the OP has already sent us to the Hell after seeing his question downvoted and closed without any comment (or with some meaningless and technically wrong comments, I'm not sure which one is worse) within half an hour after posting.
It is considered bad form (at least by some) to answer off-topic question.
This applies mainly to proper answers, but one reason being that receiving answers encourages to come back with more questions, it extends to answers in comments. (Added: The answer appearing while I type confirms this view.)
That being said, certainly a hint here or there occurs with some frequency and is not considered as some violation of a policy.
However, some more points to consider against giving too much information in comments (sometimes I do this myself, also see at the end):
It can happen that one comment is not enough. Then there are requests for clarfication. Maybe even an edit of the question, keeping it still longer visible and so on.
Until now I only gave reasons why it is not good "for MO" but I feel it is not only this. It can also be bad for OP. There is a site math.SE where many questions that are off-topic here as too low level but otherwise reasonable (such as this one, I think) are perfectly on-topic. And there they would typically receive a proper or even several good answers. And not just some quick comment. Giving a quick answer might have as effect that OP does not ask there and get a better answer. Also, this might not be the only question an OP has, and the next ones will likely also fit better there on math.SE. I think the best course of action is to direct such posters, having reasonable math questions but not at the level appropriate for MO, to math.SE as directly as possible. (If not via a proper migration, just via telling them about the other site.)
There can be cases where sending an OP to math.SE is not optimal. For example, the question could be very unclear or likely mistated or some other things. In these cases I sometimes leave a comment making OP aware of these issues, with the encouragement to fix them first and then ask on math.SE if still needed.
So, in brief, while I do not consider it as a big problem if people answer off-topic questions a bit in comments, I do not think it is good to do so in general also as most of the time it seems to me it is not even the most helpful course of action for OP.