The new review feature is interesting, and I have experimented with it. I have a number of concerns- I wonder whether my concerns are shared, and whether there is anything that can be done to address them.

My primary concern is that, if one wants to review, one is presented with a random question/answer which is probably out of one's area of expertise. The right thing to do is to "skip". But the system is gamefied in such a way as to incentivise taking action anyway. If a question or answer "looks" bad but is actually good, I'm pretty sure that this system significantly increases the risk of incorrect closure (perhaps even deletion); especially as other people's votes are not visible. This appear to me to be a serious problem, which the previous incarnation of MO didn't have, certainly not as badly.

Additionally, we are asked to review first posts by new users, and late answers. But users who are new to MO are not necessarily new to mathematics. It seems strange that we should have to review a first post by a great and eminent mathematician just because they are new to MO. Also, in mathematics, it sometimes takes time to work things out. I'm not sure that late answers are necessarily suspect (although this feature does provide certain utility to clean up old questions and answers).

It makes more sense to me, for example, that we should review answers with zero or negative votes for a a long time (over a few weeks) to see whether they are in fact answers, irrespective of the experience of the respondent.

What do others thing of these new features?

The review system is not meant for finding posts that contain bad math, or that are just wrong. First posts are in the review queue because they are much more likely to be from a user that doesn't understand how this site works yet and e.g. posts a comment or other non-answer as an answer. Spammers or other users that vandalize the site are also usually found in the first post and late answer categories.

The system also contains review audits that present known bad or good posts, and will remind the user to not blindly click through all reviews if they fail the audit. If a user fails too often, they are automatically blocked from reviewing for a short time. Moderators can also ban specific users from the review queues for a certain time, if necessary. I'm not sure if they are enabled here, though.

The posts are also not random, the system takes the tags into account in which you are active and tries to show you posts from your area of expertise.

  • Thank you for your answer. I'm not convinced by the spam argument (not saying you're wrong, just that I personally am skeptical) because we were catching spam fine before- there didn't seem to have been a problem there. As for non-answers... well, maybe... but I suspect that the result might be more incorrect closures for the wrong reasons (which causes a lot of damage) than accurate flaggings of non-answers (whose effect on the site as a whole is marginal)... but maybe others think differently? – Daniel Moskovich Jun 26 '13 at 10:15
  • There is some potential for misuse in the review feature, so this is certainly something one should keep an eye on. Questions are also removed from the queue again if enough users vote to keep them open. – Mad Scientist Jun 26 '13 at 10:20

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .