I was recently surfing around the Physicsoverflow and I have to admit I was impressed by their Review section. I could not help myself from thinking how -and if- such a feature could also be implemented here at Mathoverflow.
I can clearly understand that maybe this is not that simple: Refereeing a paper in Physics can be significantly different than refereeing a paper in mathematics. Detailed checks of computations and statements seem -well at least ideally- more important in refereeing of mathematical papers while the overall estimation of the novelty of an idea or model and its relation to experimental data or phenomenological approaches seem to be the focus in lots of cases of physics papers.
However, I feel that the community might have the power to add such a feature in this site, at least for some specific categories, where there are enough experts willing to undertake the job (for example algebraic geometry or number theory). I am thinking that initially, it would not necessarily have to do with refereeing of full papers (I guess that few mathematicians would like to initially submit their full paper in the site; lots would prefer a mainstream journal). However it might have to do with refereeing specific proofs of new results or parts of the whole argument.
P.S.: Although one might say that initially this could just be a check-my-proof feature, what I am having in mind is something more substantial, with the ambition to evolve in more conceptual community-reviewing of both problems and arguments and finally whole papers.