I was recently surfing around the Physicsoverflow and I have to admit I was impressed by their Review section. I could not help myself from thinking how -and if- such a feature could also be implemented here at Mathoverflow.

I can clearly understand that maybe this is not that simple: Refereeing a paper in Physics can be significantly different than refereeing a paper in mathematics. Detailed checks of computations and statements seem -well at least ideally- more important in refereeing of mathematical papers while the overall estimation of the novelty of an idea or model and its relation to experimental data or phenomenological approaches seem to be the focus in lots of cases of physics papers.

However, I feel that the community might have the power to add such a feature in this site, at least for some specific categories, where there are enough experts willing to undertake the job (for example algebraic geometry or number theory). I am thinking that initially, it would not necessarily have to do with refereeing of full papers (I guess that few mathematicians would like to initially submit their full paper in the site; lots would prefer a mainstream journal). However it might have to do with refereeing specific proofs of new results or parts of the whole argument.

P.S.: Although one might say that initially this could just be a check-my-proof feature, what I am having in mind is something more substantial, with the ambition to evolve in more conceptual community-reviewing of both problems and arguments and finally whole papers.

  • 4
    $\begingroup$ Isn't peer-review quite orthogonal to the purpose of MathOverflow? -- I'd rather think such feature may be a worthwhile addition to the arXiv, but I can't really imagine how to sensibly integrate this into MathOverflow. $\endgroup$ – Stefan Kohl Jan 5 '17 at 23:43
  • 2
    $\begingroup$ Quite orthogonal? In which metric? or ...geometry ;) $\endgroup$ – Konstantinos Kanakoglou Jan 6 '17 at 1:26
  • 6
    $\begingroup$ Now seriously: arXiv isn't really much of a community, at least not in the terms of the communities of the stackexchange network. In this sense, it more resembles (imo) traditional journals. I was thinking of motivating ideas for sensibly integrating such a feature in Mathoverflow community. $\endgroup$ – Konstantinos Kanakoglou Jan 6 '17 at 1:29
  • 1
    $\begingroup$ What would be the legal status of a paper you post here? Things on the SE network has a free license. How do you think a journal would react when these license terms overrule charging money for publication? $\endgroup$ – Asaf Karagila Jan 6 '17 at 15:45
  • 1
    $\begingroup$ (And please don't give me that "you won't have to publish in a conventional journal!" answer. When search committee/tenure committees/or otherwise committees agree that MO reputation is equivalent to actual reputation, and I can get a permanent position over my participation here, then we can talk. Until then, this is just something nice to write in CV in addition to having my papers published properly.) $\endgroup$ – Asaf Karagila Jan 6 '17 at 15:47
  • 7
    $\begingroup$ @AsafKaragila and others it should probably be clarified that for the Reviews section of PhysicsOverflow the papers themself do not actually get posted to the site. Each question that suggests a paper to be reviewed (called submission) contains a link to its online source (mostly ArXiv) together with the abstract and potentially additional comments. Should MathOverflow decide to adopt a similar procedure that just links to the papers to get discussed, there should be no legal issues or obstructions coming up ... $\endgroup$ – Dilaton Jan 6 '17 at 16:09
  • $\begingroup$ @Dilaton: It's true, I don't know what or how the feature works over at the other website. However, the answers and arguments here are in favor of piloting a feature that would act as a de facto refereeing process for a paper. $\endgroup$ – Asaf Karagila Jan 6 '17 at 16:12
  • 7
    $\begingroup$ May I point out pubpeer.com ? They currently get most activity around biology papers, and mostly people pointing out 'mistakes', innocent or otherwise, but everything on the arXiv, and everything with a DOI is indexed and can be commented on. This would cover almost all mathematics papers. $\endgroup$ – David Roberts Jan 6 '17 at 21:29
  • 1
    $\begingroup$ You should probably describe in the body of your question what the Review section of Physics Overflow does. A link is not enough for us to understand. $\endgroup$ – Federico Poloni Jan 8 '17 at 22:07
  • 2
    $\begingroup$ @FedericoPoloni the basic idea behind the Reviews secion of PhysicsOverflow is rather simple: the question suggests a paper to be reviewed, the answers contain (partial) reviews, and further discussion can take place in comments to both. There are two different criteria the paper can be rated on separately (accuracy and originality), but maybe MO would be just fine keeping the normal voting system for the review questions. $\endgroup$ – Dilaton Jan 11 '17 at 22:18
  • 1
    $\begingroup$ The only thing a bit tricky I could imagine is to achieve that the author(s) of the paper earn(s) the rep from the votes on the review question, in case the person who suggests the paper to be reviewed is not an author of it. On PhysicsOverflow, we have a dedicated meta thread where people can claim authorship and subsequently have the review questions that contain their papers assigned to their account. $\endgroup$ – Dilaton Jan 11 '17 at 22:22

I am intrigued by this idea and I think it would be a wonderful thing if it gained the support and acceptance of the MathOverflow community. So I am supportive of further discussion of the proposal.

My opinion is that our current refereeing process in mathematics is basically broken, and that the professional mathematics research community should pursue alternative experimental efforts, to find out how we should best reform it. I would view such a paper-review project on MathOverflow as such an experimental effort. Perhaps it could be very successful, which would be good to know. Or perhaps it will be flawed in a way that will be good to learn.

With the support of the MO community, such a system could be implemented at first on a provisional basis, with the promise to remove it after a year or two, say, if it fails.

There are, of course, many practical issues to consider. At the very least, the review posts should look different than ordinary MO questions, with a different background, or they should exist in a separate stream. Does one want to impose rep limits for people to contribute papers?

  • 1
    $\begingroup$ thank you for your feedback. This is what I had in mind initially: trying an experimental form of community peer-reviewing (possibly in some seperate stream) and watching it over time to see whether it will prove successfull and to what extend. I have some more concrete thoughts about it but I am sure lots of other people will also have. I will wait for some more feedback and maybe I will come back with some edit or some new relevant post. Maybe it would be also interesting and relevant to discuss what mathematical peer-reviewing is today and what it should be .... $\endgroup$ – Konstantinos Kanakoglou Jan 5 '17 at 20:40
  • 2
    $\begingroup$ There have been previous attempts at such discussion. Perhaps someone will recall Andrew Stacey's website which was like old meta for publishing (I forgot). Also some overlay journals and things like the Selected Papers Network exist. Also, the nLab is a successful variation but using notes instead of papers. I encourage an independent effort by motivated members of this community. I discourage making any tight links between such an effort and MathOverflow as we don't understand how frail or robust is the forum ecology. Gerhard "Do Not Globally Warm MathOverflow" Paseman, 2017.01.05. $\endgroup$ – Gerhard Paseman Jan 5 '17 at 21:23
  • 1
    $\begingroup$ Joel, this sounds like "This sounds like a very good another site to have linked from the main page, perhaps even an RSS-feed thereof". I'm not disagreeing with your claims about refereeing being broken (although I admittedly had very limited tenure on both side of the process), but it seems like a bad fit to this website. I do think, though, that if such an experiment is constructed properly, having an RSS feed or some link highly visible on the main page is not necessarily a bad thing. $\endgroup$ – Asaf Karagila Jan 6 '17 at 9:32
  • 4
    $\begingroup$ Oh, my view is that the peer-review system would be most successful and interesting, if it should be integrated with the MO site, and not merely linked or conceived as a separate site. Indeed, I think this kind of integration is what could make it successful. It would add to MO. The expertise is here, the interest, the number of knowledgeable people. Imagine having great papers come up in the MO feed---it would be great! Commenting on papers is similar enough to answering questions, a kind of long-form question, that I think it would work well as an integrated system. This is why I support it. $\endgroup$ – Joel David Hamkins Jan 6 '17 at 10:50
  • $\begingroup$ Oddly enough, however, despite being on arXiv for a while, for the whole world to see, I receive little to no comments on my papers (the one about Fodor's lemma received some comments from people who were asked specifically to read it, but not much else). Do you think that if I had posted them here, it would be any different? $\endgroup$ – Asaf Karagila Jan 6 '17 at 13:04
  • 8
    $\begingroup$ Yes, I do, and that is why I support this idea, as an integrated system. The difference is that we have here a community of expert mathematicians ready to engage in mathematics online. If the reviews are part of some other site, merely linked to here, then it will be dead. $\endgroup$ – Joel David Hamkins Jan 6 '17 at 13:19
  • 3
    $\begingroup$ So all those people, many of which I met already in real life, who have looked at my paper in the arXiv announcements and said "Eh, I'll read it later", would suddenly be eager to help. Other than the novelty factor, which tends to dry out after a couple of weeks (or months at best), how is this sustainable? Especially since the expert of a given field remain the same expert over a long period of time. $\endgroup$ – Asaf Karagila Jan 6 '17 at 15:43
  • 18
    $\begingroup$ My opinion is that MathOverflow works in large part due to the genius social engineering of the site, which for various reasons that the social scientists are studying motivates people to participate (e.g. even in the absense of financial compensation). It is a mistake to underestimate the power of this social engineering. I believe that it is the public nature of the participation that will help such a peer-review system to succeed on MathOverflow. The arxiv and the article-review system generally suffer from a lack of well-designed social engineering motivating participation. $\endgroup$ – Joel David Hamkins Jan 6 '17 at 15:53
  • 7
    $\begingroup$ I know several people who actively try to avoid MathOverflow. Some highly reputed mathematicians, whose reasoning is "I'll probably won't be able to stop, and I won't work on my own research as much as I'd want to"; and young people "I am too afraid to embarrass myself by saying stupid things". MO works largely because of social engineering, yes, but also because there is a large part of the mathematical community which "fits the profile". Here we intentionally ignore the other part, which is probably even larger. [...] $\endgroup$ – Asaf Karagila Jan 6 '17 at 16:08
  • $\begingroup$ [...] I am currently preparing a talk for the arctic set theory conference in a few weeks. I am very very reluctant to say a lot of things because it's a public lecture, and because there are slides. I chose a topic which is still work in progress, work which I haven't discussed with many people outside my closer circle of mathematicians, because I don't want to spread "mistakes" and not everything is verified. Making mistakes in public (both as a reviewer and a writer) does not make it easier to make the mistakes, and some people find those to be even more off-putting. $\endgroup$ – Asaf Karagila Jan 6 '17 at 16:10

Very interesting suggestion. While I definitely think that, if doable, this should be a separate activity not related to MO in any way, something related came to my mind.

I've seen several great answers here on MO which contain highly nontrivial proofs/arguments. For me in terms of provided information they are similar to an arXiv preprint: a piece of math to be treated with great care; in particular, if going to use it in, say, a paper, I feel obliged to check it myself first.

So would not it be great if there would be something like such a Review section but with the purpose of submitting MO answers? After all this is a Q&A site, and eventually the answers are supposed to reach finalized state, so why not use a separate section to certify tricky answers once and for all? They could then be tagged appropriately, and if some people would decide to use them in a publication they would know that the information provided has been double-checked by some other members of the community.

I know that implicitly this is being done all the time - as soon as I see an interesting nonobvious answer the first thing I try to do is to check it myself, and I am sure many of us here are like this. But without being responsible I can never be sure I will not overlook something, out of laziness or lack of time or whatever. On the other hand if I decide to review an answer explicitly designated for reviewing and will know that the result will be some sort of certificate from the community, I will be much more attentive and careful.


Although I maintain that MathOverflow is not the place to establish such a review system, I am willing to participate in a discussion of it here on meta, until a better place for this discussion appears.

I recommend looking at previously logged discussions of this and similar issues. I think some are on publishing.mathforge.org, and there are open publishing fora and various blogs (including that of John Baez and his mention of the Selected Papers Network). Collectively, they should cover many of the main issues concerning an online review system. I invite others to edit this post and add links to any discussions they find helpful. (I start it with a post of Daubechies on WML.)

Math Annotate Platform?

I see quality and maintaining interest in running such a system as primary issues. It seems to me that (like ArXiv) an add-only repository is needed, so that context for review efforts is stable. I think issues like research arising from reviews needs to be anticipated. (If everyone were meticulous about saying "Based on web paper X (starting from lexeme Y through Z), we can state the following improvement...", there would be few problems. Even then there may be political and social issues surrounding attribution of ideas.)

It might make sense to include discussion of existing papers as well. Until the WML is set up, one needs to take care of establishing and archiving a copy of the paper so as not to violate copyright issues. Even if a digital copy is not established, one needs to make a reference to a non digital copy that all reviewers can resolve reliably.

I am in favor of open and transparent reviews where the reviewer has a registered and established identity. I also see the need for sponsored reviews. These would be anonymous reviews except instead the (identity of the) originator of the review is known only to a sponsor or small team of sponsors who agree to reveal this information only under certain conditions.

There are system moderation and maintenance issues as well. I invite others to add to this post (or create their own) to indicate other issues around online collaborative review for discussion.

Gerhard "This Collaboration Comes To Order" Paseman, 2017.01.05.

  • $\begingroup$ Sadly neither publishing.mathforge.org nor Selected Papers Network seem to be getting any "life" lately, and there seem to be no other options. MO would seem to remain the only choice then. Would my answer mathoverflow.net/a/284322/66983 address the possible objections or there are still further arguments left unaddressed? $\endgroup$ – Dmitri Zaitsev Nov 15 '17 at 21:59
  • $\begingroup$ I suspect author controlled commentary would be too biased. Community controlled commentary is also biased, but presumably biased in favor of improving a paper, or determining the flaws in an unfixable paper. @Dmitri, I think your answer raises more issues than it addresses satisfactorily. Gerhard "And That Is My Bias" Paseman, 2017.11.15. $\endgroup$ – Gerhard Paseman Nov 16 '17 at 2:35
  • $\begingroup$ I suppose biased in favour of improving paper should be a great thing. As any bias encouraging positives and suppressing negatives. Sounds like a strong argument to do it! About the issues with my answer, I would be curious to learn what they are. We need some constructive discussion to move forward. $\endgroup$ – Dmitri Zaitsev Nov 16 '17 at 11:19

My understanding is that PhysicsOverflow is inspired by (and in some fashion derives from) MathOverflow and StackExchange, but is not part of the forum network. Thus they have more freedom to innovate than this community does. (Not that we can't innovate, it's just that we have more conditions to satisfy.)

The major strength of this forum and of the StackExchange network is that it is directed at specific Q&A, not discussions or any lengthy considerations. Within this framework brief ideas and proof snippets can be reviewed and critiqued; there is community resistance for reviewing anything longer.

Of course, you can try rolling your own, and see what community builds around your efforts.

Gerhard "Likes Them Short And Sweet" Paseman, 2017.01.04.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .