25
$\begingroup$

I was recently surfing around the Physicsoverflow and I have to admit I was impressed by their Review section. I could not help myself from thinking how -and if- such a feature could also be implemented here at Mathoverflow.

I can clearly understand that maybe this is not that simple: Refereeing a paper in Physics can be significantly different than refereeing a paper in mathematics. Detailed checks of computations and statements seem -well at least ideally- more important in refereeing of mathematical papers while the overall estimation of the novelty of an idea or model and its relation to experimental data or phenomenological approaches seem to be the focus in lots of cases of physics papers.

However, I feel that the community might have the power to add such a feature in this site, at least for some specific categories, where there are enough experts willing to undertake the job (for example algebraic geometry or number theory). I am thinking that initially, it would not necessarily have to do with refereeing of full papers (I guess that few mathematicians would like to initially submit their full paper in the site; lots would prefer a mainstream journal). However it might have to do with refereeing specific proofs of new results or parts of the whole argument.

P.S.: Although one might say that initially this could just be a check-my-proof feature, what I am having in mind is something more substantial, with the ambition to evolve in more conceptual community-reviewing of both problems and arguments and finally whole papers.

$\endgroup$
14
  • 7
    $\begingroup$ Isn't peer-review quite orthogonal to the purpose of MathOverflow? -- I'd rather think such feature may be a worthwhile addition to the arXiv, but I can't really imagine how to sensibly integrate this into MathOverflow. $\endgroup$
    – Stefan Kohl Mod
    Commented Jan 5, 2017 at 23:43
  • 2
    $\begingroup$ Quite orthogonal? In which metric? or ...geometry ;) $\endgroup$
    – Konstantinos Kanakoglou
    Commented Jan 6, 2017 at 1:26
  • 7
    $\begingroup$ Now seriously: arXiv isn't really much of a community, at least not in the terms of the communities of the stackexchange network. In this sense, it more resembles (imo) traditional journals. I was thinking of motivating ideas for sensibly integrating such a feature in Mathoverflow community. $\endgroup$
    – Konstantinos Kanakoglou
    Commented Jan 6, 2017 at 1:29
  • 1
    $\begingroup$ What would be the legal status of a paper you post here? Things on the SE network has a free license. How do you think a journal would react when these license terms overrule charging money for publication? $\endgroup$
    – Asaf Karagila Mod
    Commented Jan 6, 2017 at 15:45
  • 1
    $\begingroup$ (And please don't give me that "you won't have to publish in a conventional journal!" answer. When search committee/tenure committees/or otherwise committees agree that MO reputation is equivalent to actual reputation, and I can get a permanent position over my participation here, then we can talk. Until then, this is just something nice to write in CV in addition to having my papers published properly.) $\endgroup$
    – Asaf Karagila Mod
    Commented Jan 6, 2017 at 15:47
  • 8
    $\begingroup$ @AsafKaragila and others it should probably be clarified that for the Reviews section of PhysicsOverflow the papers themself do not actually get posted to the site. Each question that suggests a paper to be reviewed (called submission) contains a link to its online source (mostly ArXiv) together with the abstract and potentially additional comments. Should MathOverflow decide to adopt a similar procedure that just links to the papers to get discussed, there should be no legal issues or obstructions coming up ... $\endgroup$
    – Dilaton
    Commented Jan 6, 2017 at 16:09
  • $\begingroup$ @Dilaton: It's true, I don't know what or how the feature works over at the other website. However, the answers and arguments here are in favor of piloting a feature that would act as a de facto refereeing process for a paper. $\endgroup$
    – Asaf Karagila Mod
    Commented Jan 6, 2017 at 16:12
  • 7
    $\begingroup$ May I point out pubpeer.com ? They currently get most activity around biology papers, and mostly people pointing out 'mistakes', innocent or otherwise, but everything on the arXiv, and everything with a DOI is indexed and can be commented on. This would cover almost all mathematics papers. $\endgroup$
    – David Roberts Mod
    Commented Jan 6, 2017 at 21:29
  • 1
    $\begingroup$ You should probably describe in the body of your question what the Review section of Physics Overflow does. A link is not enough for us to understand. $\endgroup$
    – Federico Poloni
    Commented Jan 8, 2017 at 22:07
  • 2
    $\begingroup$ @FedericoPoloni the basic idea behind the Reviews secion of PhysicsOverflow is rather simple: the question suggests a paper to be reviewed, the answers contain (partial) reviews, and further discussion can take place in comments to both. There are two different criteria the paper can be rated on separately (accuracy and originality), but maybe MO would be just fine keeping the normal voting system for the review questions. $\endgroup$
    – Dilaton
    Commented Jan 11, 2017 at 22:18
  • 1
    $\begingroup$ The only thing a bit tricky I could imagine is to achieve that the author(s) of the paper earn(s) the rep from the votes on the review question, in case the person who suggests the paper to be reviewed is not an author of it. On PhysicsOverflow, we have a dedicated meta thread where people can claim authorship and subsequently have the review questions that contain their papers assigned to their account. $\endgroup$
    – Dilaton
    Commented Jan 11, 2017 at 22:22
  • 3
    $\begingroup$ This question is from January 2017. It would be interesting to hear comments on how well Reviews on PhysicsOverflow have worked in the last 2.5 years. $\endgroup$
    – Gerald Edgar
    Commented Jul 19, 2019 at 13:38
  • $\begingroup$ I plan to start an area 51 site. Any support is welcome! Area 51 community peer review $\endgroup$
    – Thanasis Stylianou
    Commented Aug 13, 2019 at 13:53
  • 1
    $\begingroup$ @GeraldEdgar PO has 165 review requests to date, around one review request per month and almost every second request has a review. $\endgroup$
    – Thanasis Stylianou
    Commented Aug 13, 2019 at 14:20

5 Answers 5

Reset to default
27
$\begingroup$

I am intrigued by this idea and I think it would be a wonderful thing if it gained the support and acceptance of the MathOverflow community. So I am supportive of further discussion of the proposal.

My opinion is that our current refereeing process in mathematics is basically broken, and that the professional mathematics research community should pursue alternative experimental efforts, to find out how we should best reform it. I would view such a paper-review project on MathOverflow as such an experimental effort. Perhaps it could be very successful, which would be good to know. Or perhaps it will be flawed in a way that will be good to learn.

With the support of the MO community, such a system could be implemented at first on a provisional basis, with the promise to remove it after a year or two, say, if it fails.

There are, of course, many practical issues to consider. At the very least, the review posts should look different than ordinary MO questions, with a different background, or they should exist in a separate stream. Does one want to impose rep limits for people to contribute papers?

$\endgroup$
11
  • 1
    $\begingroup$ thank you for your feedback. This is what I had in mind initially: trying an experimental form of community peer-reviewing (possibly in some seperate stream) and watching it over time to see whether it will prove successfull and to what extend. I have some more concrete thoughts about it but I am sure lots of other people will also have. I will wait for some more feedback and maybe I will come back with some edit or some new relevant post. Maybe it would be also interesting and relevant to discuss what mathematical peer-reviewing is today and what it should be .... $\endgroup$
    – Konstantinos Kanakoglou
    Commented Jan 5, 2017 at 20:40
  • 2
    $\begingroup$ There have been previous attempts at such discussion. Perhaps someone will recall Andrew Stacey's website which was like old meta for publishing (I forgot). Also some overlay journals and things like the Selected Papers Network exist. Also, the nLab is a successful variation but using notes instead of papers. I encourage an independent effort by motivated members of this community. I discourage making any tight links between such an effort and MathOverflow as we don't understand how frail or robust is the forum ecology. Gerhard "Do Not Globally Warm MathOverflow" Paseman, 2017.01.05. $\endgroup$
    – Gerhard Paseman
    Commented Jan 5, 2017 at 21:23
  • 1
    $\begingroup$ Joel, this sounds like "This sounds like a very good another site to have linked from the main page, perhaps even an RSS-feed thereof". I'm not disagreeing with your claims about refereeing being broken (although I admittedly had very limited tenure on both side of the process), but it seems like a bad fit to this website. I do think, though, that if such an experiment is constructed properly, having an RSS feed or some link highly visible on the main page is not necessarily a bad thing. $\endgroup$
    – Asaf Karagila Mod
    Commented Jan 6, 2017 at 9:32
  • 6
    $\begingroup$ Oh, my view is that the peer-review system would be most successful and interesting, if it should be integrated with the MO site, and not merely linked or conceived as a separate site. Indeed, I think this kind of integration is what could make it successful. It would add to MO. The expertise is here, the interest, the number of knowledgeable people. Imagine having great papers come up in the MO feed---it would be great! Commenting on papers is similar enough to answering questions, a kind of long-form question, that I think it would work well as an integrated system. This is why I support it. $\endgroup$
    – Joel David Hamkins
    Commented Jan 6, 2017 at 10:50
  • $\begingroup$ Oddly enough, however, despite being on arXiv for a while, for the whole world to see, I receive little to no comments on my papers (the one about Fodor's lemma received some comments from people who were asked specifically to read it, but not much else). Do you think that if I had posted them here, it would be any different? $\endgroup$
    – Asaf Karagila Mod
    Commented Jan 6, 2017 at 13:04
  • 8
    $\begingroup$ Yes, I do, and that is why I support this idea, as an integrated system. The difference is that we have here a community of expert mathematicians ready to engage in mathematics online. If the reviews are part of some other site, merely linked to here, then it will be dead. $\endgroup$
    – Joel David Hamkins
    Commented Jan 6, 2017 at 13:19
  • 4
    $\begingroup$ So all those people, many of which I met already in real life, who have looked at my paper in the arXiv announcements and said "Eh, I'll read it later", would suddenly be eager to help. Other than the novelty factor, which tends to dry out after a couple of weeks (or months at best), how is this sustainable? Especially since the expert of a given field remain the same expert over a long period of time. $\endgroup$
    – Asaf Karagila Mod
    Commented Jan 6, 2017 at 15:43
  • 21
    $\begingroup$ My opinion is that MathOverflow works in large part due to the genius social engineering of the site, which for various reasons that the social scientists are studying motivates people to participate (e.g. even in the absense of financial compensation). It is a mistake to underestimate the power of this social engineering. I believe that it is the public nature of the participation that will help such a peer-review system to succeed on MathOverflow. The arxiv and the article-review system generally suffer from a lack of well-designed social engineering motivating participation. $\endgroup$
    – Joel David Hamkins
    Commented Jan 6, 2017 at 15:53
  • 9
    $\begingroup$ I know several people who actively try to avoid MathOverflow. Some highly reputed mathematicians, whose reasoning is "I'll probably won't be able to stop, and I won't work on my own research as much as I'd want to"; and young people "I am too afraid to embarrass myself by saying stupid things". MO works largely because of social engineering, yes, but also because there is a large part of the mathematical community which "fits the profile". Here we intentionally ignore the other part, which is probably even larger. [...] $\endgroup$
    – Asaf Karagila Mod
    Commented Jan 6, 2017 at 16:08
  • $\begingroup$ [...] I am currently preparing a talk for the arctic set theory conference in a few weeks. I am very very reluctant to say a lot of things because it's a public lecture, and because there are slides. I chose a topic which is still work in progress, work which I haven't discussed with many people outside my closer circle of mathematicians, because I don't want to spread "mistakes" and not everything is verified. Making mistakes in public (both as a reviewer and a writer) does not make it easier to make the mistakes, and some people find those to be even more off-putting. $\endgroup$
    – Asaf Karagila Mod
    Commented Jan 6, 2017 at 16:10
  • $\begingroup$ I plan to start an area 51 site: meta.mathoverflow.net/q/4320/40644 Any support is welcome! $\endgroup$
    – Thanasis Stylianou
    Commented Aug 13, 2019 at 20:41
11
$\begingroup$

Very interesting suggestion. While I definitely think that, if doable, this should be a separate activity not related to MO in any way, something related came to my mind.

I've seen several great answers here on MO which contain highly nontrivial proofs/arguments. For me in terms of provided information they are similar to an arXiv preprint: a piece of math to be treated with great care; in particular, if going to use it in, say, a paper, I feel obliged to check it myself first.

So would not it be great if there would be something like such a Review section but with the purpose of submitting MO answers? After all this is a Q&A site, and eventually the answers are supposed to reach finalized state, so why not use a separate section to certify tricky answers once and for all? They could then be tagged appropriately, and if some people would decide to use them in a publication they would know that the information provided has been double-checked by some other members of the community.

I know that implicitly this is being done all the time - as soon as I see an interesting nonobvious answer the first thing I try to do is to check it myself, and I am sure many of us here are like this. But without being responsible I can never be sure I will not overlook something, out of laziness or lack of time or whatever. On the other hand if I decide to review an answer explicitly designated for reviewing and will know that the result will be some sort of certificate from the community, I will be much more attentive and careful.

$\endgroup$
6
$\begingroup$

Although I maintain that MathOverflow is not the place to establish such a review system, I am willing to participate in a discussion of it here on meta, until a better place for this discussion appears.

I recommend looking at previously logged discussions of this and similar issues. I think some are on publishing.mathforge.org, and there are open publishing fora and various blogs (including that of John Baez and his mention of the Selected Papers Network). Collectively, they should cover many of the main issues concerning an online review system. I invite others to edit this post and add links to any discussions they find helpful. (I start it with a post of Daubechies on WML.)

Math Annotate Platform?

I see quality and maintaining interest in running such a system as primary issues. It seems to me that (like ArXiv) an add-only repository is needed, so that context for review efforts is stable. I think issues like research arising from reviews needs to be anticipated. (If everyone were meticulous about saying "Based on web paper X (starting from lexeme Y through Z), we can state the following improvement...", there would be few problems. Even then there may be political and social issues surrounding attribution of ideas.)

It might make sense to include discussion of existing papers as well. Until the WML is set up, one needs to take care of establishing and archiving a copy of the paper so as not to violate copyright issues. Even if a digital copy is not established, one needs to make a reference to a non digital copy that all reviewers can resolve reliably.

I am in favor of open and transparent reviews where the reviewer has a registered and established identity. I also see the need for sponsored reviews. These would be anonymous reviews except instead the (identity of the) originator of the review is known only to a sponsor or small team of sponsors who agree to reveal this information only under certain conditions.

There are system moderation and maintenance issues as well. I invite others to add to this post (or create their own) to indicate other issues around online collaborative review for discussion.

Gerhard "This Collaboration Comes To Order" Paseman, 2017.01.05.

$\endgroup$
3
  • $\begingroup$ Sadly neither publishing.mathforge.org nor Selected Papers Network seem to be getting any "life" lately, and there seem to be no other options. MO would seem to remain the only choice then. Would my answer mathoverflow.net/a/284322/66983 address the possible objections or there are still further arguments left unaddressed? $\endgroup$
    – Dmitri Zaitsev
    Commented Nov 15, 2017 at 21:59
  • $\begingroup$ I suspect author controlled commentary would be too biased. Community controlled commentary is also biased, but presumably biased in favor of improving a paper, or determining the flaws in an unfixable paper. @Dmitri, I think your answer raises more issues than it addresses satisfactorily. Gerhard "And That Is My Bias" Paseman, 2017.11.15. $\endgroup$
    – Gerhard Paseman
    Commented Nov 16, 2017 at 2:35
  • 1
    $\begingroup$ I suppose biased in favour of improving paper should be a great thing. As any bias encouraging positives and suppressing negatives. Sounds like a strong argument to do it! About the issues with my answer, I would be curious to learn what they are. We need some constructive discussion to move forward. $\endgroup$
    – Dmitri Zaitsev
    Commented Nov 16, 2017 at 11:19
3
$\begingroup$

My understanding is that PhysicsOverflow is inspired by (and in some fashion derives from) MathOverflow and StackExchange, but is not part of the forum network. Thus they have more freedom to innovate than this community does. (Not that we can't innovate, it's just that we have more conditions to satisfy.)

The major strength of this forum and of the StackExchange network is that it is directed at specific Q&A, not discussions or any lengthy considerations. Within this framework brief ideas and proof snippets can be reviewed and critiqued; there is community resistance for reviewing anything longer.

Of course, you can try rolling your own, and see what community builds around your efforts.

Gerhard "Likes Them Short And Sweet" Paseman, 2017.01.04.

$\endgroup$
-3
$\begingroup$

This is a very good idea.

Since the question was asked there has not really been any progress in making the peer-review system healthier. Also initiatives such as Plan-S in Europe have not really contributed in this direction. There seem to be larger and larger masses of papers submitted to classical peer-reviewed journals and most of them are congested.

My opinion is that such a system (posting abstracts of linked arxiv articles on MO) and asking for reviews would be a big relief to the system. However I think that the main problem would be of finding a way to include anonymized (sponsored) reviews (which is a very important aspect of the peer-review system). Maybe one could have a group of moderators acting as editors and they would be responsible for searching/receiving a review (which could also come from outside the MO community) and posting it as an answer.

I also think that math annotations and reviews are two different things. I would first stick to the review context, where answers are only allowed to be classical reviews and comments would only concern specific (technical) questions. This would imply some sort of "strong" moderation from the community, but I think that the success of MO has shown that this is possible (I myself would be prepared to be involved in sections related to my subjects - for sectioning we could for example use MSC classifications).

Moreover, I do not think that MO rep of the user (author) who submits a paper for review is going to be relevant in the hiring process (or at least as relevant as RG's score or H-index etc. is), but I somehow envision this MO section as becoming a new type of "journal", which assigns reputation to an article. Having a linked paper with an "accepted review" and a good number of upvotes could be considered as having a paper with a substantial impact.

If you ask me, I think we should just do it and see how it evolves. Otherwise discussions will just continue until the people don't have anymore time to invest.

EDIT: I have posted an example review here (Example) Peer-Review request: Independent, Negative, Canonically Turing Arrows of Equations and Problems in Applied Formal PDE

Hopefully I have not broken around 1000 forum rules.

EDIT2: I have deleted the above post and will give a try with Gerhard Paseman's article he was kind enough to "volunteer it" for a community review. Since I regard the current problems of the peer-reviewing system as dire, it would be great if the downvoters also comment on the subject. The main goal is to find a way to improve the system-this is a problem of the community and it has to be solved within it.

EDIT3: Gerhard Paseman's article has been posted: Peer-Review request: Updating an upper bound of Erik Westzynthius

EDIT4: Sadly the experiment ended quickly. It appears there is a strong opposition in using MO in this way. I only see three alternatives:

  1. We start a new community in area 51
  2. We join PhysicsOverflow.
  3. We create something like the Review section of PhysicsOverflow (maybe we can get some help from them?)
  4. We do nothing.

I am not in favour of 2. This is a mathematics community and, although there is a math.phys tag in PO, I think that joining them would be too much. 3. makes some sense, but probably requires a lot of time and effort which many of us (including myself) are not able to afford. Starting a new community in area51 is something I would do. Joining a group of people that want to start sth like PO would also be sth I would do.

Here is how the system works at PO: (3) The review section

$\endgroup$
10
  • 1
    $\begingroup$ Maybe not 1,000, but given a little more time.... $\endgroup$
    – Gerry Myerson
    Commented Jul 17, 2019 at 11:50
  • 1
    $\begingroup$ I applaud your courage in making the attempt. I disapprove of your example choice of article to peer-review. If you wish for something more serious I volunteer my article on ArXiv 1311.5944 as strawman. It contains some original work, but is not suitable for publication as the main result is not new (even though it improves upon recently published results). I am interested in rewriting it to improve accessibility and style of presentation, and would welcome feedback on these issues. Gerhard "Maybe It Could Help Someone" Paseman, 2019.07.17. $\endgroup$
    – Gerhard Paseman
    Commented Jul 17, 2019 at 15:04
  • 1
    $\begingroup$ @GerhardPaseman Thank you! The reason I chose this article was that I did not want to use somebody's work without permission. So, I would gladly go on and use your paper! I would also post it directly on MO, and I will act as an "editor", that is, I will try to find suitable reviewers and accept sponsored reviews. Is there a place to keep the original reviewer's name which is only seen by moderators? $\endgroup$
    – Thanasis Stylianou
    Commented Jul 17, 2019 at 19:07
  • $\begingroup$ I just noticed that one is allowed to post as a guest, but has to enter an email which is not shown. We could allow guest reviewing but only with an institutional email. $\endgroup$
    – Thanasis Stylianou
    Commented Jul 17, 2019 at 19:27
  • $\begingroup$ I would recommend (for the initial experiment) that you post to the thread you created an answer. This answer you post can contain the article name, article version number, author names, link to detailed information (e.g. page length, journal name, etc.), and then a brief summary of review, followed by a link to a more detailed review, and name of reviewer and date of review. You oversee the whole process. Gerhard "Your Initiative, You Implement It" Paseman, 2019.07.17. $\endgroup$
    – Gerhard Paseman
    Commented Jul 17, 2019 at 20:00
  • $\begingroup$ By the way, guest reviewing will not work, since the email will be visible to any moderator, possibly including an author. @Dilaton how does this work in PhysicsOverflow? $\endgroup$
    – Thanasis Stylianou
    Commented Jul 17, 2019 at 20:47
  • $\begingroup$ My intention was that you experiment on meta, by adding an answer to your existing meta post. I believe the community is not ready for such an experimental post on MathOverflow itself. Gerhard "Thanks For Trying It Anyway" Paseman, 2019.07.17. $\endgroup$
    – Gerhard Paseman
    Commented Jul 17, 2019 at 22:01
  • 2
    $\begingroup$ Regarding EDIT3, note that the question was quickly closed and deleted. There appears to be some opposition to using MO in the manner Thanasis intended. $\endgroup$
    – Gerry Myerson
    Commented Jul 18, 2019 at 0:13
  • 1
    $\begingroup$ @GerryMyerson This seems to be true. I did not expect such a quick reaction. I expected that since this question (and the few more linked to it) has a good number of positive votes, the experiment would at least get some chance. I suppose the only way to go would be starting a new community in area 51 or ask the guys over ate PhysicsOverflow if we could use their platform (maybe professional mathematicians will be reluctant to join a physics forum...). $\endgroup$
    – Thanasis Stylianou
    Commented Jul 18, 2019 at 8:26
  • $\begingroup$ I plan to start an area 51 site: meta.mathoverflow.net/q/4320/40644 Any support is welcome! $\endgroup$
    – Thanasis Stylianou
    Commented Aug 13, 2019 at 14:22

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .