# Improving citations of MathOverflow posts

There are already a lot of citations of MathOverflow in the literature. Nevertheless, there is some resistance in the greater mathematical community since MathOverflow is not a traditional medium. While some of this is simply resistance to change that is bound to slowly disappear, some of it reflects some real problems of the medium. The main issue is that MathOverflow is continually updating and this hinders the citability of MathOverflow. Indeed, since a post can change drastically or even completely disappear at any time, a simple link to a question or answer on MathOverflow isn't a reliable citation mechanism. The site does keep a revision history of each post which may help if after a post is edited but this doesn't help if a post has been deleted.

### How can we fix this problem?

One suggestion came up in an earlier discussion on the long-term archiving of MathOverflow. We could publish monthly or weekly static version of new and updated posts at archive.mathoverflow.net. The static site could even have volume and issue numbers, an ISSN, DOIs for posts, just like a traditional journal would have. Posts on this static MathOverflow Archive would be just as easy to cite as any journal article.

Maybe there are completely different ways to address this problem? There are also important implementation details to discuss with the MathOverflow Archive and other ideas. We seek input from the community on this important issue for the future of MathOverflow.

• I don't know how it works, but I'm wondering how you get DOIs. Do you pay once for a whole namespace, or do you pay for each entry? The first might work here, the latter would be problematic, I suspect.
– user35354
Feb 1 '14 at 22:38
• You pay crossref for each DOI assignment; prices vary. Feb 1 '14 at 23:25
• Are there many citations to MO that actually rely on the post being accessible? If I were to write say "The following lemma is due to F. G. Dorais [cite MO]. <Lemma> <Proof of Lemma>" it is not very relevant it is accessible, it is for giving credit, which is important, but nobody really needs to check it. But if I write "The following lemma is due to F. G. Dorais [cite MO]. <Lemma> (no proof!)" then it is key it is accessible. I suspect most quotes to be of former way (but I could be wrong, to know this is the main point of the cmnt). So "resistance" is a matter of style not accessability.
– user9072
Feb 1 '14 at 23:59
• @quid: That could be interesting to check. It might be a chicken/egg problem in the end. Incidentally, I was just reading another kind of citation: "This section benefits a lot from the discussion in [16]" where [16] is a reference to my answer here. Feb 2 '14 at 0:03
• Creating DOIs via CrossRef crossref.org/02publishers/20pub_fees.html requires a \$275 per annum fee (for a nonprofit) plus \$1 per DOI. There are alternative registration agencies for DOIs doi.org/registration_agencies.html perhaps Medra (marginal cost of DOIs 0.65 euros) may be applicable to us. Feb 2 '14 at 2:40
• @ScottMorrison: Yes, those are standard fees for journals. It's cheaper for other content types and it's not clear MO would fall into the \$1 category. If we find there is need for DOIs, we will need to negotiate with CrossRef (or others) for a better rate. Feb 2 '14 at 2:58 • What is (the legal representation of) StackExchange's position on archiving content? Feb 2 '14 at 8:35 • Archiving content is easily accommodated by the CC license all content is under. The archive would obviously have to link back to the main URL in the appropriate places, but I can't imagine any serious obstacle. Feb 2 '14 at 10:45 • I'd also suspect that deletion is far less of a problem that one might think. Posts that are worth being cited are extremely unlikely to be removed by the community or moderators. The only case I can imagine is a user rage-quitting and trying to delete their own posts, but that is not possible on a large scale without being noticed. – user35354 Feb 2 '14 at 12:02 • @MadScientist: Undeleting inappropriately deleted posts is not something we do every day but it happens often enough. I suspect it would be more frequent if we were more proactive about monitoring self-deletions. I just had a quick look and there were a few self-deletions in January that I would investigate further if I had time... Feb 2 '14 at 14:41 ## 4 Answers I'd be inclined against anything that could be interpreted as trying to make the mathoverflow archives look or behave like a journal. (E.g., assigning an ISSN, using volume or issue numbers, giving each post a DOI, etc. Strictly speaking, DOIs have nothing to do with journal articles, but that's by far the most common use case in mathematics, so it would look like it's imitating journals.) I think this could annoy people by seeming like MO is trying to become more citable by pretending to be something it's not. On the other hand, I think the citation issues can be resolved without any journal-like features. Here's what I think is needed: 1. Some organization needs to commit to long-term archiving. I'd recommend asking a university library and perhaps the Internet Archive. 2. MO needs a policy for how to deal with deletion, since archiving is useless if material can be deleted from the archives. The simplest solution is to preserve everything, but that has several drawbacks: you'll be preserving some total garbage, and crackpots may take advantage of it (even if their work is immediately deleted from the live MO site, it will be permanently accessible and citable in the archives). On the other hand, you need to make sure nothing of any actual value could ever be deleted from the archives. One solution could be to have a one-month window, and permanently archive anything that survives for a month on MO without being deleted. 3. It's also worth thinking about what else should be preserved. For example, comments certainly need to be archived, since they may play an important mathematical or historical role. Presumably it's safest to archive everything, even relatively superficial things like upvotes on comments. 4. MO's handling of permalinks is suboptimal for citations. For example, the "share" link automatically appends the user ID for tracking purposes (to award badges to those whose shared links are widely used), which is probably not what someone looking for a citable link wants. There are permalinks for every revision, but they are a little less obvious to find, and they link just to the specific text in question (for example, https://mathoverflow.net/revisions/153731/1 requires several clicks before you can find any of the answers). It would be great to have a more flexible system that archives snapshots at a given time while still making it easy to see any related material and to see any later updates. For example, an archive link could show the whole question/answer page as it looked at the time that the link was made, with a prominent option to see what happened at other times. Adding the full time to the permalink could be cumbersome, but all that's needed is a numbering of the changes made to each page. Getting this working is not conceptually difficult, but it could be time-consuming, and there are a lot of irritating aspects. (For example, how much do we care about things like chronology of votes? Is it possible that someone citing an MO question would refer to something like "the most upvoted answer" in a way that couldn't be understood later without knowing the voting history and when the assertion was made? On the other hand, changing the recommended permalink every time someone votes seems silly.) 5. If all this is taken care of, then we could largely solve the citation problem by adding a "citable link" button, maybe even replacing the current "share". It would give an archival permalink together with a recommended citation format. (Of course there might still be psychological issues regarding which sorts of citations are considered acceptable, but these steps would accomplish about as much as I think MO reasonably could in the near future.) Monthly or weekly static versions would be much easier, and they would certainly help solve this problem, but the limited time resolution would be a problem. Ideally, a "citable link" button should immediately produce an archival link that reflects what you see now, rather than an approximation that may be a week or more off. • Are there actual cases of where cited MO posts were deleted? -- I'd doubt there are many. – Stefan Kohl Mod Feb 2 '14 at 10:50 • I like the one month window in which someone can delete and after which things become permanent. @StefanKohl: There was an excellent mathematician on MO, whose answers now appear as User631, who posted many beautiful and informative answers, and for some reason deleted many of these. I think these are a real loss for MO, but maybe this situation is uncommon. Feb 2 '14 at 12:32 • Thanks Henry! Just a few complementary links: The snapshot archiving mechanism in 4 was suggested here and a few flaws were pointed out. This feature request is about making revision links more accessible; you should consider adding specific changes you feel would be useful there. Feb 2 '14 at 13:26 • Regarding 1, this is really the topic of the long-term archiving thread and should be discussed there. However, the fact that MO is constantly updating makes it difficult to ingest (their term) by archival organizations; they would still need a buffer like archives.mathoverflow.net to do it properly (though the buffer doesn't need to be publicly visible). Feb 2 '14 at 13:38 • The Internet Archive has a mechanism for saving pages on demand, e.g. this page right now. The question is whether we could do better. Feb 2 '14 at 13:39 • Regarding "MO is trying to become more citable by pretending to be something it's not." This is indeed a touchy issue. Unfortunately, academic citations are somewhat rigid. I'm afraid anything we could do to make MO more amenable to the format can be seen as "[MO] pretending to be something it's not." Since you're not arguing that MO shouldn't try to be more amenable to academic citations, it would be more useful to break down what aspects you see as more contentious than others. Feb 2 '14 at 14:30 • By the way, in terms of permalinks to snapshots, I was imagining that everything would be archived (not just on demand), but a permalink like archives.mathoverflow.net/x/y would link to a view of page x at stage y (i.e., after the y-th significant change). So the library would archive a time-stamped stream of posts and edits, and then a front end website would turn a permalink into a web page at a given time. Feb 2 '14 at 14:50 • In terms of being amenable to citations, long-term archiving is the only substantive issue I see, although popularizing a recommended format could also help. In terms of looking like a journal, the arXiv could be a useful comparison. It's inherently more journal-like than MO, but papers are still cited a little differently to make their unrefereed preprint status clear. This isn't a problem if you don't care about this distinction (it's still just as good a citation in terms of awarding credit or providing access), but the small difference in format makes people who do care more comfortable. Feb 2 '14 at 15:03 • For example, I'd feel uncomfortable citing a typical MO contribution in a way that looked like a journal paper, and I'd be embarrassed to be cited that way. I'd rather cite it in a way that nobody would interpret as saying "compared with all the other valuable mathematics on the web, this is particularly journal-like." I'd be fine with something like "Name, Mathoverflow answer N, date, permalink", but "Name, Title, Annals of Mathoverflow Volume (Year), No. Issue, DOI" would make me unhappy (to make up an extreme case). Feb 2 '14 at 15:17 • I'm not sure what to think about the DOI issue (instead of generic permalinks). I think right now many mathematicians would interpret it as imitating journals, but that might change over time as they see DOIs in more places, so maybe it's not worth worrying about. Feb 2 '14 at 15:20 • Thanks for the clarifications Henry. I think your example with the Annals of MO clarifies the issue very well and I completely agree. (Maybe it's worth inserting in an edit?) I think mentioning volume & issue numbers, ISSN, DOIs in the question was misleading. I was trying to illustrate the broad range of possibilities but I ended up focusing on one in depth rather than a handful in breadth. Hindsight is 20/20... Feb 2 '14 at 15:37 • Regarding DOIs, they are intended for a lot of purposes and, in many ways, can be thought of as "permalinks 2.0." Like many things, they get associated to where they are most visible: journals & books. I don't think we need to worry too much about it. For what it's worth, if CrossRef classifies the MO archives as a journal, they will charge us prohibitive fees instead of much cheaper rates for other content. It's good to keep in mind some possibilities are incompatible with each other. Feb 2 '14 at 15:48 Disclaimer. It is really helpful to make concrete proposals in order to advance the discussion. This is a tentative proposal for an archival mechanism that tries to address some issues brought up by Henry Cohn and others. I'm sure there are plenty of other issues we haven't discussed yet. So please don't think that this is an official solution, the discussion is just starting! What gets archived? Archival units are individual revisions of individual posts in a manner similar to what is found here but with some additional functionality such as direct links to the "live" post on MathOverflow, links to other revisions of the post, links to the matching archived question or matching archived answers depending on whether the post is an answer or a question, and other information described below. Rationale: This is preferable to archiving page snapshots since it focuses on the actual information cited. Imagine a situation where user A gives a brilliant answer to user B's excellent question where user C had previously given an answer with a subtle mistake. After realizing their error, user C promptly retracts their answer by deleting it. However, user B takes a snapshot of user A's answer before user C's deletion and proceeds to cite it in an influential paper. That snapshot keeps being cited by other authors and user C's subtle mistake suddenly becomes a very embarrassing one. When is it archived? Archiving would be done after a delay of$N$days. Archival permalinks links would still be available right after posting but the archived version would be marked as pending. • Posts that are deleted during the waiting period would be discarded from the archive. • Posts that are put on hold/migrated during the waiting period would have their waiting period extended by$M$days. Rationale: The archival delay is to avoid archiving spam and other inappropriate content and to give users a reasonable chance at early retraction. Reasonable delays are$N = 10$and$M = 30\$.

What about deletions? Once archived, a post is never deleted from the archive. However, archived posts can be annotated after some trigger events.

• When a post is deleted on MathOverflow, all of its archived revisions are marked as retracted in a clearly visible manner.
• When a post is closed or locked on MathOverflow, some appropriate mark may be added near the link to the post on MathOverflow but the archived version would not undergo major changes.

Rationale: Archiving is fundamentally a one-way process: archived contents should never be deleted (unless there was an error in the archiving process itself). On the other hand, it is necessary to respect the users and the community. Annotations are the best we can do in this context.

Deleting, closing and locking are very different processes. Deleting is a clear retraction and needs to be clearly annotated. Closing and locking have to do with the functionality of the site and don't seem to need much annotation.

What isn't archived? Post comments are never archived, neither are vote counts, view counts and similar ancillary post data.

Rationale: These primarily have to do with the functionality of the site. It is true that some of this content is worth archiving but it seems too difficult to sort. If a comment really does contribute significantly to the archival value of a post, it should be integrated into that post or promoted to a separate post.

How does this integrate with MathOverflow? A new citation button would be added below each post, separate from the share button. The button would give the archive permalink of the current version of the post (even if archiving is still pending) as well as properly formatted BibTeX and AMSref entries.

Rationale: Sharing and citing are fundamentally different; it is confusing to merge the two into one. The share button will still exist and it will give a link to the actual MathOverflow post. The cite button needs to be separate since it links to the archived MathOverflow post.

If archiving is still pending, it's the user's responsibility to ensure that the archiving process has completed before using the link in a citation. Since there is usually a very long delay before citations are published, this shouldn't be a problem in practice. In cases where the link needs to be immediately usable, the sharing link is probably more appropriate.

• One thing I don't see clearly is a need to give a single link. A citation of the form [User](1). [Answer Title](2), MathOverflow ([accessed Date](3)). can give a link to the 'live' question in (2) and to the 'frozen' archive version in (3), and I think it's fairly clear what the role of each link is. Feb 2 '14 at 21:25
• @EmilioPisanty: The problem is that link (3) doesn't exist yet. You are free to cite MO whichever way you want! Feb 2 '14 at 22:58
• Well, for now link (2) can be to the post and link (3) to the revision history. Having such a scheme as a recommended citation now can help settle the workflow into something that can help melt the resistance to change, and pave the way for when an archive is set up. Feb 2 '14 at 23:07
• @EmilioPisanty: Yes, that is pretty much the way citations ought to be done at this time (but they aren't). Feb 2 '14 at 23:13
• @EmilioPisanty: that scheme is great for on-screen reading, but not so suitable for printed (or intended-to-be-printed) documents. Mar 2 '14 at 22:21
1. Database, which forms all the MO content, should be changed in a way that any posting, altering, deletion, interface change, etc. deletes nothing from the database, but just adds more entries. This is acute, if not already in place.

2. User interface to MO pages should be changed so that it includes something like a "mini way back machine" element. Uncovering and fiddling with this visual element should bring the visual state of the page to that of the time selected. This could be done in short/midterm.

3. A separate visual element could show the historical status of the page. For example, "locked in 1970's", "new", "building up during decades/centuries", "activity dying up", "claims backing up/recent deletion" so that a visitor can be informed if much/anything can be gained from looking in the past. This is not acute and could be implemented over time. This visual element should make visitors aware if there have been major refubrications of the page content.

4. There could be a special string added to URI of string that forms the web adress in the way the http get method works. For example, mathoverflow.org/example-page?date=20140207&other=value could be direct link to a the page mathoverflow.org/example-page?other=value as seen on February 7th, 2014.

5. Citations could follow the format proposed by Emilio Pisanty in a comment made on February 2nd at 21:25, to the another answer of this question, or be supplemented with the URI from my point no. 4.

6. Scholars following a citation to MO should be aware how MO works. This awareness will be built over time, so that following a decades old citation would be dealt appropriately by researchers, say 20-50 years in the future.

7. Archiving as done now, which is public database dump + free current MO source code, would stay enough, in case of a catastrophe.

• 1. This is in place. In fact deleted content is always visible to users with 10k points. 2/3. Is this or this what you have in mind? Feb 7 '14 at 18:15
• 5/6. I'm not sure what you mean here. We have no way of controlling how users cite MO. Feb 7 '14 at 18:16
• 7. Some of that is not possible but our contract with Stack Exchange includes provisions to get around this. Feb 7 '14 at 18:17
• @François G. Dorais In no. 2 I was thinking about a visual interface like that on top of www.archive.org, or slider of some kind. One of your links returns page not found, the other gives all the revisions but not how the page looked like at the time of revision. This is pointed out in the question. On the other hand, the question does speak about how peeple cite MO and about how a proposed solution would help them in the future. This is why I included no. 5 and 6 in the answer. They show how the intent behind the question (easy citing to historic MO state) would be materialized. Feb 7 '14 at 18:40
• The proposal to have a URL that displays a page as it appeared at a certain date is actually pretty reasonable; this could be written on top of the public data dump (with the exception of some deleted content, which is also removed from dumps, I think). Stripped down to a minimal idea this might deserve a separate thread. Feb 9 '14 at 1:19

This problem is not specific to MathOverflow, but concerns citation of Internet resources in general. Thus, it requires a general solution. Such a solution exists: several non-profit organizations offer archiving of web content for citation in scholarly literature. Examples are perma.cc and WebCite. Wikipedia has a page with a partial list of archiving services, where more information can be found.