Long-term archiving of MathOverflow

Question

Suppose 'the worst' happens. Stack Exchange folds (commerical entities have been known to), and at the same time the moderators lose all interest and/or decide the despise mathematics. Where would this leave us?

People are citing MathOverflow in many published papers, so it seems we have some obligation to attempt to ensure that the content remains accessible forever.

Public dumps exist (although they are up to 3 months), and it's likely that someone would have retained a copy, even if the site they are currently served from became inaccessible. From this dump it would be relatively easy to create a read-only website containing the contents of MO, and if one could obtain ownership of the domain name (which is owned by the MathOverflow corporation, not StackExchange) it would be relatively easy to have all the links from the literature resolve to the correct (but read-only) places.

Is there something else we can do to ensure the long-term archiving of MathOverflow?

Are there libraries that would be interested? The Internet archive? The arXiv itself?

It would be relatively easy to produce a giant PDF snapshot of MO (e.g. I know how to script creating PDFs from webpages, including rendered MathJax). The question becomes whether anyone really reliable would want to take them.

Answer 1

Contrary to some other answers, I think the MathOverflow archive should be as textually literal as possible, though there is no need to preserve fancy formatting, logos or any bells and whistles that are meant to make MathOverflow more pleasant to use rather than to read. I would be very happy with something very plain (such as what you see below the separation line when looking at the revision history of a post).

I don't think there is any need to convert to PDF or any other format. The reasons for this are multiple. Markdown (the format MO posts are currently composed and stored in) is very compact and simple. It's perfectly readable even without markdown processing, so little is lost even if markdown goes out of fashion. Storing the math as LaTeX code is also generally better than compiling it and preserving the output. Alternately, LaTeX code could be converted to something which is standardized (such as MathML, for example) but LaTeX is pretty much a standard at this point. The only thing that might require maintenance is upgrading the character encoding from UTF to whatever comes next but that won't be a problem at all if and when it happens.

I think the archive should include comments since many do contain much substance and some posts don't make sense without them. It's true that a lot of comments are just chatter but I don't see any good reason not to preserve those too and I can't think of a good way to sift through comments. Besides, knowing that their comments will be preserved forever might make users think a lot more before posting something inappropriate.

One contentious issue is how to handle deleted material. Archives typically work only one way: you can put stuff in but you can never remove anything. I can think of two ways of handling this: (a) make deleted material unavailable but still preserved, (b) clearly mark deleted material as redacted. Maybe there is another way? (Note that the current site uses (a) for <10k users and (b) for 10k+ users.)

Answer 2

The Keepers Registry currently has 8 participating archiving agencies:

• British Library
• CLOCKSS Archive
• Global LOCKSS Network
• HathiTrust
• Portico
• Archaeology Data Service
• National Science Library, Chinese Academy of Sciences
• e-Depot

A quick look at their descriptions suggests that some may be interested in archiving MathOverflow.

The Center for Research Libraries appears to be the main international entity for assessing and certifying the trustworthiness and reliability of major archiving agencies. Some reports can be found here.

---

I looked in more detail at Portico since it has a very nice description of its services. I think they would classify MathOverflow as an e-journal. Their aim is to provide digital preservation, not a backup system nor a redundant access point. So this service would be suitable for a separate read-only "e-journal" archive of MathOverflow content, not for a site mirror (even if read-only) nor for permanent storage of MathOverflow data dumps. Their publication agreement can be seen here and the terms look very reasonable to me. Their pricing model is also very reasonable: so long as MathOverflow does not make \$250k in revenues, our annual contribution would be \$250.

---

It appears that HathiTrust and e-Depot currently only accept "monolithic" documents such as books and cannot accept periodicals or similar objects. Archaeology Data Service only accepts documents relating to archaeology and history.

---

The Global LOCKSS Network has an interesting framework.

This section describes how a LOCKSS Box works. Specifically, a LOCKSS Box performs five main functions:

• It ingests content from target websites using a web crawler similar to those used by search engines.

• It preserves content by continually comparing the content it has collected with the same content collected by other LOCKSS Boxes, and repairing any differences.

• It delivers authoritative content to readers by acting as a web proxy, cache or via Metadata resolvers when the publisher’s website is not available.

• It provides management through a web interface that allows librarians to select new content for preservation, monitor the content being preserved and control access to the preserved content.

• It dynamically migrates content to new formats as needed for display.

The system seems rather flexible but I couldn't determine what content formats they could support in their archival units. To use that option, we would need to maintain our own archive (distributing content in a format that LOCKSS can support) and the participating libraries would each have a permanent and authoritative cache of our archive in their LOCKSS Box.

---

The CLOCKSS Archive has a basic mechanism similar to the Global LOCKSS Network but the goal is not to provide continued access to the resource like some of the above. Instead, CLOCKSS is designed to publicly release an archive only after a trigger event. The Title No Longer Offered event corresponds to the type of failure that Scott mentioned in the question.

---

I haven't yet found the relevant information for the British Library and the National Science Library of the Chinese Academy of Sciences.

From what I can tell so far, Portico, the Global LOCKSS Network and the CLOCKSS Archive all offer potentially useful services for the purpose of archiving MathOverflow. The Global LOCKSS Network appears to be the most flexible but it does not offer as much security as Portico and CLOCKSS. CLOCKSS offers fewer services than Portico, but combining CLOCKSS with the Global LOCKSS Network is a very competitive alternative to Portico.

Answer 3

Preamble. When thinking about situations like this, it's tempting to fall back on traditional ideas. I realized earlier today that everything suggested so far would also work if MathOverflow was a giant stack of loose leaf paper instead. Though this is a characteristic of excellent ideas that are both simple and versatile, it does suggest that we might not be considering the full extent of our options. I want to encourage users to think outside the box while considering this issue. The suggestion below is a small step in this direction and I my hope is that our collective creativity will lead to a solution which is as innovative and practical as MathOverflow itself.

---

The idea that I will explain is motivated by several issues. One issue is the problem of deleted content that I raised in another answer. Similar issues are brought up by edits, revisions, later answers, etc. Another is that, although I expressed doubt about this, the fact that some users would prefer that only some of MathOverflow's content should be preserved suggests that this is a desirable feature. Finally, I realized during in my research on archiving agencies that they could find the total volume and regular flow of MathOverflow problematic.

This suggests that MathOverflow content could be archived only on demand. That is, there would be a MathOverflow Archive which stores a static copy of a question or answer in its current state when, and only when, a user requests it. Thus, if you want to cite something on MathOverflow, you make such a request for the content you want to cite, that content gets archived and your citation will always point to a static copy of that content.

This basically solves the deletion and editing issue since I doubt anyone would make such a request for deleted content and static copies remain as they are even if the original gets edited or modified in other ways. This also solves the selectivity problem since (presumably) only good content would be requested in this way. Finally, this also solves the volume problem since much less content would be archived by request.

This kind of archiving mechanism might also serve as support for other possibly desirable features such as an option to have the content peer reviewed (preferably in an open way using the Selected Papers Network, for example) and, if the content is approved, it would get stored in a designated "peer-reviewed" part of the MathOverflow Archive (along with the referee report). Such enhancements would definitely give more credibility to MathOverflow as a micropublishing platform.

---

Scott pointed out that a defect of this approach is that few would bother making such requests. A potential solution is to have posts automatically archived as soon as someone presses the cite button to get a bibtex/amsref entry and have the urls there point to the proper place on archive.mathoverflow.net instead of the original.

Answer 4

One interest of long term archiving is that MathOverflow could then be issued an ISSN. One could make the case that MathOverflow is already eligible for an ISSN, but a long-term archive would make an even stronger case. Here is the relevant list of eligibility criteria from the ISSN Manual.

0.3.2 Ongoing integrating resources

Ongoing integrating resources are resources that are updated over time and with no predetermined conclusion, for which the updates are integrated into the resources and do not remain discrete.

ISSN are assigned to ongoing integrating resources which fulfill all the inclusion and exclusion criteria listed hereafter. National Centres can decide to exclude ephemeral ongoing integrating resources or ongoing integrating resources of purely local interest from systematic ISSN assignment (see section 0.7)

Meeting only one of the criteria is not sufficient for ISSN assignment. These criteria apply to all the categories of ongoing integrating resources, whether print or electronic: databases, websites, wikis, print loose-leaf services, etc. Although blogs are considered to be serials, these same criteria should be applied to blogs.

Note: If a part of an ongoing integrating resource (part of website, for instance) is eligible for ISSN assignment, e.g., a newsletter that is part of the website is assigned an ISSN, this does not mean necessarily that the whole website is eligible for ISSN assignment.

0.3.2.1 Inclusion criteria

1. There is editorial content (i.e., the resource mostly consists of written, textual content, and there is evidence of editorial or journalistic treatment);

2. There is identified editorial responsibility (i.e., a statement indicating the name of the publisher / producer, and at least the country of publication). Generally, editorial responsibility will consist of more than one individual;

3. There is a consistent title (i.e., a title which remains consistent when the resource is updated) and the title is prominently visible on the resource;

4. There is a valid URL (i.e., a URL leading to the actual resource);

5. The resource has subject-related content or has identified subject-related audience.

0.3.2.2 Exclusion criteria

1. Personal resources (personal web sites and web pages, online diaries);

2. Resources focusing on a company, a product, an institution or organization (advertising and promotional web sites, commercial web sites, product information web sites, company and institutional web sites, web sites of organizations);

3. Web sites consisting only of links;

4. Ephemeral resources (i.e., resources known to be ephemeral).

Answer 5

The decision to archive must be made with care. In addition to the technical issues of preserving rendering of typeset math, static and dynamic images, and various links, there are the social issues of how to keep and arrange the comments if any, and how to include chat room transcripts and even whether to post annotate to explain cultural and historical references. ("What's +1 Daddy? Did mathematicians use only one finger instead of five?")

I think a partial solution may be appropriate, which is to have a static rendering of some parts of the content, with references to the dynamic database. That way a curated and edited archivable version can be created and preserve something until all the other bugs are ironed out of making a full archive. The curated versioncan also be made useful independent of a full archive with some care.