Archiving of MathOverflow threads

Question

We've had some discussion of adding an archiving feature to MathOverflow on the private moderators list, and I thought I should report on the current situation. (You might also read earlier threads Long-term archiving of MathOverflow, Should top mathoverflow threads get DOIs and be permanently archived?, and Improving citations of MathOverflow posts.)

Proposal: We have a server that responds to URLs of the form http://archive.mathoverflow.net/questions/QQQQ/TTTT, where QQQQ is a question number, and TTTT is a timestamp, returning to the best of its ability a snapshot of question QQQQ at the moment TTTT, or the nearest moment thereafter for which a snapshot is available.

Further notes:

1. If the timestamp TTTT is not available, the server should redirect to the earliest available later timestamp SSSS (possibly the current moment), so that the displayed URL once the page renders accurately reflects what is shown.
2. A URL of the form http://archive.mathoverflow.net/questions/QQQQ (i.e. no requested timestamp) should redirect to http://archive.mathoverflow.net/questions/QQQQ/TTTT, where TTTT is the current moment.
3. Pages returned should be immutable, and if a timestamp has been delivered, it should always be available in the future.
4. To cite a page, a user could request a snapshot using the URL http://archive.mathoverflow.net/questions/QQQQ, and then use the URL they are redirected to.
5. Archiving of all of MathOverflow's content can be arranged merely by having a script periodically retrieve new snapshots. (Obviously, there would be some rate-limiting involved here.)

What does everyone think? Would this be useful? What should be changed?

Answer 1

There is a very basic implementation of the above suggestion, available for example at http://ec2.mathoverflow.org:8090/questions/137287/20150917/0626. Currently this just returns a JSON representation of the page, not something particularly human-readable. Except for that glaring problem, it functions as described above.

The project (written in Scala) is available at github, or you might want to jump straight to the actual code, skipping the scaffolding.

In order to make this usable we'd have to do the following:

1. Make the output human readable. In fact, this can be done "client side" purely via templating and javascript. Already, it displays the JSON output by pushing it through a trivial template. See github or ask me if you want to help.
2. Move the underlying SQL database to also run on the same server, so it's less incredibly slow than it is right now.
3. Ensure that requests to the Stack Exchange API respect their quota and throttling limits, and return error messages when it is not possible to provide a snapshot because of these limits.
4. Perform lots of testing!
5. Possibly have the entire (short) code rewritten by someone who knows how to program.
6. Move everything over to the final intended URL.
7. Add a snapshot link on all the MathOverflow pages.

Answer 2

One foreseeable problem is the fact that data forms a web.

Namely, it is not rare to see answers which give out some substantial information, and point out to other related questions and answers. This means that in order to preserve the entire data structure, it might be essential to go by recursion through links in the question, answers or even comments, and preserve them as well (or link to previously preserved copies).

It's not entirely unsolvable, but you have to decide how deep into the web you want to go, and how to decide when it's reasonable to link to an existing archived copy (comparing activity dates is one way), and when to archive a new copy.

Answer 3

https://web.archive.org already implemented something very close to yours.

Check: snapshots of main page

Answer 4

Here are some additional features, which may become less useful than at first glance.

a. with every request of time TTTT, a return of a header including TTTT, actual timestamp SSSS, and hovertext or plaintext resolving either to Gregorian date, or a link to a timestamp-human-readable date converter. In the case SSSS resolves to later than TTTT, use a different color or notation to indicate TTTT was too early/invalid.

b. along with SSSS, a link to a version-crawler for that question. For a small number of versions of the same item, this could be back and forward buttons. For a larger number, direct to a page which allows somewhat random access of the version (e.g. as the Wayback Machine presents, or even less fancy with a field where a user can type in just the time stamp or just the version number).

It may turn out that certain questions/responses have changed so much that a single URL may not suffice. In this case, a policy should be formed in which an additional field VVVV may be set, which indicates for example that version 1111 of the question is substantially thematically different from version 2222, (perhaps because a finiteness assumption was added). The policy should declare who can add/alter such a field, and for what reason. However, the policy should also state that it is formative and likely to be changed while the policy-makers figure out what is really going on. This field can also be multipurpose: it can be used to redirect to a set of answers, or some other multisubset of the entity currently called a question.

I would like to know if it has been determined what ancillary information will be stored besides the question and answers. Will all (undeleted) comments be preserved? I sometimes found the Related and Linked questions on the same page to be useful; will a snapshot of those be stored along with the question? The basic archive of pages with links will be useful: will we help the archivers with special indices of users, questions, answers, and links? Is it clear how searchable this archive will be made?

Gerhard "Signatures Should Be Forever, Maybe" Paseman, 2015.09.17