13
$\begingroup$

We've had some discussion of adding an archiving feature to MathOverflow on the private moderators list, and I thought I should report on the current situation. (You might also read earlier threads Long-term archiving of MathOverflow, Should top mathoverflow threads get DOIs and be permanently archived?, and Improving citations of MathOverflow posts.)

Proposal: We have a server that responds to URLs of the form http://archive.mathoverflow.net/questions/QQQQ/TTTT, where QQQQ is a question number, and TTTT is a timestamp, returning to the best of its ability a snapshot of question QQQQ at the moment TTTT, or the nearest moment thereafter for which a snapshot is available.

Further notes:

  1. If the timestamp TTTT is not available, the server should redirect to the earliest available later timestamp SSSS (possibly the current moment), so that the displayed URL once the page renders accurately reflects what is shown.
  2. A URL of the form http://archive.mathoverflow.net/questions/QQQQ (i.e. no requested timestamp) should redirect to http://archive.mathoverflow.net/questions/QQQQ/TTTT, where TTTT is the current moment.
  3. Pages returned should be immutable, and if a timestamp has been delivered, it should always be available in the future.
  4. To cite a page, a user could request a snapshot using the URL http://archive.mathoverflow.net/questions/QQQQ, and then use the URL they are redirected to.
  5. Archiving of all of MathOverflow's content can be arranged merely by having a script periodically retrieve new snapshots. (Obviously, there would be some rate-limiting involved here.)

What does everyone think? Would this be useful? What should be changed?

$\endgroup$
  • 1
    $\begingroup$ Is the idea more to have the ability to link to specific timestamps, or to have an archive independent of SE? For the former, it might be possible to convince SE to implement the ability to view and link questions with a specific timestamp. $\endgroup$ – Mad Scientist Sep 17 '15 at 7:29
  • $\begingroup$ It's true that this would be much more easily implemented within Stack Exchange. This started as a plan for external archiving. (It's really only a component of an external archiver --- you'd also have to connect it to some institutional repository, e.g. LOCKSS or some library repository.) $\endgroup$ – Scott Morrison Sep 17 '15 at 9:41
  • $\begingroup$ Wikipedia, for example, currently archives every edit made to every page, incorporating a timestamp and optional "edit description" comment. This allows for easy reversion of pages to previous states, as well as comparison between two versions, among other things. Something like this could be even more valuable on a site like Mathoverflow, where content is liable to evolve quickly over a short period and then change very rarely. Would some variant of this be feasible as an alternative to more sporadic archival by a "crawler" script? $\endgroup$ – Robin Saunders Sep 21 '15 at 18:43
  • $\begingroup$ @RobinSaunders, I guess I didn't say this, but a criteria for the above proposal was that it could be implemented without needing cooperation from Stack Exchange. (Not because I expected them to be unhelpful, just to simplify the process and ensure that the goal was achievable.) $\endgroup$ – Scott Morrison Sep 22 '15 at 0:38
  • $\begingroup$ Ah, sorry - I skimread the previous comments. Maybe if the crawler was "active" enough and only looked at recently-updated pages then it could approximate the functionality I described? $\endgroup$ – Robin Saunders Sep 23 '15 at 7:51
6
$\begingroup$

There is a very basic implementation of the above suggestion, available for example at http://ec2.mathoverflow.org:8090/questions/137287/20150917/0626. Currently this just returns a JSON representation of the page, not something particularly human-readable. Except for that glaring problem, it functions as described above.

The project (written in Scala) is available at github, or you might want to jump straight to the actual code, skipping the scaffolding.

In order to make this usable we'd have to do the following:

  1. Make the output human readable. In fact, this can be done "client side" purely via templating and javascript. Already, it displays the JSON output by pushing it through a trivial template. See github or ask me if you want to help.
  2. Move the underlying SQL database to also run on the same server, so it's less incredibly slow than it is right now.
  3. Ensure that requests to the Stack Exchange API respect their quota and throttling limits, and return error messages when it is not possible to provide a snapshot because of these limits.
  4. Perform lots of testing!
  5. Possibly have the entire (short) code rewritten by someone who knows how to program.
  6. Move everything over to the final intended URL.
  7. Add a snapshot link on all the MathOverflow pages.
$\endgroup$
  • $\begingroup$ Wouldn't it be significantly easier and very close to this to run read only mirror of MO on available open source QA software supporting SE import (e.g. OSQA)? Just disable login/register. This will keep the latest snapshot, not all. Revisions of question/answers will be visible as on MO if the implementation allows it. $\endgroup$ – joro Sep 18 '15 at 5:42
  • $\begingroup$ Another option for server side display is to reuse code from existing open source QA software (possibly supporting SE import). $\endgroup$ – joro Sep 18 '15 at 6:18
  • $\begingroup$ I think it's appealing to have snapshots from fixed points in the last, so people can cite the page as it was at a particular time. $\endgroup$ – Scott Morrison Sep 18 '15 at 10:18
  • $\begingroup$ If you know a quick way to render SE API JSON to something human readable, let me know! $\endgroup$ – Scott Morrison Sep 18 '15 at 10:19
  • $\begingroup$ Do you mean to render it in the client with the help of javascript? If you can produce database in SE export format, I believe OSQA can return HTML/CSS/js. OSQA is here osqa.net $\endgroup$ – joro Sep 18 '15 at 10:31
  • $\begingroup$ What is wrong with crawling the HTML (like archive.org), not JSON? $\endgroup$ – joro Sep 18 '15 at 16:25
  • 1
    $\begingroup$ Crawling, and reproducing, HTML is pretty hard. Do you download all the CSS and image files as well, or just hope that they stay static? Do you try to render the page to PDF? It seems hard to produce a reliable archive, that for example could be stored by a library or other institutional repository. $\endgroup$ – Scott Morrison Sep 20 '15 at 1:48
4
$\begingroup$

One foreseeable problem is the fact that data forms a web.

Namely, it is not rare to see answers which give out some substantial information, and point out to other related questions and answers. This means that in order to preserve the entire data structure, it might be essential to go by recursion through links in the question, answers or even comments, and preserve them as well (or link to previously preserved copies).

It's not entirely unsolvable, but you have to decide how deep into the web you want to go, and how to decide when it's reasonable to link to an existing archived copy (comparing activity dates is one way), and when to archive a new copy.

$\endgroup$
2
$\begingroup$

https://web.archive.org already implemented something very close to yours.

Check: snapshots of main page

$\endgroup$
0
$\begingroup$

Here are some additional features, which may become less useful than at first glance.

a. with every request of time TTTT, a return of a header including TTTT, actual timestamp SSSS, and hovertext or plaintext resolving either to Gregorian date, or a link to a timestamp-human-readable date converter. In the case SSSS resolves to later than TTTT, use a different color or notation to indicate TTTT was too early/invalid.

b. along with SSSS, a link to a version-crawler for that question. For a small number of versions of the same item, this could be back and forward buttons. For a larger number, direct to a page which allows somewhat random access of the version (e.g. as the Wayback Machine presents, or even less fancy with a field where a user can type in just the time stamp or just the version number).

It may turn out that certain questions/responses have changed so much that a single URL may not suffice. In this case, a policy should be formed in which an additional field VVVV may be set, which indicates for example that version 1111 of the question is substantially thematically different from version 2222, (perhaps because a finiteness assumption was added). The policy should declare who can add/alter such a field, and for what reason. However, the policy should also state that it is formative and likely to be changed while the policy-makers figure out what is really going on. This field can also be multipurpose: it can be used to redirect to a set of answers, or some other multisubset of the entity currently called a question.

I would like to know if it has been determined what ancillary information will be stored besides the question and answers. Will all (undeleted) comments be preserved? I sometimes found the Related and Linked questions on the same page to be useful; will a snapshot of those be stored along with the question? The basic archive of pages with links will be useful: will we help the archivers with special indices of users, questions, answers, and links? Is it clear how searchable this archive will be made?

Gerhard "Signatures Should Be Forever, Maybe" Paseman, 2015.09.17

$\endgroup$
  • 1
    $\begingroup$ The sample pages at present state clearly at the top, e.g. as it appeared at 0626 on 20150917 (UTC). I don't think it's a good idea for the text of the displayed page to say anything about the requested date, because it's essential for archiving purposes that these pages are immutable, and can be cached by other parties. $\endgroup$ – Scott Morrison Sep 17 '15 at 23:22
  • $\begingroup$ The version crawler is a good idea, but again I'm hesitant to include it as part of the returned snapshots. Version crawling 'by hand' is possible because the URLs contain the date, and you can directly modify this. Further, this sounds like more complicated feature (it might double the size of the current codebase!), so should come far down on the todo list. $\endgroup$ – Scott Morrison Sep 17 '15 at 23:23
  • 1
    $\begingroup$ The intention is to store everything it's possible to retrieve by a Stack Exchange API call, and nothing more. I built what I think is the most useful 'filter', and you can see exactly what would be saved by looking through the JSON output displayed on the sample server. Suggestions to modify that filter are welcome. $\endgroup$ – Scott Morrison Sep 17 '15 at 23:25

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .