Trackbacks are no longer being generated for the arXiv

Question

(Although this is a bug report, I or Anton are likely the people who need to fix it; this post is a bug tracking item, not necessarily a plea for help :-)

We previously been scraping the front page of MathOverflow for questions that linked to the arXiv, using a long list of regular expressions to try to capture everything. For each of these links, we generated a 'trackback' to the arXiv, and they then added a link back to the MathOverflow post citing that article.

We'd also periodically run a different script against a database dump, to catch any links that we failed to notice by scraping.

We're no longer generating any trackbacks; c.f. http://arxiv.org/tb/recent, set the list to show the 100 most recent trackbacks, and notice that they stop after last Monday, the migration.

Presumably this is because our screen scraping code no longer matches up with the HTML that MO 2.0 produces. An easy fix will be to tweak that code. A harder but better fix would be to use the Stack Exchange public API to achieve the same result.

Answer 1

I think that trackback notification should be working again. Can someone find a suitable arXiv link to add, then wait an hour and check http://arxiv.org/tb/recent to see if it comes through?

I haven't yet done anything about arXiv links posted between the migration and now, but that should be relatively easy now.

Answer 2

I haven't fixed this yet. This is just an update.

Unfortunately, code I'd written previously that interacted with the Stack Exchange API used API v1.1. They've since upgraded to v2.1 of the API, and v1.1 won't be available for MathOverflow. This reduces my personal enthusiasm for writing this code (because it's more work than I'd expected), but shouldn't be an obstacle for anyone else.

Here's the basic idea for a program that generates trackbacks.

---

There are two essentially independent programs, that need to be run on a recurring basis (at least daily, maybe as often as hourly). They interact only via an append-only text file (or something equivalent). Each has a tiny amount of 'internal state' that needs to be preserved between runs of the program, namely the time at which the program was last run.

Program 1 is the "scraper", responsible for finding new MathOverflow content that contains arXiv links. It should request, via the SE API, all new content (i.e. new posts, new comments, as well as edited posts) between the last run of the scraper and the current time. For each such piece of content, it should run a certain set of regex's (there are important details here; we should reuse the existing well-tested set) on the content. These regex's will output any arXiv ids found in the content (possibly several). For each id found, the scraper should append to its log 4 pieces of data:

1. the current time
2. the arXiv id
3. the URL of the content (it doesn't really matter if this is the URL of the question or a more specific URL)
4. the title of parent question

Program 2 is the "poster", responsible for sending trackbacks. On each run, it should inspect the output log from the scraper, and take all entries since the last run of the poster. For each entry, it should perform a GET request to the arXiv, according to this example:

http://arxiv.org/trackback/1307.1702?blog_name=MathOverflow&title=A%20Question&url=http://mathoverflow.net/questions/123

---

That's pretty much it!

A few comments:

1. The "poster" program will eventually gain extra duties; for example the selected papers network would like us to notify them too.
2. We need to notify the arXiv administrators of the IP address of the machine doing the posting, so they can whitelist submissions.
3. It would be nice to be able to "re-run" the scraper over old content, without generating duplicate trackbacks, in case we miss something, update regex's, etc. This might be as simple as resetting the 'last run' timestamp, then deduplicating the log file.