(Although this is a bug report, I or Anton are likely the people who need to fix it; this post is a bug tracking item, not necessarily a plea for help :-)

We previously been scraping the front page of MathOverflow for questions that linked to the arXiv, using a long list of regular expressions to try to capture everything. For each of these links, we generated a 'trackback' to the arXiv, and they then added a link back to the MathOverflow post citing that article.

We'd also periodically run a different script against a database dump, to catch any links that we failed to notice by scraping.

We're no longer generating any trackbacks; c.f. http://arxiv.org/tb/recent, set the list to show the 100 most recent trackbacks, and notice that they stop after last Monday, the migration.

Presumably this is because our screen scraping code no longer matches up with the HTML that MO 2.0 produces. An easy fix will be to tweak that code. A harder but better fix would be to use the Stack Exchange public API to achieve the same result.

  • 4
    $\begingroup$ If anyone would like to jump in and write something that uses the public API, please let me know and I can point you in the right direction. Our (abysmal) shell scripts for doing the scraping are in a mercurial repository, c.f. code.google.com/p/mathoverflow/source/browse/#hg%2Ftrackbacks $\endgroup$ Commented Jun 28, 2013 at 23:13
  • $\begingroup$ I think the MO API isn't enabled yet (it ought to be in a couple of weeks). I could try, but there's too much code to read (not enough time). Exactly what needs to be done? Fetch posts linking to arXiv from MO and submit them to arXiv? (I don't see why you need so many files for that) $\endgroup$ Commented Jun 29, 2013 at 11:23
  • $\begingroup$ I basically would need to understand the how of submitting a TB to arXiv. $\endgroup$ Commented Jun 29, 2013 at 11:25
  • $\begingroup$ Nevermind, I got it: code.google.com/p/mathoverflow/source/browse/trackbacks/… $\endgroup$ Commented Jun 29, 2013 at 11:29
  • $\begingroup$ I think modifying the code is a better option. The API doesn't allow regexes. However, it may be a nice idea to store the last accessed post ID and then periodically fetch all new posts from the API. $\endgroup$ Commented Jun 29, 2013 at 11:33

2 Answers 2


I think that trackback notification should be working again. Can someone find a suitable arXiv link to add, then wait an hour and check http://arxiv.org/tb/recent to see if it comes through?

I haven't yet done anything about arXiv links posted between the migration and now, but that should be relatively easy now.

  • 1
    $\begingroup$ Seems to work now $\endgroup$
    – Yemon Choi
    Commented Aug 23, 2013 at 4:15
  • $\begingroup$ The links posted during the 'gap' may have to wait until we have a database dump. I updated my scripts so that they should have been able to go back and fill in the gap, but I hit throttling limits on the Stack Exchange API. I'll try again tomorrow, being a bit more careful, and otherwise wait til we have a dump. $\endgroup$ Commented Aug 23, 2013 at 11:30
  • $\begingroup$ I just noticed that arxiv trackbacks are not generated for links posted in comments (e.g. mathoverflow.net/questions/52585/… and arxiv.org/abs/1304.6659 ). Is this intended behavior? $\endgroup$
    – j.c.
    Commented Feb 26, 2014 at 10:41
  • $\begingroup$ @j.c., no, that's no intended behaviour. I'll look into it. $\endgroup$ Commented Feb 26, 2014 at 23:37
  • 1
    $\begingroup$ @j.c. I've fixed the bug (the filter I was using on the Stack Exchange API was omitting all the comments on questions). Going forward the script should catch everything. I'm trying to re-run the script over the old stuff now. $\endgroup$ Commented Feb 27, 2014 at 3:27
  • $\begingroup$ Thanks! It seems the old stuff hasn't all been caught yet (e.g. my previous example, among others). Are the trackbacks just being spaced out? $\endgroup$
    – j.c.
    Commented Mar 15, 2014 at 13:40
  • $\begingroup$ No, nothing is happening at the moment in terms of collecting missed trackbacks. Someone (me)? Will have to write something so the new trackback script knows how to go through a database dump. $\endgroup$ Commented Mar 15, 2014 at 21:06
  • $\begingroup$ It seems that new links posted in comments are not generating trackbacks anymore. mathoverflow.net/a/199429 and arxiv.org/abs/1505.02323 . $\endgroup$
    – j.c.
    Commented May 1, 2016 at 18:54
  • $\begingroup$ Thanks for the note. I will look into this. I recently fixed the script to pick up the new, longer, identifiers, but perhaps I broke something at the same time. $\endgroup$ Commented May 2, 2016 at 21:51
  • 1
    $\begingroup$ @j.c., I'm not quite sure what happened. Everything seems in order! I'm again re-running it over all the old stuff. If you see any more examples, please let me know. $\endgroup$ Commented May 3, 2016 at 1:00

I haven't fixed this yet. This is just an update.

Unfortunately, code I'd written previously that interacted with the Stack Exchange API used API v1.1. They've since upgraded to v2.1 of the API, and v1.1 won't be available for MathOverflow. This reduces my personal enthusiasm for writing this code (because it's more work than I'd expected), but shouldn't be an obstacle for anyone else.

Here's the basic idea for a program that generates trackbacks.

There are two essentially independent programs, that need to be run on a recurring basis (at least daily, maybe as often as hourly). They interact only via an append-only text file (or something equivalent). Each has a tiny amount of 'internal state' that needs to be preserved between runs of the program, namely the time at which the program was last run.

Program 1 is the "scraper", responsible for finding new MathOverflow content that contains arXiv links. It should request, via the SE API, all new content (i.e. new posts, new comments, as well as edited posts) between the last run of the scraper and the current time. For each such piece of content, it should run a certain set of regex's (there are important details here; we should reuse the existing well-tested set) on the content. These regex's will output any arXiv ids found in the content (possibly several). For each id found, the scraper should append to its log 4 pieces of data:

  1. the current time
  2. the arXiv id
  3. the URL of the content (it doesn't really matter if this is the URL of the question or a more specific URL)
  4. the title of parent question

Program 2 is the "poster", responsible for sending trackbacks. On each run, it should inspect the output log from the scraper, and take all entries since the last run of the poster. For each entry, it should perform a GET request to the arXiv, according to this example:


That's pretty much it!

A few comments:

  1. The "poster" program will eventually gain extra duties; for example the selected papers network would like us to notify them too.
  2. We need to notify the arXiv administrators of the IP address of the machine doing the posting, so they can whitelist submissions.
  3. It would be nice to be able to "re-run" the scraper over old content, without generating duplicate trackbacks, in case we miss something, update regex's, etc. This might be as simple as resetting the 'last run' timestamp, then deduplicating the log file.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .