I haven't fixed this yet. This is just an update.
Unfortunately, code I'd written previously that interacted with the Stack Exchange API used API v1.1. They've since upgraded to v2.1 of the API, and v1.1 won't be available for MathOverflow. This reduces my personal enthusiasm for writing this code (because it's more work than I'd expected), but shouldn't be an obstacle for anyone else.
Here's the basic idea for a program that generates trackbacks.
There are two essentially independent programs, that need to be run on a recurring basis (at least daily, maybe as often as hourly). They interact only via an append-only text file (or something equivalent). Each has a tiny amount of 'internal state' that needs to be preserved between runs of the program, namely the time at which the program was last run.
Program 1 is the "scraper", responsible for finding new MathOverflow content that contains arXiv links. It should request, via the SE API, all new content (i.e. new posts, new comments, as well as edited posts) between the last run of the scraper and the current time. For each such piece of content, it should run a certain set of regex's (there are important details here; we should reuse the existing well-tested set) on the content. These regex's will output any arXiv ids found in the content (possibly several). For each id found, the scraper should append to its log 4 pieces of data:
- the current time
- the arXiv id
- the URL of the content (it doesn't really matter if this is the URL of the question or a more specific URL)
- the title of parent question
Program 2 is the "poster", responsible for sending trackbacks. On each run, it should inspect the output log from the scraper, and take all entries since the last run of the poster. For each entry, it should perform a GET request to the arXiv, according to this example:
That's pretty much it!
A few comments:
- The "poster" program will eventually gain extra duties; for example the selected papers network would like us to notify them too.
- We need to notify the arXiv administrators of the IP address of the machine doing the posting, so they can whitelist submissions.
- It would be nice to be able to "re-run" the scraper over old content, without generating duplicate trackbacks, in case we miss something, update regex's, etc. This might be as simple as resetting the 'last run' timestamp, then deduplicating the log file.