15
$\begingroup$

(Although this is a bug report, I or Anton are likely the people who need to fix it; this post is a bug tracking item, not necessarily a plea for help :-)

We previously been scraping the front page of MathOverflow for questions that linked to the arXiv, using a long list of regular expressions to try to capture everything. For each of these links, we generated a 'trackback' to the arXiv, and they then added a link back to the MathOverflow post citing that article.

We'd also periodically run a different script against a database dump, to catch any links that we failed to notice by scraping.

We're no longer generating any trackbacks; c.f. http://arxiv.org/tb/recent, set the list to show the 100 most recent trackbacks, and notice that they stop after last Monday, the migration.

Presumably this is because our screen scraping code no longer matches up with the HTML that MO 2.0 produces. An easy fix will be to tweak that code. A harder but better fix would be to use the Stack Exchange public API to achieve the same result.

$\endgroup$
  • 4
    $\begingroup$ If anyone would like to jump in and write something that uses the public API, please let me know and I can point you in the right direction. Our (abysmal) shell scripts for doing the scraping are in a mercurial repository, c.f. code.google.com/p/mathoverflow/source/browse/#hg%2Ftrackbacks $\endgroup$ – Scott Morrison Jun 28 '13 at 23:13
  • $\begingroup$ I think the MO API isn't enabled yet (it ought to be in a couple of weeks). I could try, but there's too much code to read (not enough time). Exactly what needs to be done? Fetch posts linking to arXiv from MO and submit them to arXiv? (I don't see why you need so many files for that) $\endgroup$ – Manishearth Jun 29 '13 at 11:23
  • $\begingroup$ I basically would need to understand the how of submitting a TB to arXiv. $\endgroup$ – Manishearth Jun 29 '13 at 11:25
  • $\begingroup$ Nevermind, I got it: code.google.com/p/mathoverflow/source/browse/trackbacks/… $\endgroup$ – Manishearth Jun 29 '13 at 11:29
  • $\begingroup$ I think modifying the code is a better option. The API doesn't allow regexes. However, it may be a nice idea to store the last accessed post ID and then periodically fetch all new posts from the API. $\endgroup$ – Manishearth Jun 29 '13 at 11:33
6
$\begingroup$

I think that trackback notification should be working again. Can someone find a suitable arXiv link to add, then wait an hour and check http://arxiv.org/tb/recent to see if it comes through?

I haven't yet done anything about arXiv links posted between the migration and now, but that should be relatively easy now.

$\endgroup$
  • 1
    $\begingroup$ Seems to work now $\endgroup$ – Yemon Choi Aug 23 '13 at 4:15
  • $\begingroup$ The links posted during the 'gap' may have to wait until we have a database dump. I updated my scripts so that they should have been able to go back and fill in the gap, but I hit throttling limits on the Stack Exchange API. I'll try again tomorrow, being a bit more careful, and otherwise wait til we have a dump. $\endgroup$ – Scott Morrison Aug 23 '13 at 11:30
  • $\begingroup$ I just noticed that arxiv trackbacks are not generated for links posted in comments (e.g. mathoverflow.net/questions/52585/… and arxiv.org/abs/1304.6659 ). Is this intended behavior? $\endgroup$ – j.c. Feb 26 '14 at 10:41
  • $\begingroup$ @j.c., no, that's no intended behaviour. I'll look into it. $\endgroup$ – Scott Morrison Feb 26 '14 at 23:37
  • 1
    $\begingroup$ @j.c. I've fixed the bug (the filter I was using on the Stack Exchange API was omitting all the comments on questions). Going forward the script should catch everything. I'm trying to re-run the script over the old stuff now. $\endgroup$ – Scott Morrison Feb 27 '14 at 3:27
  • $\begingroup$ Thanks! It seems the old stuff hasn't all been caught yet (e.g. my previous example, among others). Are the trackbacks just being spaced out? $\endgroup$ – j.c. Mar 15 '14 at 13:40
  • $\begingroup$ No, nothing is happening at the moment in terms of collecting missed trackbacks. Someone (me)? Will have to write something so the new trackback script knows how to go through a database dump. $\endgroup$ – Scott Morrison Mar 15 '14 at 21:06
  • $\begingroup$ It seems that new links posted in comments are not generating trackbacks anymore. mathoverflow.net/a/199429 and arxiv.org/abs/1505.02323 . $\endgroup$ – j.c. May 1 '16 at 18:54
  • $\begingroup$ Thanks for the note. I will look into this. I recently fixed the script to pick up the new, longer, identifiers, but perhaps I broke something at the same time. $\endgroup$ – Scott Morrison May 2 '16 at 21:51
  • 1
    $\begingroup$ @j.c., I'm not quite sure what happened. Everything seems in order! I'm again re-running it over all the old stuff. If you see any more examples, please let me know. $\endgroup$ – Scott Morrison May 3 '16 at 1:00
4
$\begingroup$

I haven't fixed this yet. This is just an update.

Unfortunately, code I'd written previously that interacted with the Stack Exchange API used API v1.1. They've since upgraded to v2.1 of the API, and v1.1 won't be available for MathOverflow. This reduces my personal enthusiasm for writing this code (because it's more work than I'd expected), but shouldn't be an obstacle for anyone else.

Here's the basic idea for a program that generates trackbacks.


There are two essentially independent programs, that need to be run on a recurring basis (at least daily, maybe as often as hourly). They interact only via an append-only text file (or something equivalent). Each has a tiny amount of 'internal state' that needs to be preserved between runs of the program, namely the time at which the program was last run.

Program 1 is the "scraper", responsible for finding new MathOverflow content that contains arXiv links. It should request, via the SE API, all new content (i.e. new posts, new comments, as well as edited posts) between the last run of the scraper and the current time. For each such piece of content, it should run a certain set of regex's (there are important details here; we should reuse the existing well-tested set) on the content. These regex's will output any arXiv ids found in the content (possibly several). For each id found, the scraper should append to its log 4 pieces of data:

  1. the current time
  2. the arXiv id
  3. the URL of the content (it doesn't really matter if this is the URL of the question or a more specific URL)
  4. the title of parent question

Program 2 is the "poster", responsible for sending trackbacks. On each run, it should inspect the output log from the scraper, and take all entries since the last run of the poster. For each entry, it should perform a GET request to the arXiv, according to this example:

http://arxiv.org/trackback/1307.1702?blog_name=MathOverflow&title=A%20Question&url=http://mathoverflow.net/questions/123

That's pretty much it!

A few comments:

  1. The "poster" program will eventually gain extra duties; for example the selected papers network would like us to notify them too.
  2. We need to notify the arXiv administrators of the IP address of the machine doing the posting, so they can whitelist submissions.
  3. It would be nice to be able to "re-run" the scraper over old content, without generating duplicate trackbacks, in case we miss something, update regex's, etc. This might be as simple as resetting the 'last run' timestamp, then deduplicating the log file.
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .