25
$\begingroup$

This question from March 2020 points out that many questions/answers link to papers at the http://front.math.ucdavis.edu/ frontend instead of at the arXiv itself, but that the frontend had been down a few weeks. There are c.600 affected questions and it looks like a large proportion of the links could be fixed by a simple search-and-replace; so comments and the well-received answer suggested asking SE employees to do such a search-and-replace fix (moderators don’t have that ability), but felt “down for a few weeks” was too soon.

The frontend has now been down for 17 months; presumably this is no longer too soon. Can moderators now petition SE for a search-and-replace fix? Or if that’s too administratively difficult, should we just fix the links by hand, trying not to bump too many at once as usual?

Useful search queries, copied from the older question:

Search-and-replace suggestion: The links fall into several groups, some that should be easy to recognise and fix by regex, others less clear. Hopefully a large proportion can be fixed by regex, leaving the rest to fix by hand.

For links to preprints after April 2007, the correspondence is straightforward — we want to replace a url like http://front.math.ucdavis.edu/0809.1179 with https://arxiv.org/abs/0809.1179, and the first four digits represent YYMM. So a regex replacing https?://front.math.ucdavis.edu/([12]|0[8-9]|071|070[4-9]) with https://arxiv.org/abs/\1 should recognise and fix these links, I think. (Details depend on the regex flavour used, of course.)

Links to older preprints are less consistent. Some have the form https://front.math.ucdavis.edu/math.LO/9909115, and these should be easy again: the identifier part agrees with the arXiv’s, so a regex replace of https?://front.math.ucdavis.edu/(math[.a-zA-Z]*/\d+) by https://arxiv.org/abs/\1 should recognised and fix these. But others use a different identifier, whose correspondence with the arXiv’s identifiers seems unclear — e.g. http://front.math.ucdavis.edu/0310.5109 corresponds to https://arxiv.org/abs/math/0310109 (and https://arxiv.org/abs/0310.5109 doesn’t work). Does anyone else know or see how the correspondence works for these?

Finally there are links to category searches, and the like, which I don’t think we can hope to fix by regex. This edit history has examples of all three kinds.

$\endgroup$
22
  • 4
    $\begingroup$ I will add that there are 248 comments with such links. (If Stack Exchange would be willing to use some automated proces to change such links, they can probably do so also in comments, not only in posts.) $\endgroup$ Aug 25, 2021 at 9:22
  • 1
    $\begingroup$ @AsafKaragila: I’ve added a regex suggestion which I think should get a large proportion of the broken links, and hopefully no false positives, although of course I haven’t been able to test/debug it on the full input set. $\endgroup$ Aug 25, 2021 at 9:43
  • 3
    $\begingroup$ Thanks! With the status-review, I hope we'll attract the attention of some StackOverlords soon. I'll see if I can speed it up as well. $\endgroup$
    – Asaf Karagila Mod
    Aug 25, 2021 at 9:51
  • 1
    $\begingroup$ @MartinSleziak: Right — the confusing thing is that for pre-2007 papers, the frontend seems to provide url’s in at least two different forms, some like the example you give, some like the ones in the edit history I linked. For the ones in the edit history, the correspondence with arXiv’s url/identifier is totally unclear to me. For examples like your form, the conversion is clearer — the identifier math.LO agrees with the arXiv, so we just need to change the url prefix — but it would take a little thought to write a regex that’ll recognise url’s of that form without false positives. $\endgroup$ Aug 25, 2021 at 11:23
  • 2
    $\begingroup$ @Martin: The [status-review] puts the thing on their todo list, which is probably long enough that it will take a while. We absolutely should figure out the algorithm for replacing until then, though. $\endgroup$
    – Asaf Karagila Mod
    Aug 25, 2021 at 11:30
  • 3
    $\begingroup$ Maybe for some of them it is removing .5 and adding /math and replacing .47 by 0 and adding /math-ph? (But I only tested it on a few URLs.) $\endgroup$ Aug 25, 2021 at 12:20
  • 1
    $\begingroup$ Shouldn’t the regex for post-2007 include numbers starting with 2 as well? $\endgroup$ Aug 25, 2021 at 12:22
  • 2
    $\begingroup$ @EmilJeřábek: Thanks, fixed! @ Martin: yes, from a few examples it looks like the system works something like that. But at least for my part, I wouldn’t feel confident proposing a regex for a half-guessed system without being able to test the regex out a bit. If you feel confident enough to add a regex to handle these then that’d be great! $\endgroup$ Aug 25, 2021 at 13:59
  • 1
    $\begingroup$ Does somebody know whether lanl.arxiv.org is down only temporarily or whether all links to that domain should be considered dead, too? (This is not directly related to the front.math.ucdavis.edu, but it seems like a similar problem, so I thought that it might be worth mentioning.) $\endgroup$ Sep 11, 2021 at 8:03
  • 1
    $\begingroup$ Just to note, Martin and I found yet another oddity. The arXiv front identifier "9612.5114" which I thought should be "math/9612114", really points to "dg-ga/9612014". So maybe ***.51** = dg-ga? $\endgroup$
    – David Roberts Mod
    Sep 23, 2021 at 3:27
  • 3
    $\begingroup$ Aha, I found the old FAQ posting where the Front explained the numbering technicalities web.archive.org/web/20060901072412/http://…: "Until March 2000, the Front renumbered articles in the old mathematical archives alg-geom, funct-an, dg-ga, and q-alg as math archive articles. To avoid duplicate numbers, the system added 50 to each funct-an number, 100 for dg-ga, and 140 for q-alg. Since this system was never adopted at the arXiv, it has for now been scrapped.... $\endgroup$
    – David Roberts Mod
    Jan 20, 2022 at 2:54
  • 2
    $\begingroup$ @Asaf can we get the [status-review] back on this? We are probably in a position to have an automated solution applied. See my answer for more details $\endgroup$
    – David Roberts Mod
    Mar 30, 2022 at 11:37
  • 4
    $\begingroup$ Sorry for the noisy edits - the system (for an unknown reason) did not send this to our board for review when the status-review tag was added so I'm trying to force it to re-run. $\endgroup$
    – Catija
    Apr 6, 2022 at 13:54
  • 2
    $\begingroup$ @Slate Just to clarify - this change will be on MathOverflow only, right? AFAICT, nobody checked whether some of the posts on other sites contain the links before April 2007 - where simple replacement doesn't work. Moreover, number of posts on other sites seems to be small. (From search I got 277 results networkwide and 262 results on MO.) $\endgroup$ Jun 19, 2022 at 11:51
  • 3
    $\begingroup$ @MartinSleziak I've clarified in the request that it should only be run on Math Overflow. $\endgroup$
    – Slate StaffMod
    Jun 21, 2022 at 16:15

5 Answers 5

6
$\begingroup$

Thanks for raising this one to our attention. We've now executed the requested changes on Math Overflow. However, it looks like, given the work y'all put in, this change set is relatively issue-free, which is pretty fantastic. Still, as with all link replacements, we do expect a bit of routine cleanup work, so be on the lookout for any cases that look odd.

Let us know if you run into any major fallout as a result of the change and we'll see what we can do to help.

$\endgroup$
1
  • 2
    $\begingroup$ Thanks heaps to the team! $\endgroup$
    – David Roberts Mod
    Jun 25, 2022 at 1:56
9
$\begingroup$

As of 29 March 2022, I claim all the links in questions and answers to front.math.ucdavis.edu that predate April 2007, when the arXiv moved to the new, uniform identifiers YYMM.NNNN[N], have been fixed, as well as every link to things with ids up to the end of 2007. This can be verified by checking the struck-through SEDE links to the various older categories in Martin's answer. This might also be verified by using fancier tools, like finding all mentions of the url string, but removing anything that's a comment, and anything where the arXiv id starts with 08, 09 or 1. There should only be the ≤14 posts with ids starting with 07, all of which are fixed, if any.

There are comments with links that point to the older link syntax used by the UCDavis Front, and (hopefully!) all of them are replied to pointing out a working link.

The remaining changes that need to be made are as follows, copied from Martin's answer:


Links which can be mass-converted


$\endgroup$
0
8
$\begingroup$

I've reviewed your request and - while I'd generally want to say "yes, we can do this" - in this case, I'm sorry to say that I don't think it will be possible - at least, not without some effort from you all first. I really do think it's important that the links on this site are functional, so I appreciate your bringing this up for discussion.

The tool we have to make this change is a pretty simple find/replace tool that only works for post content, not comments. It only works for URLs and it can't accept RegEx. So, I think you may already see the problem. While I really appreciate the effort that y'all went to to find the various treatments of the URLs, we can't even only update the ones that are an easy swap without changing all of the others since the content needing to be replaced is the same in all cases.

As you might also see, this doesn't mean that we can't ever do this - it means that we'd need you to help us out before we can. Based on the comments, it sounds like about 2/3 of the ~900 links are the "easy" first and second type. If that's the case, you could manually repair the remaining ~300 URLs first and we can use the link editing tool to fix the remainder of the links which, despite being in two formats require the same change to happen since arXiv helpfully automatically redirects to the correct URL.

So to put it in other words:

  1. First you all work together to fix the outstanding links of the second and third type manually.

    • Based on a comment from Martin Sleziak it looks like there's about 300 links in the third category. It seems like correcting these links in an automated way may be difficult regardless since it's unclear how they should change.
  2. Once the difficult URLs are fixed, we will bulk find/replace the remaining using the following method:

    • Find http://front.math.ucdavis.edu/ and replace with https://arxiv.org/abs/

As far as I understand, we can not bulk edit all 900 URLs first as it will make the pre-2007 URLs even more "wrong" and may make them more difficult to find and repair at a later point in time.

If you can find a consistent way to convert from one of the pre-2007 URLs to the new arXiv URL, you may be able to have someone create a userscript to update these over time, a few questions per day but that's a big "if".

Please let me know if this is clear or if you have any other questions about this response.

$\endgroup$
12
  • $\begingroup$ As someone whose edit history will show his endless appetite for tidying, I would be happy to be part of such a manual effort, which presumably it would be unobjectionable to spread over time. However, is there any way to do it without meaninglessly promoting hundreds of posts to the front page? $\endgroup$
    – LSpice
    Sep 14, 2021 at 3:28
  • $\begingroup$ @LSpice Something like that was discussed several times, you can see various feature requests around this. (I still hope that this is going to be implemented at some point - but AFAICT, at the moment there is no way to edit posts without bumping.) See also: Minor edits, subject to review (on this meta) or Allow non-bumping minor edits, but review them on /review (on Meta Stack Exchange). $\endgroup$ Sep 14, 2021 at 5:10
  • 2
    $\begingroup$ The tool we have to make this change is a pretty simple find/replace tool that only works for post content, not comments. It only works for URLs and it can't accept RegEx. You are a software company, so making new tools to interact with your main product should be a normal thing for you. And a regex find/replace engine is something that could have many other applications in additions to this task. I imagine it there are prioritization and cost/benefit issues here; "we don't have the tools to do that" surely is not the whole story. $\endgroup$ Sep 16, 2021 at 10:18
  • 2
    $\begingroup$ @FedericoPoloni considering the vast list of things that need to be addressed, building a better tool for this specific issue is extremely low priority. $\endgroup$
    – Catija
    Sep 16, 2021 at 12:18
  • $\begingroup$ @LSpice I've been attacking them slowly, using the search tools in Martin's answer. It would be nice to divide the effort. Perhaps attacking the "Maybe manually, maybe mass converted" ones would be a good place to start, they could be knocked off over a week or so. The pre-2000 ones likewise. The ~200 posts with links from 2000–06 are going to be a slow slog. There may be some overlap, though, which will help—I've seen posts with multiple links from all over the shop. $\endgroup$
    – David Roberts Mod
    Sep 23, 2021 at 2:13
  • $\begingroup$ @DavidRoberts, I definitely still want to help, although, as for everyone, the semester being underway cuts down dramatically on my MO time (hence my near-two weeks' delay). Have you just been editing them as regular posts, in a way that bumps them to the front page? (I understand that that is the norm, and know no better way, but wanted to make sure it was the appropriate way to proceed before diving in.) $\endgroup$
    – LSpice
    Oct 4, 2021 at 17:06
  • 1
    $\begingroup$ @LSpice yes, I've been just doing a couple at a time, the normal way. Even just attacking one a day would be progress. The posts are spread over a wide range of topics, so it's not like fixing up a whole bunch on a niche topic and crowding the front page. $\endgroup$
    – David Roberts Mod
    Oct 4, 2021 at 23:06
  • 1
    $\begingroup$ Meanwhile, the tool was apparently significantly improved. Does this change the answer in any way? $\endgroup$ Oct 7, 2021 at 14:55
  • 2
    $\begingroup$ @EmilJeřábek No, it doesn't. The improvements made don't include regex - the main improvement was that we now can run the query across the network rather than per-site. CMs also have the ability to preview the impact of the changes before requesting the dev team makes them. Regex hasn't been added. $\endgroup$
    – Catija
    Oct 7, 2021 at 15:03
  • $\begingroup$ All right, thanks for the explanation. $\endgroup$ Oct 7, 2021 at 15:24
  • 2
    $\begingroup$ @Catija we have made big progress on the manual edits, see my new answer here with a summary. This should be ready to be revisited by next week, if not before. The aim is to have only arXiv Front links from 2008 onwards outstanding and needing replacement. This is not a problem for a simple find/replace. $\endgroup$
    – David Roberts Mod
    Mar 29, 2022 at 7:23
  • 4
    $\begingroup$ All the manual edits seem to be done, now. Time to revisit this one, if I may be so bold... $\endgroup$
    – David Roberts Mod
    Mar 30, 2022 at 11:40
7
$\begingroup$

As some of you may have noticed, I have a script which repairs broken links / images, with some more options than the mass-replacement tool available to Stack Exchange staff. It can do all kinds of substitutions, checks if the new URL actually exists and if not, queries the Wayback Machine for a snapshot.

I fed it the following rules:

    {
        "urlPart": "front\\.math\\.ucdavis\\.edu/(0[0-6]\\d{2})\\.5(\\d{3})",
        "status": "KNOWN_BROKEN",
        "replacements": [
            "arxiv.org/abs/math/$3$4"
        ]
    },
    {
        "urlPart": "front\\.math\\.ucdavis\\.edu/(0[0-6]\\d{2})\\.47(\\d{2})",
        "status": "KNOWN_BROKEN",
        "replacements": [
            "arxiv.org/abs/math-ph/$30$4"
        ]
    },
    {
        "urlPart": "front\\.math\\.ucdavis\\.edu/(\\d{4}\\.\\d{4})",
        "status": "KNOWN_BROKEN",
        "replacements": [
            "arxiv.org/abs/$3"
        ]
    },
    {
        "urlPart": "front\\.math\\.ucdavis\\.edu/math(\\.LO)?/(\\d{7})",
        "status": "KNOWN_BROKEN",
        "replacements": [
            "arxiv.org/abs/math/$4"
        ]
    }

I just did a dry run and these are the replacements it would make. The first column is the number of posts the link appears in. I can't post the results here because the table is too large (an answer is limited to 30,000 characters).

The only link it cannot replace is http://front.math.ucdavis.edu/0309.3208 which appears in number-theoretic congruences with geometry and topology?.

When everything checks out, I can limit the edits it makes to posts which cannot be repaired by the mass-replacement tool; after all, my edits will bump posts to the homepage.

$\endgroup$
7
  • 1
    $\begingroup$ In case it is useful for some people reading this, I will add a link to a previous discussion on this meta where your script was brought up: Are edits performed by external, non-SE robots allowed? $\endgroup$ Jan 19, 2022 at 12:34
  • $\begingroup$ I found a paper that one might think corresponds to the arXiv Front identifier 0309.3208, namely arxiv.org/abs/math/0309208, but the topic is wrong. It might be arxiv.org/abs/math/0309108, but then I can't see how the identifier transforms. Looking at web.archive.org/web/20071012191626/http://…, the paper math/0309108 has url link containing 0309.5108. $\endgroup$
    – David Roberts Mod
    Jan 20, 2022 at 3:07
  • 1
    $\begingroup$ @DavidRoberts thanks. We shouldn't discount the possibility that the link was/is wrong to begin with ... $\endgroup$
    – Glorfindel
    Jan 20, 2022 at 6:31
  • $\begingroup$ @Glorfindel True, but unexpected! I was surprised to see the 90s papers in your list, and then doubly so when I checked the progress I'd made on manual edits and saw I'd crossed off a bunch in that format! $\endgroup$
    – David Roberts Mod
    Jan 20, 2022 at 11:58
  • $\begingroup$ I fixed the ones in answers with front.math.ucdavis.edu/math in the urls from my last comment. The comments with links like this had been fixed by Asaf, and so in the bump of Martin's answer I'd missed that they weren't crossed off. $\endgroup$
    – David Roberts Mod
    Jan 21, 2022 at 0:09
  • 2
    $\begingroup$ Thanks for all your help on this! It would have taken a lot longer, otherwise... $\endgroup$
    – David Roberts Mod
    Mar 30, 2022 at 11:38
  • $\begingroup$ @DavidRoberts you're welcome, it was a pleasure. Thanks for your efforts as well! $\endgroup$
    – Glorfindel
    Apr 13, 2022 at 19:16
6
$\begingroup$

At this moment, it is unclear whether a project like this should be done. (After all, until we heard from people running the arXiv frontend, we still should count as a possibility that it will be running again at some point.)

Still, it might be useful to analyze a bit what would the scope of the project be - how many links would need manual editing, how many links can be mass-converted, etc.

I have started this CW answer - please do edit it if you have additional information that might be useful here.

Resources

Notice that the numbers listed below are about the status at the time when this answer was first posted - you can find how the counts have changed using the links given below and from the updates added to this post.

Links which can be mass-converted


(DR: I claim all the rest are dealt with.)

Links which can be mass-converted

Links which have to be edited manually

Maybe manually, maybe mass converted

  • Links with math.AG, math.AT, math.LO etc.: They probably can be converted automatically, but a separate replacement for each category would be needed. Considering the low count, maybe manual editing might be preferable. SEDE returns 27 posts (DR: these have all been manually fixed, I took the opportunity to add more value by giving paper details and links to published versions and also fix other nearby broken links) and 6 comments (DR: all these comments have a reply giving the updated link and title of the paper.) SEDE - last activity, SEDE - already edited
  • 6 posts with links to front.math.ucdavis.edu/math/: search, SEDE (SEDE - last activity, SEDE - already edited). There are also 3 such comments. (These comments are listed also here in chat.) Those comments have been edited by Asaf Karagila.

Links not going to articles

$\endgroup$
8
  • 1
    $\begingroup$ I have struck of a couple that needed only two post edits. We should be able to get some of the items here, with only a few posts each, cleaned up within a couple of days, then likewise clean up the list. $\endgroup$
    – David Roberts Mod
    Sep 20, 2021 at 5:30
  • $\begingroup$ @Martin: I've fixed the comments for you. $\endgroup$
    – Asaf Karagila Mod
    Jan 10, 2022 at 20:48
  • $\begingroup$ @AsafKaragila You mean these three comments, right? $\endgroup$ Jan 10, 2022 at 20:54
  • $\begingroup$ @Martin: As one can guess. Seeing how the links there are now correct... $\endgroup$
    – Asaf Karagila Mod
    Jan 10, 2022 at 20:55
  • 1
    $\begingroup$ Down to 98 manual ones in questions and answers, now! $\endgroup$
    – David Roberts Mod
    Jan 30, 2022 at 3:19
  • 2
    $\begingroup$ @DavidRoberts Thank you a lot for your ongoing effort! $\endgroup$ Jan 31, 2022 at 14:16
  • 1
    $\begingroup$ Now only comments to deal with in the time 2000–06! They can be dealt with by replying with a link to the current arXiv url. There are also only a small number of 2007 links in posts and comments, which can be dealt with. Then we only have the time period 2008–2019 to fix. Can this indeed be done with a script? The arxiv IDs are completely uniform in this time period. $\endgroup$
    – David Roberts Mod
    Mar 26, 2022 at 9:59
  • 1
    $\begingroup$ Big thanks to all who have been slowly chipping away at these! $\endgroup$
    – David Roberts Mod
    Mar 26, 2022 at 10:01

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .