I would say that to some extent this could be simply based on common sense. And perhaps you could take into account your estimate whether you think that your answer (and the link/citation) might be useful in the future.
Of course, the link is working at the time when you're posting it. But maybe the link stops working at some point. It would be nice if the reader would still be able to somehow identify the paper. (At least if you assume that somebody might want to look at your answer a few years later.)
For example, probably we could assume that DOI or arXiv-identifier are relatively stable - and if they are mentioned, the reader should be able to get the information about the paper. But maybe for some other types of URLs, it might be quite difficult to get the paper just from the URL.
- I guess
springerlink.com
is quite a well-known case. There are about 1000 dead links to springerlink.com networkwide and about 500 on MathOverflow. Many such links have already been fixed. If a post says nothing beyond "see equation (3.1) in this paper", then the reader will a have hard time finding out what the paper was once the link stops working. If the post contains some additional information, then even if the link stops working, the reader will be able to find the paper.
- To use an example of your own post, here is an example of a link which no longer works: "A striking result referenced in the Shulman paper is due to Colin McLarty, establishing that the NF axiomatization of what a set is yields a ${\bf Set}$ that isn't Cartesian closed." From the context, perhaps the reader could eventually be able to find the paper in question. But I guess it is nice that somebody made an effort and replaced the broken link with a DOI-link. (In this case, there is a full citation of the paper near the end of the post - so there is actually no problem here. But finding the paper just from this paragraph - and from the broken link - would be a bit harder.)
(I would like to make clear that by including the above links I do not want to blame specific users. After all, who could have known that some springerlink or projecteuclid link might stop working. But I wanted to include at least some examples where something similar actually happened. There are probably much more egregious examples - I did not spend that much time searching. In both cases I linked above there was enough additional information in the question or in the answers to actually find the paper.)
Additionally, I will mention that adding link to Zbl can lead to link back to the MO post from zbMATH review to the MathOverflow post. Although I learned about this feature only recently, from this post on meta (and the answers posted there): Are there policies to respect in order for MathOverflow Q&As to be tracked back in relevant zbMATH Open reviews?
Related (at least tangentially):
A quote by Todd Trimble from this conversation: "I also think you make a good point about Math.SE being mindful of serving as a repository for the general audience out there. This probably is a weakness of MO, that it is not as useful as it could be."
Keeping possibility of link rot in mind is one of ways how to make a site like MathOverflow (or Mathematics Stack Exchange) useful.