Digital Legacy: Link Rot Mitigation Issues

Photo by: JTMorkis via Pixabay

Introduction

About a week ago there was a post by Trevor Morris titled The Loss of Links Continues¹ to which both Naty S.² and Michael Harley³ responded. I found these posts interesting, as they relate to another series of posts, such as Michael Harley’s post on making his blog more durable⁴ and his post on how he shares links⁵. There was also my post about embeds causing serious harm to one of my sites⁶.

From all of these articles I started doing some digging into the topic of link rot. What I found was a topic that was bigger in scope than I had initially thought, and I think is worth talking about further.

The Issue

In case I have readers that are unfamiliar with this topic, a quick definition of link rot. Linking is simply when one website has a reference that you can click that takes you to another piece of information. That information can be on the same website (like going to a different article, or pulling up a profile), or it can point to something on a different website (like all the posts I link in the end notes of this article). It is the singularly most fundamental item used for navigating the web.

Link rot is when a link no longer works. Typically, you’ll get something like a “not found” error message. There are several variations on this error, known by their numeric codes: 301, 302, and 404. The cause for these errors are many, but the most common causes are:

• The author moved the information to a new location.
• The author has removed the information from the internet.

So, what we are talking about in this article are the way in which we handle these situations where information has been moved or removed from the internet. The idea being that we are trying to provide the best solution we can.

The Most Common Scenario

The most common scenario we face is when a link has just completely disappeared. When this happens the most expedient way of handling this is to use the Internet Archive’s Wayback Machine⁷. And while this does attempt to ensure that a link is still pointing to something that is relevant, there are issues with this approach.

Incorrect Snapshot

The first issue I see with this approach is it’s quite possible to get an incorrect snapshot of the link target. For example, if a domain gets hijacked. It is possible that the most recent snapshot of a website is something completely different from what was originally intended.

Another situation is a website that has changed the underlying content. There are cases in which the owner of a website has decided (for whatever reason) to change the content of their site, and then given up. In this case it’s very likely the Wayback Machine’s image will be of the changed content.

It is possible, in these cases, to (most of the time) get an image of the intended content by searching through the snapshots. But this means that automation of updating links needs to be more aware of time. And, even then it is still possible to link to an image of a site that doesn’t match the context within an author’s post. IE, the target website updated a target (say adding a note of correction to an article), and we publish a post that is referencing that correction. What happens if the Wayback Machine doesn’t have an image of the target article containing the correction?

This suggests to me there is a need to further investigate methods of not just providing an updated link, but of ensuring that the context is being preserved as well.

Availability Concerns

The Internet Archive is one of the best tools available on the internet for finding obscure or potentially lost information. However, we need to be aware that it isn’t always the most accessible location for this that information. Unfortunately, politics has reared its ugly head when it comes to being able to access the Internet Archive.

As I write this, the Internet Archive is reportedly blocked in five countries⁸. However, one of the reports appears to be erroneous, listing the United States back in April 2026. However, there is another listing stating that it is available in the United States. (And, I have not had issues reaching the Internet Archive… But of course that doesn’t mean it isn’t blocked at a state level.)

The simple fact is, the countries that probably most need access to this information, won’t be able to access it from the Internet Archive. Of course, the fact that there are state level actors that are actively blocking parts of the internet isn’t something that we can directly effect.

However, having to switch links from targeting sources that were likely available globally, to a target that is known to be blocked in several regions is a less than optimal solution to this issue. Finding ways of providing relevant information without getting caught up in censorship filters is something we should consider.

Point of Failure Concern

This is something that I don’t like to mention, but it has to be said. The Internet Archive has been subject to outages many times over its lifespan. Unfortunately, it appears that their architecture doesn’t have enough redundancy in order to weather these outages successfully.

That isn’t to say that the information that they are storing is in jeopardy. They do have colocation facilities that have been quite successful in acting as a backup location for their storage. And, their storage and retrieval technology has greatly improved over the last decade. (I recall days when I would find something I had archived wasn’t available, and have to look at their storage system to find out what had failed. That is something I haven’t had to do in years.)

However, the fact is that both power and network disruptions have been a factor in service availability. This isn’t a new issue. It is an ongoing concern with their service.

Longevity

The Internet Archive wants to make all information that they store available in perpetuity. It’s a pretty simple statement that can be found on their site:

How long will you store files? As an archive our intention is to store and make materials in perpetuity. ⁹

However, this is only a statement of intention. The issue is that (as mentioned above) there are state actors that block the information that the Internet Archive provides. But there are other concerns, for example, publishers blocking the Internet Archive/Wayback Machine¹⁰, and even content being removed based on simple requests¹¹.

This, therefore, means that the Archive cannot be considered a definitive record for all information stored within it. It might be the best option that we have, but it cannot be considered a primary source, but rather as just a source that is the closest thing we have to an accurate record.

How to Handle These Issues

I don’t have a complete answer. These are very deep issues that we have to contend with, and I don’t believe there is a single, most reliable answer.

One of the things that might be useful is to record retrieval information in the same manner as Wikipedia.:

Screenshot of Wikipedia References

If nothing else, this might help to document potential discrepancies between a Wayback Machine link, and the original link. Further, if a retrieval date could be encoded in a way that is parsable, we could use this is selecting the retrieval copy from the Wayback Machine. Of course, this assumes that there is an image that matches what we exactly used as our original reference.

The final option, and it’s one that seems like an extreme option, but it may be the best option we have, is to capture our own images. I use Zotero¹² for storing a lot of information, even my website bookmarks. One of the best options with this is that it can store images / snapshots when an item is added to Zotero:

Screenshot of Zotero entry showing Accessed Date

The usefulness of this is it provides a record of when the link was added to a collection, when it was accessed, along with a snapshot. One option that might be worth investigating is keeping an archive of snapshots available, so if link rot is encountered, a snapshot can be provided in the link’s place.

Another option that a tool like Zotero offers is the ability to generate a PDF of the content that you are saving. This would be useful if you want to have the content in a searchable format in the future.

Of course, using a bibliography database tool like Zotero has disadvantages. It’s not necessarily the most intuitive tool to use given that it’s more focused on users doing research. And, it isn’t a local first tool: it relies on cloud storage, which itself has challenges. For example, if you run out of storage you may need to pay for more, and that can be pricey. Personally, I haven’t looked into the issue of longevity where Zotero is concerned.

There are many other options available: book managers, offline readers, and web clipping tools that offer features that may fill the needs for handling link rot. One that loos promising is an offline reader called Wallabang¹³ that can be self-hosted.

The most simple option (at least for users of Firefox based browsers) could be printing webpages to PDF files, and building an index of the PDFs. The nice thing about this is a PDF printed from Firefox (or a variant) will have the URL and date in the header / footer (as long as they are turned on):

PDF of webpage, circled URL and Date

Of course with Firefox variants you can also just save a screenshot. The problem with this, in my opinion, is that it will be on you to preserve the metadata that goes with the screenshot (ie, the URL, the date / time the screenshot was taken, storage and retrieval, etc.). And, to my surprise, a screenshot was larger than a PDF file (this wallabang.it page was 837K as a PDF, and over 1Meg as a PNG — not a huge difference, but a significant difference).

Conclusion

When I started working on an article about link rot, I didn’t expect it to take this turn. I hadn’t started with the concept that the current methods for dealing with link rot were not necessarily the best. In fact, I really wanted to look at this topic from the perspective of how it could affect our writing.

However, as I started working my way into the story, I realized that my original framing wasn’t going to be the best way to go. That talking about link rot was a whole topic unto itself that deserved some deeper investigation. The thing that really put me into this mindset was the idea of using the Wayback Machine to mitigate the situation.

While I love the Internet Archive, and the Wayback Machine, using it in this manner set off some alarm bells for me. The loudest of those alarm bells was: single point of failure. Should anything happen to the Internet Archive and / or the Wayback Machine, the mitigation would be broken. And, if it did break, it could actually make things worse.

As I investigated things from this perspective, I realized that there were already issues with using the Wayback Machine, even in it’s non-broken state. These issues include it not being accessible from all countries, and the potential for posts to not be able to link to accurate snapshots.

In the end I started thinking about how Wikipedia handles references. The thing that I noted that they do is to add a retrieval date to their references. This could at least help in providing an accurate link to a snapshot in the Wayback Machine, if that is the only option being considered.

However, I felt exploring other possibilities was appropriate. Honestly, I’m not convinced that I’ve found any that I really like. For me the most convenient (kind of) is either Zotero or possibly (even though I haven’t mentioned it) Obsidian. But there are plenty of bookmark managers, offline reading applications, RSS readers, and other tools that allow saving snapshots or screenshots of web pages for archival purposes.

So what was the point of all this? The long and short is this: don’t rely on a service that may have issues that we cannot control. The fact is, there are too many variables that aren’t controllable, from technology issues, to social issues, to political issues. Keeping some type of local archive that is accessible can allow us to update our posts if link rot occurs, and not have to rely on third party services.

There are a lot of other topics to cover related to the longevity / legacy of our sites. I will continue covering those in other posts (those will hopefully be less technical, and more philosophically oriented).