Checking external links

Simple link checking describes how I check internal links, but what about checking external links? You need to traverse every page on your website to find the external links to check, but you definitely must not traverse every page on external sites! I found a tool that supports this: linkchecker. (Note that development seems to have stopped there and moved to a new group, but I’ve linked to the version packaged by Debian stable because that’s what I’m using.)

The key to checking external links without recursing through external websites with linkchecker is to pass the flags --check-extern and --no-follow-url=!://DOMAIN-BEING-CHECKED/ (e.g. --no-follow-url=!://; this will check external links but will not recurse on any URL that doesn’t match ://DOMAIN-BEING-CHECKED/.

I ran linkchecker like this for a month or so, but after the initial cleanup where I fixed some broken links it was too noisy - there are temporary failures frequently enough that the signal-to-noise ratio was very low. Some sites consistently fail, e.g. Wikipedia and Amazon, and consistent failures can easily be excluded with --ignore-url=//, but most failures are transient. (Wikipedia and Amazon block the default linkchecker User-Agent, setting the User-Agent to match Chrome’s Usr-Agent fixes them.) linkchecker supports a simple output format of failure_count URL that is updated on each run, but the counters are never reset and it doesn’t track when the failures occurred so the signal-to-noise ratio for alerts from that would decline over time.

I decided to write a wrapper to post-process the results and only warn about URLs that fail multiple times in a short period. Happily linkchecker supports SQL output, so I can import the failures into an SQLite database and easily query it. The schema that linkchecker uses is fine except it doesn’t have a timestamp, but that was easy to solve with SQLite: when creating the database I add an extra column defined as timestamp DATETIME DEFAULT CURRENT_TIMESTAMP that will automatically be populated when rows without it are inserted. I arbitrarily picked 3 failures in 2 days as the thresholds for warning about a URL, but I increased it to 12 failures in 2 days (1 failure every 4 hours) after too many false positives. I only count each URL as failing once per hour, regardless of how many times it failed within that hour, to avoid alerting when a URL that is linked many times has a temporary failure. The output looks like this:

Bad URLs for since 2019-06-08
Output in /tmp/linkchecker-cron.t8EB6fzK2F

To investigate further I can use an SQL query like SELECT * FROM linksdb WHERE urlname LIKE '%//';. The output files are also available for debugging when linkchecker fails, otherwise they are cleaned up. Both the output files and the database contain the referring URL for failures, so it’s easy to go edit the page and fix the link if there is a genuine failure, e.g. several links in my blog needed to be updated because the destinations had been moved over the years.

The wrapper program is linkchecker-cron and my linkcheckerrc might also be useful.