An Alternative Approach to XML Sitemaps

Here’s a screenshot I took of a website’s submitted URLs last year:

And here’s that same report around 6 months later:

The number of submitted URLs has fallen from 159k to 156.6k, but the number of indexed URLs has increased from 105k to 124k. Nineteen thousand more URLs are now eligible to receive organic traffic. One template went from 58% of URLs indexed to 76%.

I realise this isn’t a hockeystick graph, but I still think it’s worth sharing. There are some brain-feel-good graphs later, this is just to ease you in.

This is an ecommerce site with a substantial inventory, but there is no faceted navigation to contend with – only category pages, pagination, and product pages. The content quality did not improve. No active linkbuilding was undertaken. The internal linking to these pages which became indexed did not improve substantially.

What Changed?

I think everyone agrees in splitting XML Sitemaps by page type to make auditing and understanding what content is being indexed easier. This website was already doing this. But they were also doing this:

"All formats limit a single sitemap to 50MB (uncompressed) and 50,000 URLs." (Source) 

This number is often included in the copy-paste recommendations of heavily templated audits, which is why we see it so often on new client websites. Thanks!

Instead:

We moved from a few XML Sitemaps capped at 50,000 URLs to many XML Sitemaps containing 10,000 URLs. This took us from ~5 XML Sitemap files to ~15.

These smaller XML Sitemaps are organised chronologically, with products1.xml containing the oldest products, and products14.xml containing the most recent products. The indexing behaviour goes how you might expect:

This makes sense – older products are more likely to be buried deeper in the internal linking architecture, and as a result receive less internal PageRank.

Whilst this is true, I don’t think it’s the correct explanation here. This particular website is heavy on discontinued products. This makes the older pages more likely to be “lower quality” and far more likely to be identified as soft 404s, whilst the new products being added are in stock.

You probably notice that this is gradually tailing off, which is not unexpected considering none of this content has improved, and these products aren’t being restocked.

I’ve been intrigued with the idea that the amount of short-term storage available impacts the indexing of lower quality content (thanks to Martino for finding the tweet). I’m especially intrigued as you could use this idea to identify areas of the website which could be classified as ‘nice to have’ (i.e shit), from Google’s perspective by the ‘will-they-won’t-they’ indexing behaviour. I also like to think they have multiple copies of the index by date on tape archives in a disused missile silo somewhere.

How Does This Work?

I haven’t seen this idea written about a lot, but it is talked about by SEOs. I managed to find Barry Adams had previously written about this approach, and shared his thoughts and observations:

"I have found that limiting sitemaps to only 10,000 URLs leads to more thorough levels of indexing. I’m not sure why – I suspect that smaller lists of URLs are easier for Google to process and crawl – but it’s been proven time and again that smaller sitemaps lead to higher degrees of indexing."

(I wasn’t sure these were the same sites as there were 2,025,000 vs 1,288,000 URLs submitted – I checked with Barry and he confirmed they were different websites).

I understand some SEOs push for 1,000 URL Sitemaps so they can get around Search Console limits so it can be used to monitor which URLs aren’t getting indexed. I believe they are also accidentally being very efficient in terms of indexing.

I don’t actually know how this works, just that every time I try it, more pages end up indexed. It may be a coincidence.

This isn’t an ideal consulting position, so I’ve spent more time than is reasonable thinking about XML Sitemaps and put some thoughts together in case a client asks something awkward like “How does this work? {Google|Moz} says this is fine“. Here are some suggestions:

• Sitemaps with fewer entries are significantly easier to process. They also tend to be faster to download.
• Lighter Sitemaps (in terms of filesize) are faster to download and are easier to process. This is similar, but different.
• Frequently updated sitemaps get crawled frequently. You might experiment with rotating sort order in order to simulate this. Redoing your sitemaps like this forces a dose of ‘novelty’
• Related: Google does not wait around for your server to cough and splutter through the transfer of the larger file, and it doesn’t relish expending the resources parsing 50,000 lines either.

"Each Sitemap file that you provide must have no more than 50,000 URLs and must be no larger than 50MB (52,428,800 bytes). If you would like, you may compress your Sitemap files using gzip to reduce your bandwidth requirement; however the sitemap file once uncompressed must be no larger than 50MB"(source)

I’m still not getting anywhere, but thinking about XML Sitemaps has gotten me somewhere more interesting.

Google News Sitemaps

Consider the Google News XML Sitemap format. If you’ve done log analysis for sites in Google News, you’ll know those things get crawled. Lots.

Why is this the case? If we review the guidelines:

"Include URLs for articles published in the last 2 days. You can remove articles after 2 days from the News sitemap, but they remain in the index for the regular 30-day period."

"Update your News sitemap with new articles as they're published. Google News crawls News sitemaps as often as it crawls the rest of your site."

"Add up to 1,000 full URLs to include more, break these URLs into multiple sitemaps, and use a sitemap index file to manage them. Use the XML format provided in the sitemap protocol. Do not list more than 50,000 sitemaps in your sitemap file. These limits help ensure that your web server isn't using large files."

"Update your current sitemap with your new article URLs. Do not create a new sitemap with each update."

We are encouraged to keep the URL of this “gateway” sitemap static, whilst the contents change.

These are probably totally different systems and it might be that it’s not really regular Googlebot crawling it when it’s hammering those requests. It doesn’t matter, we can still apply the theory to how we treat regular sitemaps.

My general recommendations are around making XML Sitemaps with fewer URLs, when perhaps they should be about making smaller XML Sitemaps (as in file size). I don’t really believe this but I am now ruining my own website by adding cruft to the start of the post Sitemap, so it weighs in at a hefty 48.6MB. Enjoy, Googlebot:

I’ll update the post when {something|nothing} happens. I can’t try this out on a real website, unfortunately.

An Alternative Approach to XML Sitemaps

I understand that so far, this wasn’t as weird as posts on this blog tend to be. I’d like to share something with actual value.

This approach is useful if it is valuable for your content to be indexed ASAP, and if your content has a decent chance of being indexed on its own merit. It is also valuable if you have a lot of URLs not currently indexed that need indexing. Admittedly, these are both rare needs, but I was lucky enough to work on a project that required both with, crucially, a client willing to implement them.

Here’s the whole idea:

You can dynamically populate your XML Sitemaps based on what Googlebot is crawling.

Here is a brief sketch in terms simple enough for me to understand:

• You have a list of URLs you want Googlebot to crawl.
• You generate an XML sitemap based on a restricted slice of the most recent of these URLs (in this example, the top 20,000).
• You monitor your access logs for Googlebot requests.
• Whenever Googlebot makes a request to one of the URLs you are monitoring, the URL is removed from your list. This removes it from /uncrawled.xml
• This is appended to the appropriate long-term XML Sitemap (e.g. /posts-sitemap-45.xml). This step is optional.

To be extremely clear – please have a professional developer write this rather than an SEO.

The benefit? New, high priority content gets indexed as soon as possible without resorting to other ideas. Once you have this set up it does not take long for it to start working:

The second image takes a while to understand. Far more than 20,000 URLs appear in the XML Sitemap per day, but the size of the sitemap is capped at 20,000 entries. The number of URLs Search Console is reporting on here only represents the sitemap at one particular time, with the rate of indexing of this snapshot of URLs rapidly increasing. This is getting far more than 20,000 URLs per day crawled.

Of course I can’t attribute the success of this project wholly to XML Sitemaps…

An Alternative Approach to Internal Linking

You’ve guessed it. Here’s a second idea, built off the same approach as the first:

You can dynamically control your internal linking in response to the URLs Googlebot is crawling.

That’s it.

This uses the same methodology as the XML Sitemap generation, but to populate a severely size limited internal linking widget.

Googlebot cannot visit a single URL on this website without being exposed to novel URLs it has not crawled yet, like an incredibly aggressive re-targeting campaign. It is almost cruel.

Even though these link widgets are site-wide, Googlebot will only see the link on a few URLs at most with this approach before the links are updated. The site above has a lower DR than ohgm.co.uk, so it is not just PageRank doing the work here. You cannot calculate internal PageRank from this sort of internal linking based on a first crawl of the website.

You also don’t have to show these links to users if you don’t want to, but the position of the internal links is probably what determines the success of this approach.

Just to emphasise, this all relies on you having your other basic SEO correct. You need ways for this content to receive PageRank and anchor text and all other “good” signals once it’s beyond the very temporary limelight.

Thanks for reading!

If you’ve read this far, share the damn post.