Search Console Crawl Stats URL Parameters

One of the lovely parts of Google Search Console is the Settings > Crawl Stats report suite:

And something I only noticed today is the unexpectedly friendly URL structure:

/search-console/settings/crawl-stats/drilldown?resource_id=sc-domain%3Aohgm.co.uk&response=1

If you haven’t had enough coffee today, this probably means the following will do something:

/search-console/settings/crawl-stats/drilldown?resource_id=sc-domain%3Aohgm.co.uk&response=2

and it does.

(I don’t usually see the parameters at the end of the URL because I have too many pinned extensions in Chrome).

This post is just me finding something interesting, rather than uncovering this THING they don’t want you to know.

And the documentation is really good here: https://support.google.com/webmasters/answer/9679690?hl=en

So it’s not like I’m discovering much here. Enjoy.

&response=

I am just going to run through and list the responses for each parameter.

1. OK (200)
2. Not found (404)
3. Unauthorised (401/407)
4. Other client error (4xx)
5. DNS error
6. Fetch Error
7. Blocked by robots.txt
8. This returns a 400 for some reason.
9. Server error (5xx)
10. DNS unresponsive
11. robots.txt not available
12. Page could not be reached
13. Page timeout
14. This returns a 400 for some reason.
15. Redirect error
16. Other fetch error
17. Moved permanently (301)
18. Moved temporarily (302)
19. Moved (other)
20. Not modified (304)

&file_type=

1. HTML
2. Image
3. Video
4. JavaScript
5. CSS
6. PDF
7. Other file type
8. This returns a 400 for some reason.
9. Unknown (failed requests)
10. Other XML
11. JSON
12. Syndication
13. Audio
14. Geographic data

&purpose=

1. Discovery
2. Refresh
3. Other fetch purpose

( ͡° ͜ʖ ͡°)

&googlebot_type=

1. Smartphone
2. Desktop
3. Image
4. Video
5. Page resource load
6. Other agent type
7. Adsbot
8. Storebot
9. AMP

Anything interesting?

Yes, please have a look yourself. I will dig in more once I have the will.

• Why are a few of these options returning a 400?
• I’ve never seen many of these in the wild – are they all functional or placeholders for planned/deprecated/currently broken reports?

For example, I don’t think I’ve ever seen Blocked by robots.txt (&response=7) show up in this report before. It does show up in Coverage with ‘last crawl’ dates (I had a look at this back in 2018 to see whether blocked crawl ‘attempts’ were recorded):

But this particular crawl stats report doesn’t seem to be populated for any sites I currently have access to:

And you would expect it to be if we’re getting the coverage report showing recently ‘crawled’ URLs.

The documentation indicates it probably is intended to function:

Maybe it works for you, let me know! It might be temporarily broken (and I’ve previously been inattentive).

I hope some tool providers can use these URL formats for more efficient scraping.

This post could be a tweet thread, but if I did that, I’d never find it ever again.