Server Errors and Crawl Rate: How 5xx and 429 Slow Crawling
Search crawlers try to fetch as much as a site can comfortably serve. They watch response times and error codes and adjust their pace accordingly. When a server responds slowly or returns errors, crawling slows down, which means new and updated pages take longer to be noticed.
How crawlers react to errors
A rise in 500, 502, 503 or 429 responses tells the crawler to back off. The effect lasts while the errors continue, and crawl rate recovers gradually after the server stabilizes. Long periods of errors on specific URLs can lead to those URLs being dropped from the index.
Errors on robots.txt itself are especially important. If the file cannot be fetched because of a server error, a crawler may pause crawling the whole site until it can read the rules again.
Measuring the problem
Break down responses to verified crawler requests by status code, path pattern and hour. Compare median and slow-percentile response times with ordinary visitor traffic. A crawler that hits uncached pages may see much slower responses than people browsing popular, cached URLs.
Crawl statistics in search consoles show trends, but server logs show which templates and endpoints are responsible.
awk '$9 ~ /^5/ {print $7}' access.log | sort | uniq -c | sort -rn | headProtecting capacity safely
Fix the slow or failing endpoints first; caching expensive pages often solves both visitor and crawler problems. For short emergencies, returning 503 or 429 is an accepted way to reduce load, but it should not stay in place for days.
Avoid blocking crawlers in robots.txt to save capacity unless the blocked URLs are truly low value, because blocking removes the pages from crawling rather than slowing it.