HTTP Status Codes Crawlers Act On: 200, 3xx, 404, 410 and 5xx
The status code is the first thing a crawler reads in a response, before any HTML. It tells the crawler whether the page exists, has moved, is gone or cannot be served right now. Returning an accurate code is one of the cheapest ways to keep an index aligned with a site.
Success and redirects
A 200 response tells the crawler the content in the body is the page. Do not return 200 for error messages or empty templates, because the crawler will treat them as real pages, which often leads to soft 404 reports.
Permanent redirects, 301 and 308, signal that the target URL should replace the old one and pass canonical signals to it. Temporary redirects, 302 and 307, suggest the original URL should stay the indexed one. Pick the code that describes the real intent, not the one a framework happens to default to.
Removed pages: 404 and 410
Both 404 and 410 tell a crawler the page is not available. Google treats them very similarly: the URL is eventually dropped from the index and crawled less often over time. A 410 states the removal is intentional, which can help people and tools reading logs, but it is not a faster removal switch.
Removed pages do not harm the rest of a site. Avoid redirecting every missing URL to the home page; that pattern is usually classified as a soft 404 anyway.
Server errors and rate limiting
Responses in the 5xx range and 429 Too Many Requests tell crawlers the server is struggling. Crawlers slow down in response, and if errors continue for a long time, previously indexed URLs can be dropped. A 503 with a Retry-After header is appropriate for short planned maintenance.
Monitor error rates by crawler and by path. A spike limited to one template usually points to an application bug rather than real capacity limits.
HTTP/1.1 503 Service Unavailable
Retry-After: 3600