Conditional Requests for Crawlers: ETag, Last-Modified and 304
Crawlers revisit pages that have not changed far more often than pages that have. Conditional requests let a server answer those visits with a short 304 Not Modified response instead of sending the full page again. Both the crawler and the server save work, and that capacity can go toward pages that did change.
Validators: ETag and Last-Modified
A server sends an ETag header containing an identifier for the current version of a response, and a Last-Modified header with the time the content last changed. On a later visit, a client can send If-None-Match with the ETag or If-Modified-Since with the date.
Under the HTTP standard, when both conditional headers arrive, If-None-Match takes precedence. Google's crawlers support both validators.
GET /guides/caching/ HTTP/1.1
If-None-Match: "v42-3f9a"
HTTP/1.1 304 Not Modified
ETag: "v42-3f9a"Making validators accurate
A validator is only useful if it changes when the content changes and stays stable when it does not. Avoid ETags that include a server node identifier, which change between machines behind a load balancer, and avoid Last-Modified values set to the current time on every request.
Base the value on the content that matters, such as a hash of the rendered main content or the last edit time stored with the page.
Checking the behaviour
Request a page twice, sending the returned validator on the second request, and confirm the server answers 304 with no body. Then edit the page and confirm the next conditional request receives a full 200 response with a new validator.
In logs, a healthy share of 304 responses to crawler revisits on stable pages shows that validation works. A share of zero usually means validators are missing or unstable.