Faceted Navigation and URL Parameters: Containing Crawl Waste
Filters for size, color, price and sorting help visitors narrow a catalog, but each combination can create a new URL. A few filters with a handful of values multiply into thousands or millions of addresses, most showing near-duplicate lists. Without limits, crawlers spend their time on these combinations instead of on product and article pages.
Deciding which combinations deserve URLs
Start from what people search for. A category filtered by a single popular attribute, such as running shoes in a specific size range, may deserve its own indexable page with a descriptive title. Combinations of several filters, sort orders and view settings rarely do.
Document the indexable combinations explicitly so developers and editors share the same list.
Controlling the rest
Keep parameter order consistent so the same selection always produces the same URL. For combinations that should not be crawled at all, apply robots.txt rules to the parameter patterns. For combinations people may share but that should not be indexed, use a canonical pointing to the main category or a noindex directive.
Google retired the URL Parameters tool in Search Console in 2022, so these controls now need to live in the site itself.
User-agent: *
Disallow: /*?*sort=
Disallow: /*?*sessionid=Measuring crawl waste
Count verified crawler requests in server logs by whether the URL contains parameters, and by which parameters appear. If a large share of requests goes to sorted or multi-filter URLs while new products wait to be crawled, tighten the rules.
Recheck after each change, because a new filter launched without rules can reintroduce the problem quickly.