How robots.txt Rules Match URLs: Paths, Wildcards and Precedence
A robots.txt file looks simple, but small details decide whether a crawler may fetch a URL. The crawler first chooses which group of rules applies to it, then compares each rule with the URL path. Understanding both steps prevents accidental blocking of whole sections and makes rule changes predictable.
Choosing the group that applies
Rules are grouped under one or more user-agent lines. A crawler follows the group whose user-agent token matches it most specifically and ignores the rest. A group for a named crawler therefore replaces the generic asterisk group for that crawler; rules are not added together from both.
Keep groups short and explicit. If a named group exists only to add one extra rule, copy the generic rules into it as well, otherwise the crawler loses every restriction that lived only in the asterisk group.
Matching paths, wildcards and the end anchor
Rules match from the start of the URL path and are case-sensitive, so a rule for /Private/ does not block /private/. An asterisk matches any sequence of characters, and a dollar sign anchors the rule to the end of the URL. Query strings are part of the matched value.
When several rules match, Google uses the most specific rule, meaning the one with the longest path. If an allow rule and a disallow rule are equally specific, the less restrictive allow rule wins. Other crawlers may resolve conflicts differently, so avoid relying on ties.
User-agent: *
Disallow: /search
Allow: /search/help$
Disallow: /*.pdf$Testing a change before publishing it
List the URLs that must stay crawlable and those that must be blocked, then evaluate every rule change against both lists with a parser that follows the published specification. Review the file after deployment as well, because a staging file accidentally copied to production can block an entire site.
Remember that robots.txt controls crawling, not indexing. A blocked URL that is linked from elsewhere can still appear in results without its content. Use a noindex directive on a crawlable page when the goal is to keep it out of the index.