Syntax and precedence rules
The file must live at /robots.txt on each host (a subdomain needs its own) and be UTF-8 encoded. It is made of groups: a User-agent line names the crawler (an asterisk targets all of them), followed by Disallow and Allow lines listing paths. A Sitemap line can point to your XML sitemap.
The protocol was standardized by the IETF as RFC 9309 in 2022, with precise rules: the most specific rule, meaning the longest matching path, wins, and when an Allow and a Disallow rule are equally long, Allow should be used. Crawlers must parse at least 500 KiB of the file and should not use a cached copy for more than 24 hours.
What robots.txt does not do
Google is explicit: robots.txt is not a way to keep a page out of its results. To deindex a page, use a noindex tag or password protection, and leave the page crawlable, otherwise Google never sees the tag. The file is not a security measure either: it is public and sometimes advertises sensitive paths such as /old-admin/.
When the file returns an error
The status code matters as much as the content. If /robots.txt returns a 4xx error such as a 404, crawlers assume there are no restrictions. If it returns a 5xx server error, they must assume the entire site is disallowed. A misconfiguration on this single file can therefore halt crawling of your whole website.
Robots.txt and AI crawlers
AI companies publish their own user agents: GPTBot for OpenAI, ClaudeBot for Anthropic, PerplexityBot for Perplexity. At Google, the Google-Extended token controls whether your content is used to train and ground its models outside of Search. Blocking Google-Extended does not remove your site from search results or AI Overviews, which rely on Googlebot.