Robots.txt is a plain text file at the root of your domain that tells crawlers which parts of your site they may request. It is about four lines of text on most sites, & it is capable of removing a business from search results entirely if someone writes it carelessly.
It is worth understanding properly, because almost every serious SEO incident we have been called in to fix has involved this file or a stray noindex tag.
Where it lives & how it is read
The file must sit at the root of the host: https://example.com/robots.txt. A file at https://example.com/blog/robots.txt is ignored entirely. Subdomains need their own, so shop.example.com is governed by its own file & not by the one on the main domain.
Before crawling, a well behaved bot requests this file, reads the group that applies to it, & obeys it. The critical phrase is "well behaved". Robots.txt is a convention, not a security control. Compliant crawlers respect it. Scrapers & malicious bots do not. Never use it to protect anything sensitive, because you are publishing a list of the directories you would rather people did not look at.
The syntax that actually matters
User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /account/
Allow: /account/public-profile
Sitemap: https://example.com/sitemap.xml
User-agent names the crawler the group applies to. An asterisk means all crawlers that do not have their own group.
Disallow gives a path prefix the crawler should not request. An empty Disallow: means nothing is blocked.
Allow carves an exception out of a broader disallow. Google resolves conflicts by the most specific rule, not by order, so the Allow above wins for that one URL.
Sitemap points to your XML sitemap. It is independent of user-agent groups & can appear anywhere in the file.
Wildcards
Google & Bing support two pattern characters. An asterisk matches any sequence, & a dollar sign anchors the end of the URL.
User-agent: *
Disallow: /*?sort=
Disallow: /*.pdf$
The first blocks any URL containing that sort parameter. The second blocks URLs ending in .pdf, without touching a URL that merely contains ".pdf" somewhere in the middle.
One trap worth knowing
Only the single most specific matching user-agent group applies. If you write a group for Googlebot & another for *, Googlebot reads its own group & ignores the wildcard group completely. Rules you assumed were global will not apply. This catches people out constantly.
Blocking is not deindexing
This is the single most important thing in this article.
Disallow stops a crawler requesting a page. It does not remove that page from search results. If other sites link to a blocked URL, Google can still index it based on those links alone, & will show it with no description because it was never allowed to read the content.
Worse, blocking a page in robots.txt prevents Google from ever seeing a noindex tag on that page. If you want a page out of the index, you must let the crawler in so it can read the instruction.
The correct choices are:
- To keep a page out of search results: allow crawling & serve
<meta name="robots" content="noindex">or anX-Robots-Tag: noindexheader. - To stop wasting crawl requests on infinite or worthless URL spaces: use
Disallow. Faceted navigation & internal search results are the classic cases. - To remove something urgently: use the Removals tool in Search Console for a temporary suppression, then apply
noindexas the permanent fix.
Never combine Disallow with noindex on the same URL. They cancel each other out, & the page stays indexed.
What to block, & what to leave alone
Reasonable to block: cart, checkout & account areas, internal search result pages, endless filter & sort combinations, staging environments, & admin paths.
Do not block: CSS & JavaScript files. Google renders pages the way a browser does. If you block the assets that build the layout, it sees a broken page & judges it accordingly. This was a widespread mistake years ago & it still appears in files copied from old templates.
Also leave alone anything you want ranking. It sounds obvious, but Disallow: / on a live site is one of the most common catastrophic errors in the industry, & it usually arrives when a staging file is deployed to production. If your traffic ever falls off a cliff overnight, check this file before anything else.
AI crawlers in 2026
The newer question is whether to block the crawlers that gather training & retrieval data for AI systems. The main named agents include GPTBot & OAI-SearchBot from OpenAI, ClaudeBot from Anthropic, Google-Extended for Gemini training, PerplexityBot, CCBot for Common Crawl, & Bytespider.
Blocking them looks like this:
User-agent: GPTBot
Disallow: /
User-agent: CCBot
Disallow: /
Whether you should is a genuine business decision, & it deserves more thought than it usually gets.
The argument for blocking is straightforward. Your content is an asset, models train on it without compensating you, & answers assembled from it may satisfy the user without ever sending them to your site.
The argument against is that being retrievable & citable by these systems is now a visibility channel. If a potential customer asks an assistant to recommend an agency in Kathmandu, you cannot be the answer if you have told the crawler to go away. Blocking is not free. It removes you from a surface where your competitors will still appear.
One important distinction: Google-Extended controls use in Gemini & AI training, & blocking it does not affect your normal Google Search rankings. Those are separate. Many sites want that separation & do not realise the control exists.
Our general position is to allow retrieval agents that can cite & link back, & to be more selective about pure training crawlers. But this depends on whether your content is the product or the marketing, & reasonable businesses land differently on it.
Crawl-delay & other non standard directives
Google ignores Crawl-delay entirely. Use the crawl rate setting in Search Console if you genuinely need to slow it down. Bing & Yandex do respect the directive.
Noindex: as a robots.txt directive was never officially supported & Google stopped honouring it in 2019. If you find it in a file you have inherited, it is doing nothing.
Testing before you deploy
Because the downside is so severe, never edit this file on production without checking it.
- Use the robots.txt report in Google Search Console to confirm what Google is currently reading. It caches the file for around 24 hours, so changes are not instant.
- Test specific URLs against the proposed rules before publishing.
- After any deploy, request
/robots.txtyourself & read it. Staging files reaching production is the most common cause of disaster here. - Monitor it. A weekly automated check that alerts you if the file changes or starts returning
Disallow: /costs almost nothing & has saved sites we work with.
A sensible starting file
User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /account/
Disallow: /*?s=
Disallow: /search/
Sitemap: https://example.com/sitemap.xml
That is genuinely enough for most sites. Robots.txt rewards restraint. Every line you add is a line that can go wrong, & the sites that get into trouble with this file are almost never the ones that kept it short.