Robots.txt Generator

What robots.txt does

robots.txt is a plain text file at the root of your domain that tells crawlers which parts of the site they may request. It is the first file most search engine bots fetch, before a single page, so a mistake in it affects everything that follows.

This generator builds a valid file from a few choices: allow normal crawling with a list of paths to keep out, or block everything while a site is in staging, plus optional rules for specific bots and a sitemap reference.

The syntax, briefly

Rules are grouped under a User-agent line. "User-agent: *" applies to every crawler that does not have its own group. Disallow blocks a path prefix, Allow re-permits something inside a blocked area, and both are matched from the start of the URL path.

"Disallow: /admin/" blocks everything beneath /admin/. "Disallow: /" blocks the entire site. An empty Disallow line blocks nothing at all. The Sitemap directive is independent of user-agent groups and should list the full absolute URL of your sitemap.

What belongs in the file, and what does not

Good candidates to disallow: admin areas, shopping carts and checkouts, internal search result URLs, faceted filter parameters that create endless combinations, and staging copies of the site.

Bad candidates: CSS and JavaScript files, which Google needs in order to render and judge your pages, and any page you want removed from search results. Blocking a page in robots.txt stops it being crawled, not indexed — it can still appear in results as a bare URL. To remove a page, allow crawling and use a noindex meta tag instead.

Controlling AI crawlers

Alongside search bots, a growing number of crawlers collect content for AI training and retrieval — GPTBot, CCBot, ClaudeBot, Google-Extended and others. They are separate user agents, so you can decline them individually while leaving Googlebot and Bingbot free to index your site normally.

This is a policy decision, not an SEO one. Blocking AI crawlers protects content from being used for training but can also remove you from AI answers that cite sources. Decide deliberately rather than by default.

Common mistakes that cost traffic

Shipping a staging robots.txt to production with "Disallow: /" still in place is the classic disaster: the site vanishes from search within days. Check the live file immediately after every launch.

Other frequent faults are placing the file anywhere other than the domain root, using a relative sitemap path, adding Crawl-delay for Googlebot (which ignores it), and assuming robots.txt provides security. It does not — the file is public, so listing a secret directory in it advertises the directory to anyone who reads it.

Installing and testing it

Save the generated text as robots.txt in the root of your site so it is served at https://yourdomain.com/robots.txt, with content type text/plain. Visit that URL in a browser to confirm it is live.

Then test a few important URLs in Google Search Console's robots.txt report to make sure nothing you want indexed is accidentally blocked, and submit your sitemap in the same tool.

Privacy

The file is assembled locally in your browser and never sent anywhere, so you can plan rules for an unreleased site without exposing its structure.

What robots.txt can and cannot do

A robots.txt file is a request, not a lock. Well-behaved crawlers — Google, Bing, and the reputable search engines — obey it. Scrapers and malicious bots read it too and simply ignore it. If a page must truly stay private, password-protect it or keep it off the public web; robots.txt is the wrong tool for secrecy.

It also does not guarantee a page stays out of search results by itself. If other sites link to a blocked page, Google can list the URL with no description, because it never crawled the content to see a noindex tag. When the goal is 'do not index this page', allow crawling and use a noindex meta tag instead; when the goal is 'do not waste crawler effort here', robots.txt is right.

The rules worth knowing before you generate

Rules are matched per user-agent, and the most specific matching group wins. A block under User-agent: Googlebot does not apply to Bingbot, and a group for User-agent: * applies only to crawlers without a more specific group. Paths match from the start of the URL path, so /admin blocks /admin, /admin/ and /administrator unless you add an Allow rule with a longer, more specific path.

The most common accident is blocking an entire staging site with Disallow: / and later copying that file to production, which quietly removes the whole site from search. The second is blocking CSS or JavaScript files that Google needs to render pages — blocked resources can hurt how your pages are understood. The generator's defaults avoid both traps.

Sitemaps and robots.txt belong together

The Sitemap line in a robots.txt file tells every crawler where your full list of pages lives, regardless of how it found your site. It is one of the few lines that helps every search engine at once, and it costs one line of text.

A good default setup for most sites is: allow everything, disallow admin and account pages, disallow internal search result pages, and list your sitemap. That is the preset this generator starts from, and it suits the large majority of sites without further editing.