What is a robots.txt file?

A robots.txt file is a text file that tells search engine crawlers which pages or files the crawler can or can't request from your site. It's part of the Robots Exclusion Protocol (REP), a standard used by websites to communicate with web crawlers.

Where should I place my robots.txt file?

Your robots.txt file should be placed in the root directory of your website (e.g., https://example.com/robots.txt). This is the first file that search engines look for when crawling your site.

Can I block specific search engines with robots.txt?

Yes, you can use the 'User-agent' directive to target specific search engines. For example, 'User-agent: Googlebot' will only apply to Google's crawler, while 'User-agent: *' applies to all crawlers.

What should I block in robots.txt?

Common paths to block include admin areas, login pages, search result pages, private or temporary files, duplicate content pages, and development or staging environments. This helps protect sensitive areas and prevents search engines from indexing unnecessary content.

How do I test my robots.txt file?

You can test your robots.txt file using Google Search Console's robots.txt Tester tool. This tool allows you to verify if specific URLs are blocked or allowed for different user agents. Our tool also provides a direct link to this testing tool.

Can I use robots.txt to hide content from search engines?

While robots.txt can prevent search engines from crawling certain pages, it should not be used as a security measure. Content blocked by robots.txt can still be accessed by users who know the URL. For sensitive content, use proper authentication and authorization methods.

Robots.txt Generator

Create and customize robots.txt files to control search engine crawling

Sitemap URL

Optional: Add your XML sitemap location

User-agent Rules

Disallow:

Additional Options

Include Crawl-delay directive

User-agent Control

Target specific search engines or apply rules to all crawlers with customizable user-agent directives.

Disallow Rules

Prevent search engines from crawling specific pages or directories with easy-to-configure disallow rules.

Sitemap Declaration

Point search engines to your XML sitemap for more efficient crawling and indexing of your website.

Crawl-delay Settings

Control how frequently search engines crawl your site to prevent server overload and manage resources.

Best Practices

Keep it Simple

Avoid overly complex robots.txt files that might confuse search engines. Keep your directives clear and straightforward.

Include Sitemap Reference

Always include a reference to your XML sitemap in your robots.txt file to help search engines discover all your pages.

Test Your Robots.txt

Use search engine webmaster tools to test your robots.txt file for errors and ensure it's working as expected.

Protect Private Areas

Use robots.txt to prevent search engines from indexing private areas of your site, such as admin pages or user data.

About Robots.txt

A robots.txt file is a plain-text crawl policy at your site root. Compliant crawlers read it to learn which path prefixes they should avoid fetching. This generator helps you assemble clear User-agent groups, Allow/Disallow rules, and Sitemap: pointers you can download and host at /robots.txt. It does not hide content from people who already know a URL, and it is not a substitute for authentication, noindex, or canonical consolidation.

Deep guide: writing robots.txt that reduces crawl noise without breaking rendering

What robots.txt controls (and what it does not)

Robots.txt governs fetching. A disallowed URL may still appear in search results if other sites link to it—often without a snippet. Removing a URL from the index usually requires noindex (meta or HTTP header) while the URL remains crawlable, or removing/blocking access entirely with auth. Never put secrets in a “blocked” path and assume they are private.

When to use this generator

You need a clean starter file for a new domain or migration.
You want to disallow noisy parameter paths, internal search result URLs, or cart/checkout drafts that waste crawl budget.
You are documenting sitemap locations for Google, Bing, and other compliant bots.
You need separate rules for a specific user-agent group after confirming how that bot identifies itself.
You are teaching teammates the difference between disallow, noindex, and canonical.

When not to use robots.txt

Do not rely on it to protect private user data—use login and authorization.
Do not disallow CSS/JS required to render page content if you care how Google understands the page.
Do not use disallow as your only de-indexation method when the URL is widely linked.
Avoid inventing exotic patterns you cannot test; prefer simple path prefixes.
Do not place robots.txt in a subdirectory—only the root file is used by spec.

Worked example

An ecommerce site wants bots to crawl products and guides, but not internal search, faceted filter combinations that create infinite URLs, and account areas. A practical robots.txt sketch:

User-agent: *
Disallow: /account/
Disallow: /cart/
Disallow: /checkout/
Disallow: /search
Disallow: /*?sort=
Allow: /

Sitemap: https://www.example.com/sitemap.xml
Sitemap: https://www.example.com/sitemap-images.xml

After deploying, test representative product, category, and blocked URLs in Search Console’s robots testing workflows (or equivalent). Confirm CSS and JS on product templates remain allowed. If a thin filtered URL must stay out of the index but remain crawlable for other reasons, prefer noindex plus stronger internal linking to the canonical category—see robots vs noindex vs canonical.

Interpretation guide

User-agent groups: Rules apply to the group until the next user-agent line. Most sites start with * and add exceptions only when needed.
Disallow / Allow: Longest matching prefix rules win in many implementations; keep patterns readable. Test edge cases rather than assuming.
Sitemap lines: Absolute HTTPS URLs to sitemap or sitemap index files. Listing them here helps discovery; you should still submit in webmaster tools.
Crawl-delay: Not universally honored (Google ignores it). Prefer server-side rate controls if bots overwhelm origin.

Common mistakes

Disallow: / on production by accident after copying a staging file.
Blocking /wp-admin while also blocking assets needed for public rendering.
Assuming case does not matter—path matching is case-sensitive on many servers.
Using robots.txt to “fix” duplicate content instead of canonicals and parameter handling.
Forgetting to update Sitemap lines after moving hosts or renaming sitemap files.

Common Directives

User-agent: Specifies which crawler the rules apply to
Disallow: Tells crawlers not to fetch matching path prefixes
Allow: Explicitly permits a narrower path inside a broader disallow
Sitemap: Points to your XML sitemap or sitemap index location
Crawl-delay: A delay hint some bots may honor; not a universal control

Decision table: disallow vs noindex vs auth vs canonical

Goal	Prefer	Why
Stop wasting crawl on infinite facet URLs	Disallow carefully tested prefixes	Reduces fetch volume; pair with cleaner internal links
Keep URL crawlable but out of the index	`noindex` (meta or header)	Disallow alone often leaves linked URLs visible without a useful snippet
Protect account data / PII	Authentication + authorization	Robots.txt is public and does not secure resources
Consolidate near-duplicates	Canonical + consistent internal links	Disallow is the wrong lever for duplicate preference
Retire a URL permanently for users and bots	301 to the replacement	Clearer than hoping disallow removes demand

For the full decision framework, see robots, noindex, and canonical and the SEO guide.

Before / after: a staging leak that blocked production

Before: A developer copied a staging robots.txt that began with User-agent: * and Disallow: / onto the live host during a Friday deploy. Search Console crawl stats collapsed; money pages stopped refreshing snippets.

After: Rollback restored the previous file listing only account, cart, checkout, and internal search disallows plus two Sitemap: lines. A postmortem added a deploy checklist item: never promote robots.txt without a diff review and a root-URL “allowed” test for homepage, product, and blog templates.

CMS and hosting placement notes

WordPress: Prefer a single source of truth (SEO plugin or physical file). Competing plugin rules can overwrite each other on save.
Shopify / SaaS storefronts: Many platforms generate robots.txt for you; customization may be limited or theme-app based—verify the live /robots.txt after edits.
Static / Jamstack: Commit the file to the site root of the production build artifact, not only the repo docs folder.
CDN / multi-host: www and apex are different hosts unless redirected; each needs its own correct robots.txt if both answer.
Staging: Blocking staging is fine; never copy that blocklist to production without stripping the sitewide disallow.

Edge cases worth testing explicitly

Disallowing /wp-admin is common; accidentally disallowing /wp-content or critical asset paths can break rendering understanding.
Trailing-slash vs non-slash paths: your server may treat them as different URLs; match the prefixes users and sitemaps actually use.
Encoded characters and case differences: /Cart vs /cart may not both match your rule.
Allow exceptions inside a broad disallow—test the exception URL, not only the blocked parent.
Sitemap lines that still point at HTTP or an old subdomain after a migration quietly waste discovery signals.

QA checklist before you call robots.txt done

Live file is at the host root over HTTPS and returns text/plain (or acceptable plain text).
Homepage, category, product/article, and key asset URLs remain allowed for Googlebot.
Account, cart, checkout, internal search, and known infinite facet spaces are blocked only after spot tests.
Sitemap: URLs resolve and match what you submit in Search Console / Bing.
Diff reviewed against the previous production file; rollback copy saved.
Documented in your technical hygiene notes so the next deploy does not reinvent rules.

Frequently Asked Questions

What should I commonly disallow?

Typical candidates include account areas, carts/checkouts, internal site-search result URLs, pure admin paths, and known infinite facet URL spaces—after you confirm public pages and required assets stay allowed. Do not disallow everything “private-looking” without testing rendering and indexation side effects.

Where must robots.txt live?

At the site root only—for example https://www.example.com/robots.txt. A file in a subdirectory is not the robots.txt for that host. Host and protocol matter: www and non-www are different hosts unless redirected.

Does disallow remove a page from Google’s index?

Not reliably. Disallow stops crawling; linked URLs can still appear. To request removal from search results while allowing a crawl, use noindex. For sensitive content, use authentication. See our guide on choosing robots, noindex, and canonical.

How do I test robots.txt safely?

Deploy carefully (avoid blocking the whole site), then test specific URLs and user-agents in Google Search Console or Bing Webmaster tools. Spot-check that CSS/JS for key templates remain fetchable. Keep a rollback copy of the previous file.

Is robots.txt case sensitive?

Directive names are conventionally written as User-agent, Disallow, Allow, and Sitemap. Path matching follows the server’s URL path casing. Prefer consistent lowercase paths in your site architecture to reduce surprises.

Further Information

Search Engine Guidelines

Related Tools

Learning Resources