Robots.txt Generator

Create and customize robots.txt files to control search engine crawling

Optional: Add your XML sitemap location
Disallow:
seconds

Generated Robots.txt

User-agent Control
Target specific search engines or apply rules to all crawlers with customizable user-agent directives.
Disallow Rules
Prevent search engines from crawling specific pages or directories with easy-to-configure disallow rules.
Sitemap Declaration
Point search engines to your XML sitemap for more efficient crawling and indexing of your website.
Crawl-delay Settings
Control how frequently search engines crawl your site to prevent server overload and manage resources.

Best Practices

Keep it Simple
Avoid overly complex robots.txt files that might confuse search engines. Keep your directives clear and straightforward.
Include Sitemap Reference
Always include a reference to your XML sitemap in your robots.txt file to help search engines discover all your pages.
Test Your Robots.txt
Use search engine webmaster tools to test your robots.txt file for errors and ensure it's working as expected.
Protect Private Areas
Use robots.txt to prevent search engines from indexing private areas of your site, such as admin pages or user data.

About Robots.txt

A robots.txt file tells search engine crawlers which pages or files the crawler can or can't request from your site. This is used mainly to avoid overloading your site with requests and to prevent search engines from indexing certain pages.

Practical guide: robots.txt limits and strengths

What this is

Robots.txt is a plain-text policy file at the site root that tells compliant crawlers which URL paths they should not fetch. It governs crawling, not secrets: blocked URLs can still appear in search results if linked elsewhere, and it does not block users who know the link. Use it to reduce crawl noise, protect fragile endpoints, and publish sitemap locations—not as your only privacy control.

How to use it

Build rules per User-agent group, then test with Search Console’s robots tester or fetch-as scenarios you care about. Keep directives simple and avoid conflicting Allow/Disallow patterns that are hard to reason about. Include absolute Sitemap: lines for each index you maintain. After edits, deploy to /robots.txt only—subfolder copies are ignored by spec.

How to read the results

Most engines respect robots.txt for crawl scheduling, but interpretation can differ for edge patterns. A disallow stops fetches; it does not equal noindex. If you need a URL out of the index while still crawlable for some reason, use meta robots or HTTP headers instead. Crawl-delay is not universally honored—treat it as a hint at best.

Common mistakes

Blocking CSS or JavaScript needed for rendering can harm how Google understands the page. Assuming disallow removes a URL from the index leads to surprises when external links exist. Wildcards and case sensitivity trip up teams migrating platforms. Duplicating sitemap lines dozens of times adds noise, not power. Pair robots.txt with canonicals, redirects, authentication, and noindex where each is appropriate.

Common Directives

Frequently Asked Questions

What should I block in robots.txt?
Common paths to block include:
  • Admin areas and login pages
  • Search result pages
  • Private or temporary files
  • Duplicate content pages
  • Development or staging environments
Where should I place the robots.txt file?
The robots.txt file must be placed in the root directory of your website (e.g., https://example.com/robots.txt). It won't work if placed in a subdirectory.
Is robots.txt case sensitive?
Yes, robots.txt directives are case-sensitive. For example, "Disallow" and "disallow" are treated differently. Always use the proper case: "User-agent", "Disallow", "Allow", "Sitemap".
How do I test my robots.txt file?
You can test your robots.txt file using Google Search Console's robots.txt Tester tool. This tool allows you to verify if specific URLs are blocked or allowed for different user agents. Our tool also provides a direct link to this testing tool.
Can I use robots.txt to hide content from search engines?
While robots.txt can prevent search engines from crawling certain pages, it should not be used as a security measure. Content blocked by robots.txt can still be accessed by users who know the URL. For sensitive content, use proper authentication and authorization methods.

Further Information

Search Engine Guidelines
Related Tools
Learning Resources