The Complete robots.txt Guide: Syntax, Templates & Free Generator

Why you need a well-configured robots.txt file

A robots.txt file is the gateway to your website for search engines. It sits at the root of your domain (e.g., yourwebsite.com/robots.txt) and provides immediate instructions to web crawlers about which pages they are allowed to visit and which they should ignore.

While we have previously covered what robots.txt can and cannot do for SEO and how search crawlers actually read the rules, this guide serves as a practical, action-oriented reference. If you are looking for syntax definitions, copy-paste templates for common CMS platforms, or instructions on how to block AI scrapers, you are in the right place.

The core syntax: how to write robots.txt directives

A robots.txt file consists of two primary components: the User-agent (who the rule applies to) and the Directive (what they are or are not allowed to do).

User-agents

The User-agent line specifies which bot the following rules apply to. You can target all bots or specific ones.

Directives

Directives tell the specified user-agent what to do. They must immediately follow the user-agent line.

Pattern matching (Wildcards)

You can use wildcards for more complex URL matching:

• * (Asterisk): Matches any sequence of characters. E.g., Disallow: /*.pdf$ blocks all PDF files.
• $ (Dollar sign): Denotes the end of a URL. E.g., Disallow: /search$ blocks /search but allows /search/results.

Copy-paste templates for common setups

Instead of writing rules from scratch, you can adapt these templates for common website configurations.

1. The universal default (Allow everything)

If you have a small, static website and want search engines to index everything, this is all you need:

User-agent: *
Disallow:

Sitemap: https://www.yourdomain.com/sitemap.xml

2. The WordPress standard

WordPress sites need to block core system files and admin areas while allowing search engines to crawl public content and necessary assets.

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-includes/
Disallow: /search/
Disallow: /?s=
Disallow: /author/

Sitemap: https://www.yourdomain.com/sitemap_index.xml

3. The E-commerce / Shopify baseline

E-commerce sites need to prevent search engines from crawling shopping carts, checkout pages, and internal search results to avoid duplicate content and wasted crawl budget.

User-agent: *
Disallow: /cart
Disallow: /checkout
Disallow: /orders
Disallow: /search
Disallow: /*?sort_by=
Disallow: /*?filter=

Sitemap: https://www.yourdomain.com/sitemap.xml

4. Blocking AI crawlers and scrapers

If you want to prevent AI companies from using your content to train large language models (LLMs), you need to block their specific user-agents.

# Block OpenAI
User-agent: GPTBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /

# Block Google's extended AI crawler
User-agent: Google-Extended
Disallow: /

# Block Anthropic (Claude)
User-agent: anthropic-ai
Disallow: /
User-agent: Claude-Web
Disallow: /

# Block Common Crawl (used by many AI models)
User-agent: CCBot
Disallow: /

# Allow normal search engines
User-agent: *
Disallow: /admin/

5. Single Page Applications (SPAs) / Nuxt / Next.js

Modern JavaScript frameworks often use API routes or specific directories for internal routing that shouldn't be indexed directly.

User-agent: *
Disallow: /api/
Disallow: /_nuxt/
Disallow: /_next/
Disallow: /build/

Sitemap: https://www.yourdomain.com/sitemap.xml

Generate your robots.txt automatically

Writing syntax manually can lead to typos that accidentally de-index your entire website. The safest approach is to use a dedicated tool.

The CampaignMorph Robots.txt Generator allows you to build a valid, error-free file in seconds.

How to use the generator:

1. Select default rules: Choose preset configurations for WordPress, Shopify, or generic sites.
2. Add custom paths: Use the interface to add specific directories you want to allow or disallow.
3. Toggle AI blocking: Use the built-in toggles to instantly add blocks for GPTBot, Google-Extended, and other AI scrapers.
4. Add your sitemap: Paste your absolute sitemap URL.
5. Copy and deploy: Copy the generated text and save it as robots.txt in the root directory of your website.

Common robots.txt mistakes to avoid

1. Blocking CSS and JavaScript files: Search engines need to render your page to understand it fully. If you block CSS or JS directories (e.g., Disallow: /css/), Google may see a broken layout and rank the page lower for mobile-friendliness.
2. Using robots.txt to hide sensitive data: A Disallow rule does not secure a page; it only asks polite bots not to crawl it. Malicious scrapers will ignore the rule. Worse, by listing the path in robots.txt, you are publicly broadcasting the location of your sensitive files. Use password protection or server-level authentication instead.
3. Conflicting Allow and Disallow rules: If you have Disallow: /blog/ and Allow: /blog/post-1/, make sure the user-agent targeting is clear and you understand how different crawlers interpret precedence.
4. Forgetting the leading slash: Disallow: admin/ is invalid. It must be Disallow: /admin/.
5. Not testing changes: Before deploying a new robots.txt, test it using the Robots Testing Tool in Google Search Console to ensure you haven't accidentally blocked critical pages.

For a deeper dive into the technical nuances of crawler behavior, read how search crawlers read robots.txt rules.

Next steps for Technical SEO

Configuring your robots.txt is just one part of a healthy technical SEO foundation. Ensure your meta tags are fully optimized, your URLs follow best practices, and your images are compressed for optimal page speed.