Skip to content
Rescue 404
Rescue

Free tool

robots.txt Generator

Generate a customized robots.txt for WooCommerce, blogs, local business, membership, and LMS sites.

Primary use
Create crawler directives with explanations — not a security lock.
Estimated time
About 5 minutes
Input type
Manual answers
Use the tool All free tools

Create a Practical robots.txt Starting Point

Choose a site profile and this generator creates crawler directives with plain-language explanations. Review the file against your live site before publishing it at /robots.txt.

  • robots.txt guides compliant crawlers; it is not a security control or a way to hide sensitive content.
  • Blocking common AI crawler names is a request, not a guarantee of training, indexing, or access behavior.

How to Publish It

Download the generated file, review the paths and sitemap URL, then place it in the public document root as robots.txt. Test it in your search tooling and check that important pages and assets remain crawlable.

When to Ask for Help

Ask for a technical review before changing directives on a large site, an ecommerce store with complex filters, or a site using multiple languages, subdomains, staging environments, or CDN rules.

Generate a robots.txt file

Review every rule before publishing. robots.txt is a crawler request, not access control or security.

Generator settings
Setup mode

Generated file

# Generated by Rescue404 robots.txt Generator
# robots.txt is a crawler directive, not access control or security.

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Disallow: /wishlist/
Disallow: /*?s=
Disallow: /*?post_type=product
Allow: /wp-admin/admin-ajax.php

Rule explanations

DirectiveWhy it is included
User-agent: *Rules below apply to crawlers that do not match a more specific block.
Disallow: /cart/ /checkout/ /my-account/Keeps personal/cart flows out of ordinary crawl budgets. Does not secure those URLs.
Disallow: /wp-admin/Asks matching crawlers not to fetch paths under /wp-admin/.
Disallow: /wp-includes/Asks matching crawlers not to fetch paths under /wp-includes/.
Disallow: /cart/Asks matching crawlers not to fetch paths under /cart/.
Disallow: /checkout/Asks matching crawlers not to fetch paths under /checkout/.
Disallow: /my-account/Asks matching crawlers not to fetch paths under /my-account/.
Disallow: /wishlist/Asks matching crawlers not to fetch paths under /wishlist/.
Disallow: /*?s=Asks matching crawlers not to fetch paths under /*?s=.
Disallow: /*?post_type=productAsks matching crawlers not to fetch paths under /*?post_type=product.
Allow: /wp-admin/admin-ajax.phpException allowing /wp-admin/admin-ajax.php even if a broader Disallow would match.

Review before publishing

  • WooCommerce filter and facet URLs vary by theme/plugin — review parameter rules after deploy.
  • No sitemap URL set — add one when available.
  • No Sitemap directive found.

Methodology

  • Profiles begin with common WordPress crawl directives and add paths commonly associated with the selected site type.
  • The generator keeps admin-ajax.php allowed because many public WordPress features rely on it.
  • Extra Allow and Disallow paths are normalized to begin with a slash and are deduplicated.
  • AI options emit user-agent blocks for common crawler names or preserve your custom rules verbatim.
  • Warnings flag missing sitemap details, resource-blocking patterns, and membership/LMS paths that still need real access controls.

Limitations

  • robots.txt syntax and wildcard support vary among crawlers.
  • Directives are voluntary and do not authenticate users, prevent downloads, or guarantee deindexing.
  • This generator cannot inspect your live routes, plugin behavior, or sitemap.
  • Review and test changes before publishing, especially on ecommerce and multilingual sites.

Privacy

Generation happens in your browser. The site URL and rules you enter are not sent to Rescue404.

FAQ

Plain answers about what this tool does and does not claim.

Does robots.txt protect private pages? +
No. URLs may still be accessed directly and may appear in search results if linked elsewhere. Protect private content with authentication and appropriate server controls.
Will blocking AI bots stop AI training? +
No. User-agent directives depend on each crawler choosing to comply, and they do not control previously collected content or every automated system.
Should I block wp-admin? +
Usually, while allowing admin-ajax.php when needed by public site features. This is an indexing/crawling preference, not protection for your admin area.
Can I block search result pages? +
Often, yes. Search and filtered URLs can create thin or duplicate pages, but confirm the patterns match your theme and plugins before publishing.
Where is my sitemap? +
Many WordPress sites expose /wp-sitemap.xml, while SEO plugins often use /sitemap_index.xml. Enter the exact public sitemap URL you have verified.

Related tools

Related guides

Related services

Website Health Check Request Rescue

Have Rescue404 investigate these technical issues

Useful when metadata, schema, or crawl directives are tangled with hosting, redirects, or theme output.

Request a technical review Website Health Check

Results stay free. No account required to use the tool.