Palz Tools
English
Menu

Robots.txt Generator & Tester

Build a robots.txt in a minute, block AI training crawlers if you want to, and test the file before you upload it: paste any robots.txt and see which addresses each crawler may visit.

All crawlers (User-agent: *)

Start each with /. Use * for any characters and $ for the end, e.g. /*.pdf$.

An Allow longer than a matching Disallow wins.

Block AI crawlers

AI training crawlers

AI search crawlers (blocking them can keep you out of AI answers)

Fetches a person asks for (not all of these follow robots.txt)

Blocking Google-Extended or Applebot-Extended doesn't remove you from Google Search or Siri; it only opts your pages out of their AI training.

Google ignores it; Bing and ClaudeBot read it. Only for servers that struggle with crawl load.

Upload it as robots.txt to the root of your site, so it opens at https://your-site/robots.txt. Each subdomain needs its own.

How to use it

  1. Generate: keep Allow the site, list the folders crawlers should skip (such as /admin/ or /cart/), tick any AI crawlers to block and add your sitemap address.
  2. Press Test this file, or open the Test tab and paste any robots.txt.
  3. Pick a crawler, list the addresses you care about and read the verdicts. Then download the file and upload it to your site's root.

How Google reads the rules

  • A crawler obeys only the group for its most specific name: Googlebot-Image uses a Googlebot-Image group if there is one, otherwise Googlebot, otherwise *.
  • Rules match from the start of the path. * stands for any characters and $ marks the end, so /*.pdf$ covers every PDF.
  • Paths are case-sensitive: /Fish doesn't match /fish.
  • Google reads only the first 500 KiB of the file and ignores fields it doesn't support, such as Crawl-delay.
  • If robots.txt is missing (a 404), Google crawls everything; if it fails with a server error, Google pauses crawling the site.

Example

With Disallow: / and Allow: /$ in the * group, the home page / is allowed, because /$ is the longer match, but /page.htm is blocked, because /$ only covers the root. With Allow: /folder and Disallow: /folder, /folder/page is allowed: the two rules are equally long, so the less restrictive one wins. Both cases come from Google's own specification, and the tester gives the same answers.

Checking the rest of your setup

Pair this with the XML sitemap generator for the Sitemap line, and with the meta tag generator for noindex tags. Fetching a live site's robots.txt from this page comes soon; until then, open the site's /robots.txt in your browser and paste it into the tester.

Frequently asked questions

Does robots.txt keep a page out of Google?

No. It stops crawling, not indexing: a blocked page can still appear in results, without a description, if other sites link to it. To keep a page out, let Google crawl it and add a noindex meta tag or X-Robots-Tag header.

If Allow and Disallow both match, which wins?

The longest matching rule, counted in characters, wins. When an Allow and a Disallow are equally long, Google uses the less restrictive one, the Allow. The tester shows which line decided each address.

Will blocking AI crawlers hurt my Google ranking?

Blocking Google-Extended only opts your pages out of Gemini training; Google Search keeps crawling with Googlebot. Blocking AI search crawlers such as OAI-SearchBot or PerplexityBot can keep your pages out of those assistants' answers, so choose per bot.

Where does the file go?

At the root of each host, as https://example.com/robots.txt. A subdomain such as shop.example.com needs its own file, and so does each protocol and port.

Does Google follow Crawl-delay?

No, Google ignores it. Bing reads it, and Anthropic says ClaudeBot does too. Use it only if crawlers really overload your server.

More SEO and creator tools

llms.txt GeneratorCreate an llms.txt in the llmstxt.org format: summary, key facts and links, imported from your sitemap.Open Graph PreviewPreview a link on Facebook, WhatsApp, LinkedIn and X, and check a page's og: tags and share image.UTM BuilderCampaign URLs with utm_source, medium, campaign and utm_id for GA4, with the channel each lands in.Keyword Density CheckerMost-used words and phrases, your target keyword's density, any language, CSV export.YouTube Title AnalyzerScore a video title on length, YouTube's limits, keyword position and clarity, with phone and search previews.YouTube Thumbnail Maker16:9 thumbnails from templates: bold outlined text, stickers, your photo with the person cut out, and a JPG under 2 MB.

See all SEO and creator tools →

Can't find the tool you need? Tell us, and the most requested tools get built first.