Loading...
Loading...
Tell search engines exactly where they can and can't go. Our Free Robots.txt Generator creates bulletproof robots.txt files with a simple click interface — no syntax errors, no accidentally blocking your entire site from Google.
Hold Ctrl (or Cmd on Mac) to select multiple. Selected: 1 agent
Disallow: /wp-admin/
WordPress admin dashboard
Disallow: /wp-includes/
WordPress system files
Disallow: /wp-content/plugins/
WordPress plugin folders
Allow: /wp-admin/admin-ajax.php
Required for site functionality
Allow: /wp-content/uploads/
Images and media files
Added as a Sitemap directive at the end of the file
Adds delay between crawler requests
# Your robots.txt file will appear here
Complete human verification to download or copy your robots.txt file.
Fetch a site's live robots.txt and see exactly which search and AI crawlers can access a URL. Follows Google's matching rules: longest path wins, Allow beats Disallow on ties, * and $ wildcards supported.
“Robots.txt is the highest-stakes small file on any website: one wrong line and Google stops crawling your entire site. I have fixed that exact disaster for more clients than I can count, almost always a hand-edited file with one typo. This generator exists so the file is correct by construction.”
Last month, a small business owner called me in a panic. “Google isn't showing any of my pages!” he said. Turns out, he'd tried to create a robots.txt file himself and accidentally blocked his entire website. One tiny typo cost him three weeks of lost traffic.
Don't be that guy.
Creating a robots.txt file might seem simple, but mess up the syntax and you could accidentally hide your whole site from Google. Or worse, you could leave sensitive pages completely exposed.
Our Robots.txt Generator takes the guesswork out of it. Just click what you want to block or allow, and we'll create a bulletproof robots.txt file that actually works.
Think of robots.txt as a bouncer for your website. It stands at the front door and tells visiting search engine bots, “Hey, you can check out these pages, but stay away from those ones over there.”
Your WordPress admin area is at /wp-admin/. You definitely don't want that showing up in Google search results, right? Robots.txt keeps it hidden. But here's what trips people up — the syntax is super picky. Miss a slash, forget a colon, or use the wrong case, and boom — your file either doesn't work or blocks the wrong stuff.
No memorizing weird syntax or Googling "how to write robots.txt" for the hundredth time.
We've built templates for every major platform that already know what to block and what to leave alone.
See exactly what your robots.txt file will look like before you download it.
User-agent: * Disallow: /admin/ Disallow: /private/ Disallow: /tmp/ Disallow: /staging/
User-agent: * Disallow: /wp-admin/ Disallow: /wp-includes/ Allow: /wp-admin/admin-ajax.php Allow: /wp-content/uploads/
User-agent: * Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /admin/
User-agent: * Disallow: /dev/ Disallow: /test/ Disallow: /staging/ Disallow: /*.pdf$
/admin/ or /wp-admin/ (admin areas)/private/ (personal stuff)/tmp/ (temporary files)/staging/ (test environments)WordPress is the platform where robots.txt mistakes happen most, because WordPress serves a virtual robots.txt by default. There is no physical file until you create one, and plugins can silently override it. Here is the setup that works:
Select the WordPress preset in the generator above and you get exactly this structure: block /wp-admin/ and /wp-includes/, but explicitly allow /wp-admin/admin-ajax.php (many themes and plugins call it from the front end) and /wp-content/uploads/ (your images need to be crawlable to rank in image search).
robots.txt and place it in your site root (the same folder as wp-config.php). A physical file always beats the virtual one.Never block /wp-content/ wholesale, that kills your CSS, JavaScript, and images, and Google demotes pages it cannot render. And do not use robots.txt to hide a page from search results: blocked pages can still appear in Google with no description. Use a noindex tag for that instead.
Once your file is live, verify your whole setup with the on-page SEO analyzer and generate a matching XML sitemap to reference in the file.
A robots.txt file fails silently, nothing errors, pages just quietly drop out of crawling. Always verify after uploading:
yoursite.com/robots.txt in a browser. You should see exactly the file you generated, not a plugin default or a 404.Changes take effect at the next crawl, usually within 24 hours for active sites. Keep a copy of your previous file before replacing it, rollback is the fastest fix if traffic dips.
Everything you need to know about robots.txt files.
Stop leaving your website's crawlability to chance. Create a professional robots.txt file that guides search engines to your good stuff while keeping the private pages private.
Don't let a tiny text file break your SEO. Use SEO Shouts' Robots.txt Generator and do it right the first time.
Built by people who've seen every robots.txt disaster imaginable, so you don't have to experience them yourself.