Free Robots.txt Generator
Tell search engines exactly where they can and can't go. Our Free Robots.txt Generator creates bulletproof robots.txt files with a simple click interface — no syntax errors, no accidentally blocking your entire site from Google.
Robots.txt Generator
Hold Ctrl (or Cmd on Mac) to select multiple. Selected: 1 agent
Disallow: /wp-admin/
WordPress admin dashboard
Disallow: /wp-includes/
WordPress system files
Disallow: /wp-content/plugins/
WordPress plugin folders
Allow: /wp-admin/admin-ajax.php
Required for site functionality
Allow: /wp-content/uploads/
Images and media files
Added as a Sitemap directive at the end of the file
Adds delay between crawler requests
Robots.txt Preview
# Your robots.txt file will appear here
- 1Complete human verification above
- 2Download the robots.txt file
- 3Upload it to your website's root directory
- 4Test at: yoursite.com/robots.txt
- 5Verify in Google Search Console
Complete human verification to download or copy your robots.txt file.
Robots.txt Tester — Check Any Live Site
Fetch a site's live robots.txt and see exactly which search and AI crawlers can access a URL. Follows Google's matching rules: longest path wins, Allow beats Disallow on ties, * and $ wildcards supported.
“Robots.txt is the highest-stakes small file on any website: one wrong line and Google stops crawling your entire site. I have fixed that exact disaster for more clients than I can count, almost always a hand-edited file with one typo. This generator exists so the file is correct by construction.”
Tell Search Engines Where They Can (And Can't) Go
Create Perfect Robots.txt Files Without Breaking Your Website
Last month, a small business owner called me in a panic. “Google isn't showing any of my pages!” he said. Turns out, he'd tried to create a robots.txt file himself and accidentally blocked his entire website. One tiny typo cost him three weeks of lost traffic.
Don't be that guy.
Creating a robots.txt file might seem simple, but mess up the syntax and you could accidentally hide your whole site from Google. Or worse, you could leave sensitive pages completely exposed.
Our Robots.txt Generator takes the guesswork out of it. Just click what you want to block or allow, and we'll create a bulletproof robots.txt file that actually works.
What The Heck Is Robots.txt Anyway?
Think of robots.txt as a bouncer for your website. It stands at the front door and tells visiting search engine bots, “Hey, you can check out these pages, but stay away from those ones over there.”
- ✓It's just a simple text file that sits on your website
- ✓Search engines read it before crawling your site
- ✓Good bots respect it, sketchy ones might ignore it
- ✓Get it wrong, and you could accidentally block everything
Your WordPress admin area is at /wp-admin/. You definitely don't want that showing up in Google search results, right? Robots.txt keeps it hidden. But here's what trips people up — the syntax is super picky. Miss a slash, forget a colon, or use the wrong case, and boom — your file either doesn't work or blocks the wrong stuff.
Why Our Generator Beats DIY Robots.txt
No memorizing weird syntax or Googling "how to write robots.txt" for the hundredth time.
- ✓Pick your website type (WordPress, Shopify, whatever)
- ✓Click the folders you want to block
- ✓Hit generate and boom — perfect robots.txt file
- ✓Because nobody has time to learn robot syntax when you've got a business to run.
We've built templates for every major platform that already know what to block and what to leave alone.
- ✓/wp-admin/ (your admin area)
- ✓/wp-includes/ (WordPress core files)
- ✓/wp-content/plugins/ (plugin folders)
- ✓But allows /wp-content/uploads/ and admin-ajax.php
See exactly what your robots.txt file will look like before you download it.
- ✓The actual code that'll go on your server
- ✓Warnings if something looks fishy
- ✓Explanations of what each rule does
- ✓Which means no surprises when you upload the file.
Common Robots.txt Rules Every Website Should Consider
User-agent: * Disallow: /admin/ Disallow: /private/ Disallow: /tmp/ Disallow: /staging/
User-agent: * Disallow: /wp-admin/ Disallow: /wp-includes/ Allow: /wp-admin/admin-ajax.php Allow: /wp-content/uploads/
User-agent: * Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /admin/
User-agent: * Disallow: /dev/ Disallow: /test/ Disallow: /staging/ Disallow: /*.pdf$
What You Should (And Shouldn't) Block
- •
/admin/ or /wp-admin/(admin areas) - •
/private/(personal stuff) - •
/tmp/(temporary files) - •
/staging/(test environments) - •User account and login pages
- •Your main content and blog posts
- •CSS and JavaScript files (Google needs these!)
- •Product pages or important landing pages
- •Your sitemap.xml file
- •Search result pages (can create duplicate content)
- •Shopping cart pages (depends on your setup)
- •PDF files (unless they're important for SEO)
Robots.txt generator for WordPress: the complete setup
WordPress is the platform where robots.txt mistakes happen most, because WordPress serves a virtual robots.txt by default. There is no physical file until you create one, and plugins can silently override it. Here is the setup that works:
The ideal WordPress robots.txt
Select the WordPress preset in the generator above and you get exactly this structure: block /wp-admin/ and /wp-includes/, but explicitly allow /wp-admin/admin-ajax.php (many themes and plugins call it from the front end) and /wp-content/uploads/ (your images need to be crawlable to rank in image search).
Three ways to install it on WordPress
- SEO plugin (easiest): Yoast SEO (Tools → File editor) and Rank Math (General Settings → Edit robots.txt) both let you paste the generated file directly. The plugin serves it, no FTP needed.
- Upload via FTP or file manager: save the generated file as
robots.txtand place it in your site root (the same folder as wp-config.php). A physical file always beats the virtual one. - Hosting panel: most managed WordPress hosts (Hostinger, SiteGround, WP Engine) include a file manager where you can create the file in the web root.
Never block /wp-content/ wholesale, that kills your CSS, JavaScript, and images, and Google demotes pages it cannot render. And do not use robots.txt to hide a page from search results: blocked pages can still appear in Google with no description. Use a noindex tag for that instead.
Once your file is live, verify your whole setup with the on-page SEO analyzer and generate a matching XML sitemap to reference in the file.
How to test and submit your robots.txt file
A robots.txt file fails silently, nothing errors, pages just quietly drop out of crawling. Always verify after uploading:
- Use the live tester above: enter your domain and any URL path, and the built-in robots.txt tester fetches your live file and shows exactly which search and AI crawlers can access it, with the matching rule for each. Google retired its standalone tester; this one follows the same matching logic.
- Check it loads: visit
yoursite.com/robots.txtin a browser. You should see exactly the file you generated, not a plugin default or a 404. - Test in Search Console: Google Search Console's robots.txt report (Settings → robots.txt) shows the fetched file, when it was last crawled, and any parse errors, line by line.
- Spot-check critical URLs: use the URL Inspection tool on your homepage, a key product page, and a CSS file. All three should show “Crawl allowed: Yes.”
- Mind the AI crawlers: the generator includes toggles for GPTBot, ClaudeBot, PerplexityBot, and Google-Extended. Blocking them removes you from AI search answers, an increasingly expensive place to be invisible. Check your AI readiness with our free AEO checker before deciding.
Changes take effect at the next crawl, usually within 24 hours for active sites. Keep a copy of your previous file before replacing it, rollback is the fastest fix if traffic dips.
Frequently Asked Questions
Everything you need to know about robots.txt files.
What is a robots.txt file?
How do I create a robots.txt for WordPress?
Where do I upload the robots.txt file?
Can robots.txt hide a page from Google?
Should I block AI crawlers like GPTBot?
Is this robots.txt generator free?
Ready to Take Control?
Stop leaving your website's crawlability to chance. Create a professional robots.txt file that guides search engines to your good stuff while keeping the private pages private.
Don't let a tiny text file break your SEO. Use SEO Shouts' Robots.txt Generator and do it right the first time.
Built by people who've seen every robots.txt disaster imaginable, so you don't have to experience them yourself.