Free technical SEO tool
Free Robots.txt Generator & Checker
Build a clean robots.txt with sensible presets, or check an existing file for accidental blocks and formatting problems.
Not every crawler supports Crawl-delay, so treat it as a hint rather than a guarantee.
robots.txt preview
User-agent: * Disallow:
Upload this file to the root of your domain, for example www.example.com.au/robots.txt. The WordPress starter only blocks the admin area and site search, leaving theme and media files alone.
What this robots.txt tool does
The generator builds a robots.txt from user-agent groups, Allow and Disallow rules, an optional sitemap line and an optional crawl delay, with presets for allowing everything, staging sites and WordPress. The checker reads a live robots.txt, shows the raw file and its directives, and flags practical problems such as a site wide block, a missing sitemap or malformed lines.
How to use it
- Generate: pick a preset, adjust the rules, add your sitemap URL, then copy or download the file.
- Upload it to the root of your domain, replacing any existing robots.txt.
- Check: enter a domain to read its current file and work through the observations.
- Use the path tester to sanity check a specific path against a specific user agent.
Tips and common mistakes
Do not block what you need indexed
Blocking CSS, JavaScript or images can stop Google rendering your pages properly. Keep rules specific, and never use robots.txt for privacy. Anything sensitive should sit behind authentication.
Always include your sitemap
The Sitemap line is the easiest way to tell crawlers where your full page list lives.
Test after any change
Small robots.txt typos have taken whole sites out of search before. After a change, use the checker and keep an eye on your search performance for a few weeks.
Frequently asked questions
Related tools
- XML Sitemap CheckerFind duplicates, invalid URLs and status problems in a sitemap.
- Canonical Tag CheckerCheck a page's canonical tag and conflicting signals.
- AI Crawler Access CheckerSee which AI search, training and user crawlers robots.txt allows.
- All free marketing toolsBrowse every tool in one place.
Want the technical side handled for you?
Crawling, indexing, sitemaps and structure are part of our ongoing SEO work. We keep the foundations right so your content can compete.