Robots.txt Configuration Guide: Control What Search Engines Crawl
Robots.txt is one of the most powerful — and most dangerous — files in SEO. A few lines of text control how search engines crawl your entire site, and a single misplaced directive can block Google from your best content or, worse, your whole domain. Every SEO professional needs to understand it cold. This article covers what robots.txt controls, the essential directives, and the mistakes that quietly wreck sites. For the complete setup, see our robots.txt configuration guide.
What Robots.txt Controls
Robots.txt tells crawlers which URLs they may or may not access. Crucially, it controls crawling, not indexing — a page blocked in robots.txt can still appear in the index if other pages link to it. To keep a page *out* of the index, use a noindex tag (and don't block it in robots.txt, or Google can't see the noindex). Confusing these two is the most common and costly robots.txt mistake.
Essential Directives
- User-agent: which crawler the rules apply to (`*` for all, or target specific bots)
- Disallow: blocks crawling of specified paths
- Allow: overrides Disallow for specific sub-paths
- Sitemap: points crawlers to your XML sitemap
- Crawl-delay: requests a delay between requests (ignored by Google)
Common Configuration Patterns
- Block admin areas, user dashboards, and internal search-result pages
- Allow all public content directories
- Block faceted-navigation parameters that create duplicate URLs
- Point to your sitemap.xml
- Allow AI crawlers (GPTBot, PerplexityBot) if you want AI visibility
Dangerous Mistakes to Avoid
- Accidentally disallowing `/` — blocks your entire site (often left over from staging)
- Blocking CSS/JS files — prevents Google from rendering your pages correctly
- Using robots.txt to 'hide' a page that's already indexed (use noindex instead)
- Forgetting to update it after a migration
- Deploying changes without testing them first in Search Console
Test every change before it ships, and re-check robots.txt after any redesign or migration — these are exactly when it silently breaks.
Frequently Asked Questions
What does robots.txt do?
It tells search-engine crawlers which parts of your site they may or may not crawl, using User-agent, Disallow, Allow, and Sitemap directives. It manages crawling, not indexing, and is mainly used to steer crawl efficiency.
Does robots.txt keep a page out of Google's index?
No. It blocks crawling, not indexing — a disallowed page can still be indexed if other pages link to it. To keep a page out of the index, use a noindex tag and don't block the page in robots.txt (or Google can't see the noindex).
What's the most dangerous robots.txt mistake?
Disallow: / — which blocks the entire site from crawling. It's commonly left in place accidentally after moving from staging to production and can deindex everything.
Should I block CSS and JavaScript in robots.txt?
Never. Google needs your CSS and JS to render and understand pages. Blocking them can make Google see a broken version of your site and hurt rankings.
How do I test robots.txt changes?
Use Google Search Console's robots.txt tester to verify your rules and check sample URLs from every major section before deploying, and re-test after any redesign or migration.
Related Articles
Most brands have no idea whether AI search engines mention them. This audit playbook shows you how to measure your AI visibility end-to-end using Vincony's SEO Studio.
SEO ToolsComplete Guide to Vincony's SEO Studio: 6 Tools Every Marketer NeedsKeyword Research, Backlink Analysis, Rank Tracking, Site Audit, Content Analysis, and AI Overviews — all in one affordable platform.
SEO ToolsRank Tracking in 2026: Monitor Your Keywords with Real Google DataTrack keyword rankings across Google, Bing, and regional search engines with real-time data.