Robots.txt Disallow All: Block All Crawlers Safely
Short answer: User-agent: * followed by Disallow: / tells every crawler that follows the robots.txt standard not to fetch any URL on your site. It stops crawling, not indexing: pages Google already knows can stay in results, shown without a description. To keep a site out of search, use a noindex tag or a login instead.
Blocking every crawler with a single robots.txt rule is one of the simplest things you can do — and one of the most dangerous when it ends up on the wrong site. This guide covers the correct syntax, what the directive actually does (and what it does not), every common use case, the safer alternatives that should be your default for staging, and how to recover if a Disallow: / rule slips into production. If you want to block a specific folder instead of the whole site, see the directory-specific disallow guide.
The Correct Syntax to Disallow All Crawlers
The full, correct robots.txt to block every well-behaved crawler from every URL on your site is exactly two lines:
# /robots.txt — block all crawlers from the entire site User-agent: * Disallow: /
The file must be served from the root of the domain at https://yourdomain.com/robots.txt with a 200 OK response and a Content-Type of text/plain. User-agent: * means "every crawler that obeys robots.txt". Disallow: / means "the entire URL path tree starting at /" — which is to say, everything.
A few details that matter. Lines beginning with # are comments and are ignored. Field names are not case-sensitive (Disallow and disallow are read the same way), but paths are (/Admin and /admin are different paths). Keep each User-agent line directly above its rules. And no Allow: line is needed; Disallow alone is enough.
Common variations that mean the same thing as Disallow: /:
# All three of these block the entire site identically: Disallow: / Disallow: /* Disallow: /*$ # This does NOT block the site — it allows everything: Disallow: # Empty Disallow value is interpreted as "nothing is disallowed".
Blocking only Google looks like this: User-agent: Googlebot followed by Disallow: /. Other crawlers can still visit, but for your Google rankings it has the same effect as blocking everyone. If you find it on a live site, remove it.
The trap in that last block catches teams who copy-paste an example, edit the path, and accidentally leave the value empty thinking it does nothing. Disallow: with no value is the explicit signal "allow everything" — the opposite of what most people who type those characters intend.
What Disallow All Actually Does (And What It Does Not)
This is the part that surprises people. Disallow: / blocks crawling. It does not block indexing. The two are different operations in Google's pipeline.
What changes immediately: Googlebot, Bingbot, and other compliant crawlers stop fetching new and updated pages from your site. Google usually caches robots.txt for up to 24 hours, so the change takes effect within about a day. After that, crawling of the blocked URLs stops.
What does not change: Pages that Google already has in its index stay indexed. They will continue to appear in search results — sometimes for weeks, sometimes for months. Google describes them as "Indexed, though blocked by robots.txt" in the Search Console Pages report. The snippet often disappears (Google cannot recrawl to refresh it), and the title may revert to anchor text from external links pointing to the page, but the URL itself remains in the index and ranks for the queries it ranked for before.
Why this matters: if your goal is to deindex content (for example, you want a folder to disappear from Google), Disallow: / is the wrong tool. To deindex, Google needs to crawl the page and read either a noindex meta tag or an X-Robots-Tag: noindex response header. If robots.txt blocks the crawl, Google never sees the noindex directive — and the page stays indexed indefinitely. See the noindex directives guide for the right pattern.
The mental model: robots.txt is a polite "please do not visit" sign. It does not delete anything. It does not retroactively undo what is already in the index. It only changes future crawl behavior for crawlers that choose to obey.
Legitimate Use Cases for Disallow All
There are a handful of situations where blocking all crawlers via robots.txt is genuinely the right move. The common thread: the site is not yet ready for the public, and the URLs are not already in any search index.
Staging sites on a separate hostname. A staging environment at staging.example.com serves a copy of the production site to internal users and QA. You do not want it competing with production for the same queries. Disallow: / on staging is a reasonable second-line defence — but never the only line of defence (see the basic auth section below).
Brand-new sites pre-launch. A site that has never been published, has zero inbound links, and has never been submitted to a search engine can safely sit behind Disallow: / while the team builds it out. The risk is that someone forgets to remove the directive on launch day. Add a launch checklist item and a CI check for it.
Soft launches and private betas. A product launching to invited users only, where you want the URL hidden from organic discovery. Robots.txt does not actually hide the URL from people who type it — for that you need authentication — but it does keep the site out of Google's discovery for the duration of the beta.
Internal tools and admin panels on subdomains. A subdomain like admin.example.com or tools.example.com that exists only for staff. These should always have authentication, but robots.txt is a useful belt-and-braces measure to keep crawlers from wasting crawl budget on the login page.
Paywalled or members-only content. If the entire site is behind a paywall and the public preview pages do not need to rank, you can block crawling. More commonly, you want crawlers to see preview content and use a structured data approach (see Google's flexible sampling guidelines for paywalled content) — so this case is rarer than it looks.
The Catastrophic Production Mistake
The single worst SEO incident in most engineering teams' history is the day someone deployed staging's robots.txt to production. It happens like this:
An engineer sets up Disallow: / on staging. The robots.txt file is checked into the repository. A deploy script pushes the contents of the static folder to production without environment-specific overrides. Once Googlebot re-reads robots.txt and sees Disallow: /, it stops crawling the site. Snippets start to disappear from search results, and because Google can no longer refresh the content, pages that stay indexed tend to lose ground over time.
The recovery (covered in detail later) is straightforward, but it is never instant. The longer the block stays live, the longer the recovery tends to take, so the fastest win is catching it early.
The two engineering practices that prevent this:
# 1. Generate robots.txt dynamically based on environment, not from a static file:
# See the Next.js robots.ts pattern below.
# 2. Add a CI check that fails the production build if robots.txt contains "Disallow: /":
if grep -E "^Disallow:\s*/\s*$" public/robots.txt > /dev/null; then
if [ "$VERCEL_ENV" = "production" ]; then
echo "FAIL: production build contains Disallow: / in robots.txt"
exit 1
fi
fiHow to Check If Your Live Site Is Blocked
This takes about two minutes:
- Open the file. Visit
https://yourdomain.com/robots.txt. Look forDisallow: /underUser-agent: *orUser-agent: Googlebot. - Test your homepage in Search Console. Open URL Inspection, enter your homepage and click "Test live URL". A block shows up as "Crawl allowed? No: blocked by robots.txt".
- Search for your site. A
site:yourdomain.comsearch that shows results without descriptions, or no results for a site you know is published, is a warning sign.
WordPress sites: check Settings → Reading → "Discourage search engines from indexing this site". Since WordPress 5.3 this box no longer adds Disallow: / to robots.txt; it adds a noindex, nofollow robots meta tag instead. Either way it hides the site, so untick it before launch.
HTTP Basic Auth: The Safer Alternative
For staging and pre-launch sites, HTTP basic authentication is almost always a better choice than robots.txt. The reason: a 401 response means Googlebot never sees any HTML. There is nothing to index, nothing to misinterpret, and no risk of a misconfigured robots.txt accidentally being deployed to production.
nginx:
# Generate the password file (run once):
# sudo htpasswd -c /etc/nginx/.htpasswd staginguser
#
# Then add to your server block:
server {
server_name staging.example.com;
auth_basic "Staging — authorized users only";
auth_basic_user_file /etc/nginx/.htpasswd;
location / {
proxy_pass http://localhost:3000;
}
}Apache:
# In .htaccess at the document root: AuthType Basic AuthName "Staging — authorized users only" AuthUserFile /var/www/staging/.htpasswd Require valid-user # Generate the password file: # htpasswd -c /var/www/staging/.htpasswd staginguser
The result: any request to the staging site without credentials returns 401 Unauthorized. Googlebot treats 401 as "cannot access" and does not index anything from the host. Crucially, this protection is at the server level, not in a static file — so it is much harder to accidentally deploy to production.
IP Allowlists: The Other Safer Alternative
If basic auth is awkward (for example, you have automated tests that need to hit the site, or third-party tools that cannot send credentials), an IP allowlist gives the same protection without the password prompt. Restrict the staging hostname at the load balancer or CDN to your office IPs, your VPN range, and any CI runner addresses.
# Cloudflare Worker — return 403 to any non-allowlisted IP:
const ALLOWLIST = [
"203.0.113.0/24", // office
"198.51.100.42", // CI runner
"192.0.2.0/28", // VPN
];
addEventListener("fetch", (event) => {
const ip = event.request.headers.get("CF-Connecting-IP");
if (!ipMatchesAny(ip, ALLOWLIST)) {
event.respondWith(new Response("Forbidden", { status: 403 }));
return;
}
event.respondWith(fetch(event.request));
});Crawlers connecting from outside the allowlist receive 403 and never see your HTML. As with basic auth, this is enforced at infrastructure level, not in a deployable static file, which removes the most common failure mode.
Next.js: Generate Robots.txt Dynamically
If you must use robots.txt for environment gating, never check a static public/robots.txt with Disallow: / into the repository. Instead, generate it dynamically based on the environment. In Next.js App Router, this is the app/robots.ts file:
// app/robots.ts
import type { MetadataRoute } from "next";
export default function robots(): MetadataRoute.Robots {
const isProduction =
process.env.VERCEL_ENV === "production" ||
process.env.NODE_ENV === "production" &&
process.env.NEXT_PUBLIC_SITE_URL === "https://example.com";
if (!isProduction) {
return {
rules: [{ userAgent: "*", disallow: "/" }],
};
}
return {
rules: [{ userAgent: "*", allow: "/", disallow: ["/api/", "/admin/"] }],
sitemap: "https://example.com/sitemap.xml",
};
}The file at /robots.txt is now derived from the environment at request time. There is no static file to deploy, no risk of staging's configuration overwriting production's, and the production version always emits the correct allow rules.
Vercel and Netlify Deploy Previews
Deploy previews on Vercel and Netlify get their own URLs, such as my-feature-branch-abc123.vercel.app, for every branch or pull request. Check whether yours are protected: both hosts let you require a login for previews. If a preview is public and a link to it leaks, Google can find it and see a copy of your content. The simplest guard is to serve Disallow: / (or a noindex header) everywhere except production.
The fix on Vercel is to detect the preview environment and emit Disallow: / only there:
// app/robots.ts — Vercel preview-aware
import type { MetadataRoute } from "next";
export default function robots(): MetadataRoute.Robots {
// VERCEL_ENV is "production", "preview", or "development"
if (process.env.VERCEL_ENV !== "production") {
return { rules: [{ userAgent: "*", disallow: "/" }] };
}
return {
rules: [{ userAgent: "*", allow: "/" }],
sitemap: "https://example.com/sitemap.xml",
};
}For Netlify, the equivalent environment variable is CONTEXT, which is production on the production branch and deploy-preview or branch-deploy elsewhere. The same conditional logic applies. Either way, the production deployment always emits an allow rule and the preview deployments always emit disallow — and there is no static file involved that could be deployed to the wrong environment.
Bot-Specific Disallow: Blocking AI Training Crawlers
You may want Google and Bing to crawl normally but keep your content out of AI training sets. Robots.txt supports this with bot-specific groups:
# /robots.txt — allow search engines, block AI training crawlers User-agent: * Allow: / User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: CCBot Disallow: / User-agent: Google-Extended Disallow: / Sitemap: https://example.com/sitemap.xml
Three things to know. First, the most-specific matching group wins — so a bot named GPTBot follows the GPTBot group and ignores the * group entirely. Second, Google-Extended is a special token that controls Google's use of your content for AI training (Gemini and Vertex AI Generative APIs) without affecting the regular Googlebot used for Search. Third, training crawlers are not the same as AI search crawlers. OAI-SearchBot (ChatGPT search) and PerplexityBot fetch pages so those assistants can cite them; blocking them removes you from their answers. If you want AI traffic, leave them allowed. New crawlers appear often, so keep the list in version control and review it now and then.
Recovering After Pushing Disallow All to Production
If the worst happens and Disallow: / ends up on production, the recovery sequence is:
Step 1 — Fix the file immediately. Replace the bad robots.txt with the correct one and verify by fetching it directly: curl -i https://yourdomain.com/robots.txt. Confirm the response is 200 OK, the body matches what you expect, and any CDN cache (Cloudflare, Vercel, Fastly) has been purged. Robots.txt is often cached aggressively — a fixed origin file does no good if the CDN keeps serving the bad version.
Step 2 — Force Google to refetch robots.txt. In Google Search Console, open Settings → robots.txt → click the three-dot menu next to your robots.txt entry and choose "Request a recrawl". This is usually faster than waiting for Google's cached copy (kept for up to 24 hours) to expire.
Step 3 — Push priority pages back into the queue. Use the URL Inspection tool on your highest-traffic pages and click "Request Indexing" on each. Google limits how many requests you can make each day, so spend them on revenue pages and category hubs, not on individual blog posts that will recover on their own.
Step 4 — Resubmit the sitemap. In Search Console's Sitemaps report, remove and re-add your sitemap. This nudges Google to re-evaluate the URL set as freshly discoverable. SitemapFixer's free check tests 30 URLs sampled from the sitemap for 404s, redirects, noindex and robots.txt blocks — a quick way to catch other problems before you resubmit.
Step 5 — Monitor. Watch the Pages report in Search Console for the "Indexed, though blocked by robots.txt" category to start declining, and watch Crawl Stats for crawl rate to recover. Crawling usually comes back first; rankings take longer, and the longer the block was live, the longer they tend to take.
How GSC Reports a Site Blocked by Robots.txt
Understanding what Search Console will show you is useful both for confirming a deliberate block is working and for diagnosing an accidental one.
Pages report: URLs that Google knows about but cannot crawl appear under "Indexed, though blocked by robots.txt" (if previously indexed) or "Blocked by robots.txt" (if discovered but never indexed). The first category is the one that matters during recovery — it counts the URLs that were live in search before the block.
URL Inspection tool: A blocked URL shows "Crawl allowed? No: blocked by robots.txt" in the Coverage section. If the URL is also indexed, you will see "URL is on Google" alongside the crawl block — that combination is the "indexed but not crawlable" state, where snippets and titles often look stale.
Search appearance: A blocked-but-indexed URL can still show up in results, but without a description: Google shows a note that no information is available for the page. Click-through rate on these listings drops sharply, even though ranking position may hold for a while. If you see this on production pages, your robots.txt is the cause.
Crawl stats: The Crawl Stats report under Settings shows total crawl requests over time. A site-wide robots.txt block produces a sharp drop in this graph within a day or two. It is the fastest visual confirmation that a block is in effect.
A Safe Robots.txt for a Live Site
Most live sites need very little. This is a safe starting point:
User-agent: * Disallow: Sitemap: https://yourdomain.com/sitemap.xml
An empty Disallow: allows everything. Add a Disallow line only for areas that should never be crawled, such as /wp-admin/, internal search results or checkout pages. Never copy a staging robots.txt to production without reading it first.
Best Practices for Staging and Dev Environments
To wrap this up: a checklist that prevents the most common robots.txt incidents.
Use authentication, not robots.txt, as the primary defence on staging. Basic auth or IP allowlist at the server or CDN level. Robots.txt is at most a secondary defence, not a primary one.
Generate robots.txt dynamically. In Next.js use app/robots.ts; in other frameworks use the equivalent dynamic-route approach. Never check a static public/robots.txt with Disallow: / into a repository that also deploys to production.
Add a CI check. A one-line grep that fails the production build if the rendered robots.txt contains Disallow: /. This catches the mistake before it ever reaches Googlebot.
Use unique hostnames per environment. Production, staging, and preview should all live on different hostnames so a fetch of the wrong robots.txt cannot affect production crawling. Avoid sharing a domain across environments with path-based routing.
Audit production robots.txt monthly. A simple scheduled check that fetches the live robots.txt and compares it with the version you expect catches drift before Google does. It costs far less than a recovery.
Document who owns robots.txt. One of the most common root causes of robots.txt incidents is an SEO change made by a marketer in a CMS plugin clashing with an engineering deploy. Pick one source of truth — usually the framework — and document it.