WordPress robots.txt: The Complete Guide (2026)

The robots.txt file is one of the smallest files on your website and one of the easiest to get wrong. A single misplaced line can tell search engines to stop crawling your entire site, and because the file is plain text with no validation, WordPress will serve a broken version without warning you. Understanding what robots.txt does — and, just as important, what it does not do — is the difference between a file that helps crawlers work efficiently and one that quietly tanks your traffic.

This guide covers the WordPress robots.txt file: the virtual version WordPress generates, a sensible default you can copy, how to edit it through Yoast, Rank Math, or a physical file, what you should never block, and how to test changes in Google Search Console before they cause damage.

What robots.txt Actually Does

The robots.txt file lives at the root of your domain — yourdomain.com/robots.txt — and tells web crawlers which parts of your site they may request. It follows the Robots Exclusion Protocol, a standard that well-behaved crawlers like Googlebot and Bingbot respect voluntarily.

The key word is request. robots.txt controls crawling — whether a bot fetches a URL. It is a set of directions posted at the entrance, not a locked door: compliant crawlers obey it, while malicious bots and scrapers ignore it entirely. It is not a security tool, so anything you truly need to keep private must sit behind authentication, not a Disallow line. Google documents the full specification at developers.google.com/search.

What robots.txt Does NOT Do

This is the single most misunderstood point about robots.txt, and getting it wrong is how sites accidentally leak pages into search results or fail to remove them.

Blocking a page in robots.txt does not remove it from Google’s index. If you Disallow a URL, you are telling Google not to crawl it — but Google can still index that URL if it finds links pointing to it elsewhere. The result is an ugly listing showing the URL with no description.

Worse, blocking a page in robots.txt actively prevents Google from seeing a noindex tag on that page. If you want a page kept out of search results, Google must be allowed to crawl it so it can read the noindex directive. Block it in robots.txt, and Google never sees the instruction to drop it.

So the rule is simple:

• To keep a page out of the search index, use a noindex meta tag (or X-Robots-Tag header) and leave the page crawlable so Google can read it.
• To manage crawl efficiency on sections that don’t need crawling — internal search results, faceted URLs, admin endpoints — use robots.txt.

Confusing these two mechanisms is the root of most robots.txt mistakes. For how crawling and indexing fit together, see our guide to technical SEO for WordPress.

WordPress’s Virtual robots.txt

Out of the box, WordPress does not create a physical robots.txt file on your server. Instead, it generates a virtual one whenever a crawler requests yourdomain.com/robots.txt. Visit that URL on a default install and you will typically see something like:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://yourdomain.com/wp-sitemap.xml

This virtual file is generated by WordPress core and is serviceable for most sites. The one situation to watch for: if you ever checked Settings → Reading → Discourage search engines from indexing this site, WordPress changes the virtual output to block crawlers more aggressively. Always confirm that setting is unchecked on a live site.

Because the file is virtual, it does not exist as an editable file in your directory — you edit it either through a plugin filter (Yoast, Rank Math) or by creating a real physical file that overrides the virtual one. We’ll cover both.

A Sensible Default robots.txt for WordPress

For the vast majority of WordPress sites, a minimal robots.txt is the right one. Resist the temptation to add long lists of Disallow rules copied from a forum thread — most are unnecessary and some are harmful. Here is a clean, safe starting point:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

# Optional: block internal search result pages
Disallow: /?s=
Disallow: /search/

Sitemap: https://yourdomain.com/sitemap_index.xml

A few notes on this file:

• Disallow: /wp-admin/ keeps crawlers out of the admin area, while Allow: /wp-admin/admin-ajax.php preserves the one admin endpoint that front-end features legitimately call.
• The search-result blocks are optional but sensible — internal search URLs generate infinite low-value combinations that waste crawl budget.
• The Sitemap: line should match your setup (/wp-sitemap.xml for WordPress core, /sitemap_index.xml for Yoast or Rank Math). See our guide to XML sitemaps in WordPress for how to find yours.

That’s it. If you find yourself adding a dozen more rules, pause and ask whether each one solves a real, observed problem.

Common Directives Explained

robots.txt uses a small vocabulary. Understanding each directive prevents most errors.

• User-agent declares which crawler the rules apply to. User-agent: * matches all bots; User-agent: Googlebot targets Google. Rules stay grouped under that line until the next User-agent declaration.
• Disallow tells the crawler not to request URLs matching the path. An empty Disallow: means “allow everything” — the opposite of what people assume — while the dangerous Disallow: / blocks your entire site.
• Allow creates an exception to a broader Disallow, as with admin-ajax.php inside a blocked /wp-admin/. When rules conflict, most crawlers follow the more specific (longer) match.
• Sitemap points crawlers to your XML sitemap. It is not tied to a User-agent group and should be an absolute URL including https:// and your domain.
• Wildcards * (any characters) and $ (end of URL) are supported by Google and Bing — Disallow: /*.pdf$ blocks all PDFs. Use them sparingly; they over-broaden easily.

What You Must Never Block

Some Disallow rules that were common advice years ago now actively hurt your rankings. The biggest offender is blocking resources Google needs to render your pages.

Never block CSS or JavaScript. Google renders pages much like a browser does, and needs your stylesheets and scripts to see the page as users do. Older tutorials told people to block /wp-includes/ or /wp-content/ to “clean up” crawling — this prevents Google from loading theme and plugin assets, leaving it to see a broken, unstyled page that hurts mobile-friendliness and rankings. Leave these directories crawlable, including /wp-content/uploads/, so images can appear in Google Images.

Be careful with /wp-admin/. Blocking the folder is standard, but keep the Allow: /wp-admin/admin-ajax.php exception. Many themes and plugins route front-end AJAX requests through that file, and blocking it can break functionality Google’s renderer relies on.

Don’t use robots.txt to hide thin or duplicate pages from the index. As covered above, that’s a job for noindex, canonical tags, or redirects — not Disallow. If you’re wrestling with pages returning errors, our guide on how to fix 404 errors in WordPress walks through the correct handling, and the WordPress redirects guide covers moving URLs properly.

How to Edit robots.txt in WordPress

You have two practical paths: edit through an SEO plugin, or create a physical file.

Editing with Yoast SEO

Yoast includes a built-in robots.txt editor under Yoast SEO → Tools → File editor. If no physical file exists, click the “Create robots.txt file” button; Yoast writes a real file you can then edit directly in the dashboard and save. It’s the easiest route for most users because it requires no FTP access.

Editing with Rank Math

Rank Math offers the same capability under Rank Math → General Settings → Edit robots.txt. One caveat: if a physical robots.txt file already exists on your server, Rank Math cannot override it and will say so — you’ll need to edit that physical file instead.

Creating a Physical File

A physical file always takes precedence over the virtual one and over plugin filters. Create a plain-text file named exactly robots.txt (lowercase) and upload it to your site’s root directory — the folder that contains wp-config.php — via FTP, SFTP, or your host’s file manager. It gives you full control but also full responsibility: WordPress can no longer help you, so a mistake stays live until you fix it manually.

How to Test robots.txt in Search Console

Never edit robots.txt on a live site without testing. Google Search Console shows exactly how Google reads your file.

1. Open Google Search Console and select your property.
2. Go to Settings → robots.txt (under “Crawling”). Google displays the version it last fetched, when, and any parsing errors or warnings.
3. After making changes, use the Request a recrawl option so Google re-fetches the file promptly instead of waiting for its normal schedule.
4. To check whether a specific URL is blocked, use the URL Inspection tool — it reports whether the page is “Allowed” or “Blocked by robots.txt.”

Testing before and after every change is the single habit that prevents robots.txt disasters — a ten-second check catches the mistake that would otherwise cost you weeks of lost crawling.

Common Mistakes That Deindex a Site

Most catastrophes come from a short list of recurring errors.

Leaving the staging block live. Development and staging sites often ship with Disallow: / to keep them out of search. When the site goes live — or when files are copied from staging to production — that single line comes along and blocks the entire live site. Always check robots.txt immediately after any launch or migration. The related trap is the Settings → Reading “Discourage search engines” checkbox, meant for development only; left on, it quietly suppresses crawling on a site you want indexed.

Blocking pages you want deindexed. Blocking a URL doesn’t deindex it and can trap it in the index permanently by hiding the noindex tag. If a page is already indexed and you want it gone, keep it crawlable, add noindex, and only consider blocking it later once Google has dropped it.

Case and location errors. The file must be named robots.txt in lowercase and live at the domain root — not in a subfolder. yourdomain.com/blog/robots.txt does nothing.

If you’d like a broader framework for keeping crawling and indexing healthy, our WordPress SEO guide ties robots.txt, sitemaps, and on-page SEO together into one workflow.

FAQ

Do I need a robots.txt file for WordPress?

Not strictly. WordPress serves a functional virtual robots.txt automatically, and a small site with nothing unusual to block runs fine on the default. You only need to edit one to add a sitemap reference, block internal search or faceted URLs, or adjust crawl behavior for a specific section.

Will robots.txt hide a page from Google search results?

No. robots.txt controls crawling, not indexing. A blocked URL can still appear in results if other pages link to it, and blocking it prevents Google from reading a noindex tag. To keep a page out of results, use a noindex meta tag and leave the page crawlable.

Where is the WordPress robots.txt file located?

It’s served at yourdomain.com/robots.txt. On a default install, no physical file exists — WordPress generates the response virtually. If you or a plugin created a physical file, it lives in your site’s root directory alongside wp-config.php and overrides the virtual version.

Should I block wp-admin in robots.txt?

Blocking /wp-admin/ is standard and safe, but always keep the Allow: /wp-admin/admin-ajax.php exception. That endpoint handles front-end AJAX for many themes and plugins, and blocking it can break functionality Google’s renderer depends on.

Why is my whole site blocked in robots.txt?

The usual culprit is Disallow: / left over from staging, or the “Discourage search engines” checkbox under Settings → Reading being enabled. Check both after any launch or migration, remove the blanket block, and request a recrawl in Search Console.