Robots.txt is a small file, and that is why mistakes in it cost so much. A single broad rule can hide the CSS, images or whole sections your visitors depend on.

Know what the file controls

It tells compliant crawlers which paths they should not request. It does not hide pages from the internet, and it does not remove URLs from search results. For the difference between crawling and indexing, see the separate article on what the file can and cannot do.

Write narrow rules

Name the exact folder or pattern and know why it is there. A rule like Disallow: /search/ is easy to reason about. A rule like Disallow: /*? can block far more than you expect, including pages that use parameters for legitimate content.

Beware of the full block

During development, Disallow: / is a common way to keep crawlers away. Forgetting to remove it at launch is one of the most frequent causes of a site disappearing from search. Add a launch-checklist item to read the live file.

Do not block what pages need to render. Search engines render pages. If you block CSS, JavaScript or images, they may misjudge your layout and content. Keep those resources crawlable.

Include your sitemap

Add the sitemap location on its own line. The Robots.txt & XML Sitemap Generator can create both files, but read the output before uploading it.

Check after every migration. A staging file can slip into production, and an old rule can outlive a URL change. After a redesign, domain move or HTTPS switch, open the live file and test important URLs in Search Console.

An example of a safe file

A simple, careful file for a small content site could be:

User-agent: *
Disallow: /admin/
Disallow: /search/

Sitemap: https://example.com/sitemap.xml

It blocks two clear areas, leaves everything else open and points to the sitemap. Each line has a known purpose.

A review checklist

  1. Open the live file in a browser.
  2. Look for a line that says Disallow with only a slash.
  3. Test three important URLs in Search Console.
  4. Check that CSS and image folders are not blocked.
  5. Confirm the sitemap address works.
  6. Save a dated copy.

Staging sites and development copies

Block staging sites with a password, not only with robots.txt. A password stops both crawlers and curious visitors. If you must use robots.txt there, make sure the file does not travel to production in a deployment. Add a check to your release steps.

Testing rules before you upload

Python's standard library has a basic robots.txt parser that can test simple rules:

from urllib.robotparser import RobotFileParser

rp = RobotFileParser()
rp.parse(open('robots.txt').read().splitlines())
print(rp.can_fetch('Googlebot', 'https://example.com/guides/'))

It does not follow every wildcard rule the way Google does, so use it for simple checks and confirm important rules in Search Console.

Wildcard traps

The pattern Disallow: /*.pdf$ blocks all PDFs, which may be right. The pattern Disallow: /*? blocks any URL with a query string, including search pages you may want crawled. The pattern Disallow: /p blocks every path starting with /p, such as /products/ and /privacy/. Always add the trailing slash when you mean a folder.

If you break something

If a mistake goes live, fix the file straight away and request a recrawl by loading the file in Search Console. Google normally refreshes its copy within a day. Then check that important pages are still crawlable and keep an eye on the indexing report for the next few weeks.

Keep a copy. Store the file in version control with a comment for each rule, so future you knows why it exists.