What is a robots.txt file?

Robots.txt is a text file placed in the root directory of a website that tells search engine crawlers which pages they can or cannot request.

What is an XML sitemap?

A sitemap is an XML file listing the key URLs of a website, helping search engines discover and crawl all pages efficiently.

Where should robots.txt be placed?

It must be placed at the absolute root of your domain, accessible via `yourdomain.com/robots.txt`.

Does robots.txt guarantee privacy?

No. Crawlers can ignore robots.txt directives, and the file is publicly readable. Use authentication for true privacy.

Why is a sitemap link needed in robots.txt?

Linking your XML sitemap in robots.txt helps crawlers find your sitemap file instantly during indexing.

Can I block indexing of a specific page via robots.txt?

While robots.txt prevents crawling, Google recommends using a 'noindex' meta tag to completely prevent indexing.

What does 'User-agent: *' mean?

It applies the following directives to all search engine robots and web crawlers globally.

Is this generator secure?

Yes, the file is compiled locally in your browser, and no domain URLs or folder configurations are logged.

Crawl-delay is a directive telling bots to wait a specified number of seconds between page requests to prevent server overload.

What does the 'Disallow' directive do?

It instructs web crawlers not to visit or read files inside a specific directory path (e.g. `/admin/`).

Back to Tools

SEO Tools

Robots.txt & Sitemap Maker

Compile standard robots.txt files and sitemap checklists to direct search indexing crawlers.

Crawler Index Directives

Default Crawler Behavior (All Bots)

Canonical XML Sitemap URL

Restricted / Disallow Folders (One per line)

Individual Search Engine Exclusions

Block Googlebot Block Bingbot Block Yandexbot

Robots.txt & Sitemap Maker

Crawler Index Directives

Robots.txt Output

What is Robots.txt & Sitemap Generator and How to Use It?

How to Use

Frequently Asked Questions

Recommended Tools

SEO Meta Tags Generator

DNS Lookup & Domain Analyzer

URL Encoder/Decoder

Robots.txt & Sitemap Maker

Crawler Index Directives

Robots.txt Output

What is Robots.txt & Sitemap Generator and How to Use It?

How to Use

Frequently Asked Questions

Recommended Tools

SEO Meta Tags Generator

DNS Lookup & Domain Analyzer

URL Encoder/Decoder

Sign In

Create Account