Summarize with AI:

Robots.txt Manager

Manage robots.txt and crawler hints from the dashboard—no FTP.

Overview

Robots.txt Manager lets you edit and manage your site’s robots.txt file from the WordPress admin. No FTP or file editor needed. Add or edit directives for search engine crawlers. Reference your sitemap. Block or allow specific paths. The module typically serves a virtual robots.txt (via rewrite rules or a query) so you do not need a physical file. Changes take effect immediately after saving.

Who is this for?

Site owners and SEOs who need to control crawler access via robots.txt without editing files directly. Useful for blocking admin, wp-includes,
other paths from indexing.

Features

Edit robots.txt from Admin

Text area or builder to edit robots.txt content. Save and it is served at yourdomain.com/robots.txt.

Sitemap Reference

Add your sitemap URL (e.g. Sitemap: https://yoursite.com/sitemap.xml). Helps search engines find your sitemap.

Standard Directives

User-agent, Allow, Disallow. Follow robots.txt standards.

Virtual File

Typically served via WordPress (rewrite or query). No physical file required. Overrides default WordPress robots.txt if present.

Use Cases

CrawlerBlocking
Block wp-admin from crawlers

PathBlocking
Block wp-includes, uploads, or other paths

SitemapReference
Add sitemap reference

CrawlerControl
Customize crawler behavior

Frequently Asked Questions

Does robots.txt block indexing?
robots.txt tells crawlers what not to crawl. It does not guarantee non-indexing. For sensitive content, use noindex meta or .htaccess. robots.txt can be ignored by some crawlers.
What if I have a physical robots.txt?
The module typically overrides the default. Check how your host serves robots.txt. The module may take precedence when enabled.