About PathIntactBot

PathIntactBot/1.0 (+https://pathintact.com/bot)

What it does

PathIntactBot fetches pages of documentation sites that a PathIntact customer asked us to monitor, plus the pages those docs link to, so we can report broken links, missing anchors, orphan pages, missing redirects, sitemap problems and on-page SEO issues. It also powers the free tools on this site, which are capped at 50 pages per run.

How it behaves

• Reads and obeys robots.txt, including Disallow rules for PathIntactBot and *, and Crawl-delay (capped at 5 seconds).
• At most 3 concurrent requests per host with a 250 ms delay between requests to the same host; at most 2 concurrent requests to any external host.
• Backs off on HTTP 429 and 503 and respects Retry-After.
• Fetches HTML and sitemap files only. Images, fonts and media are never downloaded, even when a page is rendered in a browser.
• Never submits forms, never signs in, and never crawls pages behind authentication.
• Requests time out after 15 seconds and bodies larger than 5 MB are abandoned.

How to block it

Add this to your robots.txt:

User-agent: PathIntactBot
Disallow: /

Or block a section only:

User-agent: PathIntactBot
Disallow: /internal/

Changes take effect on the next crawl. Site owners can also delete their site from the dashboard, which stops all scheduled crawls immediately.

Questions

Email [email protected] with the host name and we will look into it.