Crawlers
Who is reading the farm: AI crawler traffic, IP verification and the robots policy that invites them in.
7-day crawl volume by bot
Daily crawl volume per bot, Aug 11–17. Highlighted column is today.
Verification failures
Hits whose source IP could not be attributed to the claimed bot.
Amazonbot requests have failed IP-range verification since 14.08. Unverified hits are excluded from tenant-facing crawl and mention stats until attribution is confirmed — spoofed crawlers must never inflate a tenant's numbers.
Tracked in incidents →Robots policy
The farm is built to be crawled — every AI agent is explicitly welcomed.
robots.txt → Allow: / · Sitemap: /sitemap.xml · llms.txt published
No machine-facing surface is disallowed. Rate shaping happens at the edge, never through robots exclusion.
Recent hits
Raw hit log across all farm surfaces, newest first.
| Bot | Path | Verification | When |
|---|---|---|---|
| GPTBot | /f/nimbus-dental-tlv.md | verified | 5h ago |
| OAI-SearchBot | /f/nimbus-dental-tlv | verified | 6h ago |
| ClaudeBot | /llms.txt | verified | 8h ago |
| PerplexityBot | /f/nimbus-dental-tlv | verified | 10h ago |
| Claude-User | /f/nimbus-dental-tlv.md | verified | 11h ago |
| Googlebot | /f/nimbus-dental-tlv | verified | 13h ago |
| Amazonbot | /llms.txt | failed | 17h ago |
| ChatGPT-User | /f/nimbus-dental-tlv#faq | verified | 19h ago |