AI Crawl Control
Every site on Cloudflare now has access to AI Audit, which summarizes the crawling behavior of popular and known AI services.
You can use this data to:
- Understand how and how often crawlers access your site (and which content is the most popular).
- Block specific AI bots accessing your site.
- Use Cloudflare to enforce your
robots.txtpolicy via an automatic WAF rule.
To get started, explore AI audit.
Was this helpful?
- Resources
- API
- New to Cloudflare?
- Directory
- Sponsorships
- Open Source
- Support
- Help Center
- System Status
- Compliance
- GDPR
- Company
- cloudflare.com
- Our team
- Careers
- © 2025 Cloudflare, Inc.
- Privacy Policy
- Terms of Use
- Report Security Issues
- Trademark