SEO tools
CrawlCheck
Free AI crawler audit and robots.txt generator for 60+ AI crawlers
- 0community votes
- 23DR by Ahrefs +3 (up) since last check
Ahrefs Domain Rating listed 6 Nov 2026, relisted 2 Nov 2026.
Free toolkit to audit your robots.txt against 60+ AI crawlers and generate policies that control what ChatGPT, Perplexity and Claude can see on your site.
What it is
CrawlCheck: the homepage
CrawlCheck is a free toolkit for Generative Engine Optimization that helps site owners control what AI crawlers can access. It audits robots.txt files against a database of 60+ AI crawlers including GPTBot, ClaudeBot, PerplexityBot and Bytespider, showing exactly which crawlers are allowed or blocked and what it costs for visibility in ChatGPT, Perplexity, Claude and other AI answer engines.
The toolkit includes an AI Crawler Auditor for parsing robots.txt against known AI bots, a Generator for creating clean policies, a Schema tool for generating FAQ and HowTo JSON-LD, and a Scanner for live-testing your site as 8 AI crawlers. All core tools run entirely in your browser with no signup required.
A paid Pro Deep Report option ($49, one-time payment) sweeps up to 50 pages of a site and live-tests 8 AI crawlers on every page to catch cloaking, broken pages and noindex traps, returning a prioritized fix list. The company offers a 30-day money-back guarantee on all purchases.
Listed capabilities
- AI Crawler Auditor — paste robots.txt and get verdict for every AI crawler per path
- Structured Data Generator for FAQ and HowTo schema
- AI Crawler Scanner with live-testing of 8 crawlers per page
- Directory of 61 AI crawler profiles with operator, purpose and blocking instructions
Pricing
Free Scan: $0 (audit one URL with 8 AI crawlers, llms.txt and sitemap health check). Pro Deep Report: $49 one-time (sweeps up to 50 pages, live-tests 8 AI crawlers per page, returns prioritized fix list).
Follow CrawlCheck for an email about each milestone, update and new product they post, in a daily summary.
CrawlCheck metrics
| Metric | Value |
|---|---|
| Domain Rating | 23 +3 (up) since last check |
| Trust Flow | 7 no change since last check |
| Citation Flow | 19 no change since last check |
| Referring domains | 24 no change since last check |
| External backlinks | 53 no change since last check |
Ahrefs Domain Rating listed 6 Nov 2026, relisted 2 Nov 2026. Listed under topic: Computers/Internet/On the Web (7). Majestic listed 25 Oct 2026, relisted 25 Oct 2026. Link data listed by Majestic. Trust Flow, Citation Flow and Topical Trust Flow are trademarks of Majestic-12 Ltd.
CrawlCheck domain listing data
Show the listing data
| Domain age | 1 month |
|---|---|
| Registered | |
| Expires | |
| Registrar | CloudFlare, Inc. |
| DNSSEC | Not signed |
CrawlCheck launch receipt entry
| Measure | At launch, then latest |
|---|---|
| Domain Rating (Ahrefs) | 20 to 23 +3 (up) since launch 25 Oct 2026, then 6 Nov 2026 |
| Trust Flow (Majestic) | 7 first measured 25 Oct 2026 |
| Referring domains (Majestic) | 24 first measured 25 Oct 2026 |
Day 11 since launch. Next receipt: day 30, 24 Nov 2026. We show what changed, not what caused it: many things affect these numbers. Nothing here is estimated.
Before you launch
The playbook on SaaS Directory List lists what to prepare, what to post on launch day and where else to list your product.
Compare and explore
Quick answers
What is GEO (Generative Engine Optimization)?
GEO is the practice of making your content easy for AI answer engines like ChatGPT, Perplexity, Claude and Gemini to find, trust and cite. The foundation is technical accessibility: a robots.txt policy that lets the right AI crawlers in.
Does blocking GPTBot hurt my Google rankings?
No. GPTBot only trains OpenAI models, while Google Search uses Googlebot, a separate crawler. Blocking GPTBot has zero effect on your Google rankings, and blocking Google-Extended does not affect Search either.
Some AI crawlers ignore robots.txt — what can I do?
robots.txt is a request, not a wall. For crawlers known to ignore it like Bytespider, you can enforce blocks at the CDN or WAF level, for example using Cloudflare user-agent rules.
Is my data sent anywhere?
It depends on the tool. The robots.txt Auditor and Generator run entirely in your browser, so nothing is uploaded. The Scanner needs one server round-trip to fetch your page as each AI crawler, and the Pro Deep Report does the same across up to 50 pages; results are returned to your browser and not stored.
Leave a review for CrawlCheck
No reviews listed for CrawlCheck yet. List the first one.
Listing a review is free. No one pays for reviews. See the listed reviews
List reviews · List updates · Share the list · Your tool? Claim this list entry · Report this list entry
CrawlCheck has a free listing. The owner can unlock the full profile with a paid launch for $5:
- Founder story (locked)
- Milestones (locked)
- Demo video (locked)
- Dofollow website link (locked)
Also on the list
-
DR 19 +1 (up) since last check 2 votes. Log in to move bisibility up the list
-
Number 2: OpenSEOOpen-source SEO tools you can run yourself, read every line of, and hand to your AI agent
DR 5 +1 (up) since last check 1 votes. Log in to move OpenSEO up the list -
Number 3: schemawriter.aiAI schema markup generator to automate JSON-LD webpage schema
SEO toolsDenmarkSince 2023
DR 43 −1 (down) since last check 1 votes. Log in to move schemawriter.ai up the list -
DR 30 no change since last check 1 votes. Log in to move DR Checker up the list
