Control what AI sees on your site

ChatGPT, Perplexity and Claude now decide who gets discovered. Audit your robots.txt against 60+ known AI crawlers, then generate a policy in seconds — free, no signup, runs entirely in your browser.

AI Crawler Auditor

Paste your robots.txt below. We parse it against our database of 60+ AI crawlers and show exactly who is allowed, who is blocked, and what it costs you.

robots.txt Generator

Pick a preset or fine-tune crawler by crawler. The file updates live — copy it to https://yourdomain.com/robots.txt when you're happy.

Generated robots.txt 


      

        
        
      

    

  


  

    

      
Structured Data Generator

      
FAQ, HowTo and Article schema are what AI answer engines quote most easily. Fill the form, paste the JSON-LD into your page's <head>.

      

        
        
        
      


      

        

        

      


      

        

        

        
        

        

      


      

        

        

        

          

          

        

        

          

          

        

      

    

    

      
Generated JSON-LD

      

      

    

  


  

    
Why this matters now

    

      

        +800%
        
AI referral traffic is exploding

        
Referrals from LLM apps grew faster than any channel in 2025–2026. Early sites are capturing citations that compound as AI search becomes the default discovery layer.

      

      

        1 line
        
Your robots.txt decides it

        
One Disallow: / can make you invisible across ChatGPT, Perplexity and Claude at once. Most sites were configured years ago, before AI crawlers existed.

      

      

        60+
        
Dozens of crawlers, no visibility

        
GPTBot, ClaudeBot, PerplexityBot, Bytespider, CCBot, Google-Extended… nobody audits them by hand. CrawlCheck parses your file against a maintained database in one click.

      

    

  


  

    
FAQ

    

      

        
What is GEO (Generative Engine Optimization)?

        
GEO is the practice of making your content easy for AI answer engines — ChatGPT, Perplexity, Claude, Gemini — to find, trust and cite. The foundation is technical accessibility: a robots.txt policy that lets the right AI crawlers in.

      

      

        
Does blocking GPTBot hurt my Google rankings?

        
No. GPTBot only trains OpenAI models. Google Search uses Googlebot, a separate crawler. Blocking GPTBot has zero effect on your Google rankings, and blocking Google-Extended does not affect Search either.

      

      

        
Some AI crawlers ignore robots.txt — what can I do?

        
robots.txt is a request, not a wall. For crawlers known to ignore it (like Bytespider), enforce blocks at the CDN or WAF level, e.g. Cloudflare user-agent rules.

      

      

        
Is my data sent anywhere?

        
No. All parsing, auditing and generation runs locally in your browser. Nothing is uploaded, logged or tracked.