Free guide · part of the Claude Code Course
Get the full Claude Code Course
Every build, every prompt, the security setups, and every future section — yours forever for one payment. This guide is a free taste of what’s inside.
Make Your Site Show Up in AI Search
I asked Claude Code to scrape my own website this week and the scraper got blocked at the door. Your site is probably invisible to AI search the same way, and there is a 15 minute fix.
By Ryan Frizelle · 4 min read
I asked Claude Code to scrape my own website this week and the scraper got blocked at the door. Your site is probably invisible to AI search the exact same way and you have no idea.
Here is what you will do:
- 1.Ask Claude Code to scan your live site and check if AI crawlers can actually read it.
- 2.Have Claude fix every block in one shot. Your robots.txt, your llms.txt, your hosting rules, all of it.
- 3.Verify ChatGPT, Claude, and Perplexity can now pull from your site.
This is a 15 minute fix that will compound for years. 100% free. No new tools to install. Just one prompt in Claude Code on your existing project.
Audit and unblock your site for AI search
I want my website to actually show up in AI searches like ChatGPT, Claude, Perplexity, and Google AI Overviews. Right now I am not even sure if AI crawlers can read my site. Here is what I need you to do.
1. Try to fetch my live site at [PASTE YOUR DOMAIN HERE] as if you were an AI crawler. Tell me if anything is blocked, returning an error, or missing.
2. Check my robots.txt file (or create one at the root of my project if I do not have one) and confirm every major AI user agent is explicitly allowed. That means GPTBot (OpenAI), ClaudeBot and Claude-Web (Anthropic), PerplexityBot, Google-Extended (Google's AI crawler), CCBot (Common Crawl), Applebot-Extended, and Bytespider. If any are missing, add them.
3. Create or update an llms.txt file at the root of my site. This is the file AI tools look for to understand who I am and what my site is about. Include a clean summary of my name, what I do, and links to my most important pages.
4. Check my hosting setup for any firewall rules, Cloudflare bot protection, or middleware that might be blocking AI scrapers. If you find any, walk me through allowing AI bots through.
5. After everything is in place, scan the site one more time and confirm AI can now read it end to end.
Walk me through every step. Explain what each file does in plain English so I learn the why. I am not a developer.Liked this? There's 50x more in the course.
One-time purchase. Keep forever.
That one prompt is the whole fix. Claude scans your live site like an AI crawler would. Finds every block, every misconfigured file, every firewall rule that is silently shutting AI out. Then it writes the right robots.txt and llms.txt for your site and walks you through any hosting fixes.
Here are the AI bots you actually want crawling your site:
- 1.GPTBot. OpenAI's crawler. This is what feeds ChatGPT search results.
- 2.ClaudeBot. Anthropic's crawler. Feeds Claude, plus Claude Code when it browses for you.
- 3.PerplexityBot. Perplexity's crawler. The fastest growing AI search engine right now.
- 4.Google-Extended. Google's separate AI crawler for Gemini and AI Overviews in search.
- 5.CCBot. Common Crawl. The open dataset behind a huge chunk of every AI model training.
Verify everything is working
Test that my robots.txt and llms.txt are live and correct. Fetch https://[YOUR DOMAIN]/robots.txt and https://[YOUR DOMAIN]/llms.txt directly. Confirm both files load, list every AI user agent that is currently allowed, and flag anything missing. Then simulate a request from GPTBot, ClaudeBot, and PerplexityBot to my homepage and tell me if each one gets a clean response.Pro tip. Run this same audit every time you launch a new site. Most hosting platforms ship with default rules that quietly block AI bots, so make this your day-one ritual.
That is the whole thing. Run the audit. Let Claude fix the blocks. Verify it works. Now when someone asks ChatGPT or Claude about your business, your site is actually in the answer. Not even exaggerating.. this is a 15 minute fix that pays you back for years.
Want more 15 minute fixes like this that turn Claude Code into your personal site operator?
Liked this? There's 50x more in the course.
One-time purchase. Keep forever.
Keep going
More walkthroughs from the library.
Microsoft Clarity + Claude Code Setup
Connect Microsoft Clarity to Claude Code. Your site gets smarter off real user behavior, dead clicks, rage clicks, and session replays you can act on.
The brand.md File for Claude Code
Your AI-built site looks generic because Claude Code has no brand to follow. This is the one file that fixes it before a single line of design gets written.
The CLAUDE.md Blueprint
The one file that makes Claude Code build like a senior developer. This is the single biggest difference between AI slop and production-quality apps.