Why it matters

A public page cannot be understood by a crawler that is blocked from fetching it. Blocking training crawlers is a separate policy choice.

Do it step by step

1
Open /robots.txt on your live domain and identify the public pages you want discovered.
2
Check Googlebot access and test one important URL in Search Console URL Inspection.
3
If AI search matters to you, review OAI-SearchBot and Claude-SearchBot separately from GPTBot and ClaudeBot, which are used for model training. Check CDN bot rules as well.
4
Remove accidental blocks only after confirming the site owner’s crawler policy.

Example

Allow /services/ for search crawlers while keeping /account/ private through authentication.

Check your work

A live public URL and required CSS or script resources can be fetched; the chosen search crawlers are not blocked by robots.txt or the CDN.

Complete when

The intended public URL set is accessible to the chosen search crawlers, and intentional exclusions are documented.

Prompt for an AI coding agent

Use this with an agent that can inspect your project. Account and external actions remain yours to complete.

Inspect my existing website project before editing. Review robots.txt and search crawler access. Open /robots.txt on your live domain and identify the public pages you want discovered. Check Googlebot access and test one important URL in Search Console URL Inspection. If AI search matters to you, review OAI-SearchBot and Claude-SearchBot separately from GPTBot and ClaudeBot, which are used for model training. Check CDN bot rules as well. Remove accidental blocks only after confirming the site owner’s crawler policy. Keep the existing framework and design conventions. Verify the result with the project's checks and by inspecting the rendered page or response. Explain any manual account, hosting, or Search Console steps I must do myself.

Tools and references

Reviewed 2026-11-04