Crawling and crawler access

Let the crawlers you choose fetch public pages and their important resources.

For most websites. Some tasks apply only to live websites.

Start here

Review robots.txt and search crawler access

robots.txt tells crawlers which URLs they may request; firewalls and CDN rules can block them too.

Use real HTML links with an href for navigation to public pages.