Web Crawl Accessibility

Ensuring your site is crawlable by AI bots and not blocked via robots.txt.