We recently checked a business website and got a security challenge instead of the public page. The firewall may have worked exactly as configured. It also blocked our scan.
Customers now ask AI tools to recommend nearby providers, compare services, confirm hours, and find contact details. If an AI tool cannot read your site, it may rely on an old directory listing, incomplete search data, or a competitor's clearer website.
Not every bot does the same job
Some crawlers build search indexes. Some fetch a page because a person asked a question. Others collect training data, monitor uptime, audit accessibility, or generate link previews. You can give each type different access.
Search
Indexes public content so it can be discovered and cited later.
Agent
Visits a page in real time on behalf of a person completing a task.
Training
Collects content to train or fine-tune a model.
Cloudflare now classifies verified automation by behavior, including Search, Agent, Training, SEO, and Accessibility. Its controls let you choose which verified services can read which public pages.
You can allow discovery without agreeing to training
OpenAI uses OAI-SearchBot for search discovery and GPTBot for potential model training. You can allow search discovery and block training. Other providers publish their own crawler identities and controls.
Keep account pages, checkout flows, admin areas, private documents, and sensitive endpoints protected. Let approved crawlers reach the pages that explain who you are, what you offer, where you work, and how to contact you.
What a security challenge tells us
One failed check does not prove that every search engine or AI assistant is blocked. Public search indexes may still contain enough information for a limited assessment.
If you have been hacked, caution is reasonable
If your business has dealt with spam submissions, fake orders, stolen passwords, scraping, cyber fraud, or a breach, “allow bots” sounds like bad advice. You do not need to turn off your firewall or open private areas of your site to every automated visitor.
Give approved search crawlers narrow access to public business pages. Keep logins, forms, customer records, payment pages, admin tools, and private files behind the protections you already use.
Allow access to public service, location, about, FAQ, and contact-information pages.
Use your provider's verified-bot controls, published IP ranges, or signed-agent checks.
Continue blocking suspicious requests, abusive rates, unknown scrapers, and sensitive paths.
Review firewall logs and response codes so you can tighten access without guessing.
A practical access checklist
- Decide what must be public.Prioritize your home, service, location, about, FAQ, and contact pages.
- Check robots.txt and your firewall.An allow rule in robots.txt cannot override a CDN challenge, WAF block, CAPTCHA, login, or geographic restriction.
- Use verified crawler identities.Allow useful search and audit traffic while blocking unwanted scraping and training.
- Test the result.Confirm that approved crawlers receive the page and a successful status instead of a challenge or 403.
- Keep the facts current.Use clear names, services, locations, hours, and contact details.
A working website can still be hard to find
An AI assistant may still answer when it cannot read your site, but it has weaker evidence. Known Space checks what selected AI search tools can find, understand, and recommend at a point in time. If we cannot read your site directly, we say so and continue with the public information we can find.
Technical references
Bot policies and product behavior change. Use the provider documentation as the source of truth.
Check your business