OAI-SearchBot
Discovers public pages for ChatGPT search summaries, snippets, citations, and links.
Technical ChatGPT search guide
How to check if OAI-SearchBot is blocked
Test the crawler rule, exact page, and hosting layer that affect ChatGPT search discovery—without treating access as a ranking guarantee.
Quick answer
Check more than the crawler name
- 1. Open the canonical site's public
/robots.txt. - 2. Review any OAI-SearchBot group and the wildcard group.
- 3. Test the exact page path you want discovered.
- 4. Confirm the CDN or firewall returns normal page content instead of an error or challenge.
- 5. Verify that meaningful text is present in the server-rendered response.
Different controls, different purposes
Do not confuse search discovery with model training
GPTBot
Crawls content that may be used to improve OpenAI's generative models.
ChatGPT-User
May fetch a page in response to a user's direct request or interaction.
Three access layers
Robots rules are only the first check
01
Read robots.txt
Check both an OAI-SearchBot group and the wildcard group. A specific path rule can matter even when the homepage is allowed.
02
Request the exact page
Test the public URL you want discovered. Homepage access does not prove that a guide, product page, or directory is reachable.
03
Inspect the hosting layer
A CDN, firewall, bot challenge, server error, or client-only render can block retrieval independently of robots.txt.
A rule that blocks every path
User-agent: OAI-SearchBot
Disallow: /The slash represents the entire site for that user-agent group.
A simple allow rule
User-agent: OAI-SearchBot
Allow: /Do not paste this blindly; existing groups and path rules can change the correct configuration.
Common mistakes
Access is necessary, but it is not visibility evidence
Checking only the homepage
A path-specific rule or hosting behavior can block the resource that matters.
Ignoring wildcard rules
A wildcard group may apply when no more specific user-agent group matches.
Treating robots.txt as security
The file is public and cannot protect private information.
Assuming allowed means cited
Access removes one possible barrier; providers still control retrieval and answer composition.
Evidence notes
Primary sources behind the claims
OpenAI documents crawler purposes and discovery controls. The 100 Questions methodology separates technical readiness from observed answer visibility and does not treat crawler access as a ranking guarantee. Sources reviewed August 20, 2026.
- OpenAI publisher and developer guidanceOpenAI documents OAI-SearchBot access for content discovery in ChatGPT search separately from GPTBot training controls.
- 100 Questions benchmark methodologyDefinitions for eligibility, coverage, mentions, prominence, citations, competitor share of voice, and comparable reruns.
Run the technical preflight
Check crawler access alongside the rest of the public foundation.
The free checker reviews indexability, canonicals, metadata, schema, sitemaps, question content, AI crawler rules, and optional llms.txt discovery.
Contextual next steps
Continue from openai crawler troubleshooting
Diagnose robots, path, CDN, firewall, HTTP, and rendering blocks for OAI-SearchBot.
01
Run the free technical checker
Check indexability, crawler rules, canonicals, schema, sitemaps, and llms.txt.
02Improve recommendation readiness
Connect crawl access to entity clarity, evidence, reviews, and relevant citations.
03Audit the measurement method
Review eligibility, denominators, coverage, citations, retention, and like-for-like rerun rules.
Common questions
Use the resource without overstating the result
What is OAI-SearchBot?
OAI-SearchBot is OpenAI's crawler for discovering public content that may be surfaced in ChatGPT search experiences. Allowing access removes one technical barrier; it does not guarantee a citation, recommendation, ranking, or visit.
Is OAI-SearchBot the same as GPTBot?
No. OpenAI documents OAI-SearchBot for search discovery and GPTBot for potential model training. A site can make different choices for those user agents.
Does an Allow rule guarantee that OpenAI can fetch the page?
No. Robots rules express crawl preferences, but a CDN, web application firewall, authentication layer, server error, or rendering failure can still prevent a successful fetch.
Should private content be protected with robots.txt?
No. Robots.txt is public and is not access control. Protect private or sensitive content with authentication and appropriate server-side authorization.
Apply this resource
Put oai-searchbot access into a measured workflow
Preserve the questions, conditions, answers, citations, and failures so the next decision rests on inspectable evidence.