Search and training use different crawlers
A crawler is software that fetches web pages. OpenAI separates search discovery from potential model-training use.
- OAI-SearchBot
- Finds pages for ChatGPT search. Allow the public pages you want it to read.
- GPTBot
- Collects content that may be used for model training. You can block it while allowing search.
- ChatGPT-User
- Visits pages for certain user requests. It does not control search inclusion; robots.txt rules may not apply.
These names apply to OpenAI. Other providers have their own controls. Permission never guarantees a mention or recommendation.
1. Find the rules for your exact website
Open https://your-business.example/robots.txt, replacing the example domain with yours. The file belongs at the website root.
Rules apply to that protocol, host, and port. Your www address and another subdomain can have different files.
Save the current file before editing. WordPress may generate it through an SEO plugin instead of a physical file.
2. Check one public service page
Imagine a plumbing business with a public page at /services/boiler-repair/. This is a hypothetical example.
Find the crawler's applicable User-agent group. A matching Disallow: /services/ rule could block that page.
Disallow: / blocks every path for that group. A specific crawler group can override the general User-agent: * group.
Check the complete group before changing anything. Keep intentional restrictions and remove only the accidental service-page block.
3. Separate search access from training
This example allows search crawling of public paths and blocks GPTBot. It repeats the example's restricted paths in both relevant groups.
User-agent: OAI-SearchBot
Disallow: /account/
Disallow: /private/
User-agent: GPTBot
Disallow: /
User-agent: *
Disallow: /account/
Disallow: /private/Paths without a matching block remain crawlable. The boiler-repair page is therefore available to OAI-SearchBot in this example.
Adapt the paths to your website. Merge changes with existing groups and sitemap lines; do not replace your file wholesale.
Robots.txt is public and voluntary. Protect private information with authentication, not these rules.
4. Check what your host actually serves
- Open robots.txt and your service page while signed out. Look for errors, login walls, or human-check screens.
- Ask your host to check bot requests for denied access, rate limits, and security challenges.
- Use OpenAI's published search-bot IP ranges when reviewing security rules. A user-agent name alone can be imitated.
- Keep your firewall enabled. Make a narrow change for verified crawler requests, then inspect the logs again.
A successful browser visit does not prove a crawler can enter. Check both routes before calling access fixed.
Robots.txt controls crawling, not guaranteed removal from search. A blocked URL can still appear without its page content.
Where Tin GEO fits
Tin GEO's documented plan includes robots-file validation, crawler permissions, and accidental-block checks.
Its crawler checklist also names training bots. Decide your training policy separately before accepting any proposed permission change.
These are documented checks, not proof of completed work on your website. Allowing access is only one part of discovery.