How do I allow OpenAI's ad crawlers to access my landing page?

The short version

ChatGPT ads landing pages have to be reachable by OpenAI's own crawlers before an ad can pass review. You must allow OAI-AdsBot, and OpenAI recommends allowing OAI-SearchBot as well. Most blocks come from one of three layers, robots.txt, firewall or bot mitigation, and human verification checks, and the fix is almost always to allowlist the crawler at that layer rather than ask support for an exception.

Why OpenAI crawls your landing page at all

OpenAI uses crawlers to validate the safety of web pages submitted as ads on ChatGPT. When you submit an ad, OpenAI may visit the landing page to confirm it complies with its policies. It may also use content from the landing page to judge when the ad is relevant to show. A landing page the crawler cannot reach is a landing page OpenAI cannot validate, regardless of how the page looks to a real visitor.

This makes crawler access a launch dependency, not an optional technical detail. A perfectly written ad, with accurate copy and a compliant offer, will still fail review if the landing page it points to returns an error, a challenge screen or a blocked response to OAI-AdsBot. Because the check happens automatically during ad review, most advertisers only discover a block exists after a rejection, which is why it is worth confirming crawler access before the first submission rather than after.

Which crawler you actually need

For product feed campaigns, also allow OAI-SearchBot to crawl the product image URLs specifically. OpenAI notes that a 403 response from the image host's firewall, CDN or bot protection can block feed processing even when the main site is fully accessible.

The robots.txt lines OpenAI publishes

Review your robots.txt file and confirm both crawlers are explicitly allowed on the relevant pages and paths. OpenAI's own example is:

User-agent: OAI-SearchBot
Allow: /

User-agent: OAI-AdsBot
Allow: /

If access is disallowed in robots.txt, crawling stops immediately, since OpenAI's crawlers respect these rules. This is the first and simplest layer to check, and it is worth confirming directly rather than assuming an existing robots.txt already covers it, particularly on a site that blocks crawlers broadly by default and allowlists specific ones.

The three layers that block a crawler

Most websites have more than one layer of protection between the public internet and the page itself. Work through each one in order.

1. robots.txt. Confirm OpenAI's crawlers are explicitly allowed on the relevant paths, using the lines above as a template.
2. Web protection and bot mitigation. Services such as Cloudflare, Akamai and similar providers defend against scraping and unauthorised traffic, and can mistake a legitimate crawler for automated abuse, often returning a 403 Forbidden response. Review your firewall configuration and allowlist OpenAI's crawler traffic, ideally based on the published user agent names, and have your infrastructure team inspect bot mitigation rules that could be producing false positives.
3. Human verification and anti bot logic. CAPTCHAs, JavaScript challenges, behavioural analysis and session validation can all block an automated crawler even after it clears the first two layers, since OpenAI's crawlers are themselves automated systems. Review any such logic on the landing page and exempt OpenAI's crawlers where appropriate, again ideally by user agent.

A note on stable IP ranges

Some security systems want crawler traffic to originate from stable, published IP ranges before they will reliably allowlist it. Because crawler infrastructure can change over time, OpenAI advises against relying solely on short term IP observations pulled from logs. Instead, validate traffic using a combination of user agent identification, verified bot programs where your provider supports them, firewall allowlists, robots.txt behaviour and provider level bot verification systems. If a stable IP list is genuinely required, OpenAI points to openai.com/searchbot.json and openai.com/adsbot.json as the reference.

A note on rate limiting

Large batch uploads or a sudden spike in crawler traffic can trigger automated rate limiting or bot protection on its own, separate from any deliberate block. If you suspect this is happening, have your engineering team review HTTP response codes, especially 429 Too Many Requests, alongside firewall or CDN logs, bot mitigation events, request throttling rules and traffic analytics around the time of the crawl attempt. Uploading ads over a longer period in smaller batches can also reduce the chance of triggering it in the first place.

A note on Cloudflare specifically

OAI-AdsBot is officially verified and allowlisted by Cloudflare. If your site sits behind Cloudflare and a crawler block still occurs, the cause is more likely a custom firewall rule, a page rule, or a bot management setting layered on top of Cloudflare's own allowlist, rather than Cloudflare blocking the crawler by default.

Reading the layers as a single checklist

Because the three layers stack on top of each other, a page can pass one check and still fail the next. Working through them as a single checklist, rather than fixing the first thing you find and assuming the rest is fine, saves a second rejection cycle.

A page that fails at robots.txt never reaches the second or third layer, so fixing robots.txt first and re-testing before touching the firewall is the more efficient order, even though all three ultimately need to be correct.

What your engineering team should check first

OpenAI's own troubleshooting order is short: confirm the landing page returns a successful HTTP response to OAI-AdsBot, confirm robots.txt allows the relevant path, and confirm that no WAF, CDN, bot mitigation, JavaScript challenge, CAPTCHA, authentication requirement or geo rule is blocking automated access. Working through these in this order avoids chasing a firewall issue when the actual block is a single disallow line in robots.txt, or the reverse.

What is not a supported landing page

Use a directly reachable web landing page wherever possible. OpenAI flags app store links, deep links, documents, and destinations that require an app, a login, region specific access, or an unsupported redirect as potentially not providing enough crawlable content for validation or review. If your funnel currently sends ChatGPT ads traffic to one of these, plan for a standard web landing page as the ad destination instead, even if the eventual conversion still happens inside an app.

What to do after you fix the block

Do not expect a manual bypass from support. OpenAI states this directly: make the landing page actually crawlable by fixing robots.txt, WAF, CDN, bot mitigation, authentication or rate limiting, since there is no workaround that skips crawler validation. Once the fix is in place, re-upload or resubmit affected ads if their status does not update on its own. For a bulk upload, submitting smaller batches can reduce the chance of hitting a rate limit or bot protection trigger again while your team confirms the fix holds.

Common mistakes and what goes wrong

• Testing only in a browser. A page that loads perfectly for a human tester can still block OAI-AdsBot if a CAPTCHA, JavaScript challenge or session check sits in front of it. Confirm crawler access specifically, not just human access.
• Fixing robots.txt but leaving the firewall untouched. All three layers, robots.txt, bot mitigation and human verification, have to be checked. Clearing one does not guarantee the others are not still blocking the crawler.
• Forgetting the product feed image host. The main site can be fully open while the image CDN, often a different domain or subdomain, still returns a 403 to OAI-SearchBot and silently blocks feed processing.
• Relying on an old IP allowlist. Crawler infrastructure can change, and OpenAI is explicit that short term log based IP observations are not a reliable long term allowlist strategy.
• Assuming support can override a block. There is no manual bypass. The page has to genuinely become crawlable, and ads may need to be resubmitted afterwards even once it is.
• Uploading a large batch immediately after fixing a block. A sudden spike right after a fix can itself trigger rate limiting, which looks identical to the original problem. Consider smaller batches while confirming the fix is stable.

Why this shows up as an ad rejection

In the Ads tab, an ad blocked by a crawler issue typically shows a status of Not serving, with the reason visible on hover, and common causes listed by OpenAI include ad copy or landing pages that do not align with its ad policies, or landing pages that block OpenAI's user agents or IP addresses. If you see a rejection with no obvious policy problem in the copy itself, a crawler block on the landing page is one of the first things worth ruling out, alongside the account and billing checks covered in the ChatGPT ads not showing guide. Getting this right the first time also matters at account setup, since a rejected first ad can delay the review timeline covered in the account setup guide.

A UAE hosting note

Many UAE businesses run their site behind a regional CDN or a hosting provider's own bundled firewall, sometimes on top of Cloudflare, sometimes instead of it. Do not assume a UAE specific configuration is the cause of a block, since OpenAI does not publish any region specific crawler rules. Treat it as the same three layer check as anywhere else, starting with robots.txt, then the firewall or CDN, then any human verification logic, and confirm with your hosting provider directly whether OpenAI's crawler user agents are allowlisted by default or need to be added.

If your landing page sits on a shared hosting plan rather than a dedicated server, the bundled security layer is often managed by the hosting provider's support team rather than being configurable from your own dashboard. In that case, the practical path is a support ticket to the host asking them to allowlist the OAI-AdsBot and OAI-SearchBot user agents specifically, referencing OpenAI's published crawler guidance, rather than trying to locate a firewall setting that may not be exposed to you at all.

Quick answers

Which OpenAI crawler is required for ChatGPT ads?

OAI-AdsBot. OpenAI states it is required for ChatGPT ads landing page validation and review, and advertisers should prioritise it. OAI-SearchBot is recommended in addition, because it may help OpenAI understand public web content.

What does my robots.txt need to say?

OpenAI's own example is User-agent: OAI-SearchBot, Allow: / and User-agent: OAI-AdsBot, Allow: /, confirming both crawlers are explicitly allowed to access the relevant pages and paths.

Why was my ad rejected even though my landing page works fine in a browser?

A landing page that blocks OpenAI's crawlers, through robots.txt, a firewall, bot mitigation or a CAPTCHA, can fail ad review even though a human visitor sees the page normally, because the crawler is what validates the page, not a human tester.

Can OpenAI support manually bypass a crawler block?

No. OpenAI states not to rely on a manual bypass. The landing page has to actually be made crawlable for OAI-AdsBot, by fixing robots.txt, WAF, CDN, bot mitigation, authentication or rate limiting, and ads may need to be re-uploaded after the fix.

Does Cloudflare block OpenAI's ad crawler by default?

OpenAI states that OAI-AdsBot is officially verified and allowlisted by Cloudflare, though other bot protection layers on top of Cloudflare, such as custom firewall rules, can still block it.

Should I allowlist OpenAI's crawlers by IP address?

OpenAI advises against relying solely on short term IP observations from logs, since crawler infrastructure can change. If a stable IP range is required, it points to openai.com/searchbot.json and openai.com/adsbot.json, combined with user agent identification and provider level bot verification.

Are app store links or login protected pages supported as ChatGPT ads landing pages?

OpenAI recommends a directly reachable web landing page whenever possible. App store links, deep links, documents, or destinations needing an app, login, region restricted access or unsupported redirects may not provide enough crawlable content for validation.

Related