Does your site load in 4-5 seconds? Google still tolerates it, but AI systems like Perplexity don't — they discard slow pages without even trying, and yours never gets considered as a source for an answer. It's not a ranking problem: it's an access problem. Either you're fast enough to be read, or for the AI you don't exist — period. Three specific technical issues almost always cause this block, they can be measured in a few minutes, and they're fixed with a clear set of priorities.
You’ve optimized your site for Core Web Vitals, your PageSpeed score is green, Google rewards you with good rankings. Then you ask Perplexity to recommend someone in your field and you don’t show up. The problem might not be the content. It might be that Perplexity’s crawler tried to read your page, waited, and after a few seconds gave up.
Not demoted. Discarded. That’s a huge difference.
Generative AI engines that perform real-time retrieval — Perplexity, Bing Chat, the newer agentic systems — don’t have Googlebot’s patience. Googlebot crawls, indexes, and then evaluates at its own pace. A real-time RAG system has a tight time budget: it has to fetch the page, extract the content, pass it to the model, and generate an answer. All in a few seconds. If your page takes too long to serve the content, the system moves on to the next source. It doesn’t penalize you — it ignores you.
Optimized pages: the data point that changes the perspective
In 2026, Volpini et al. published a paper that puts the relationship between a page’s technical quality and the AI’s accuracy in extracting information in black and white. The most significant finding concerns the impact of optimized pages on retrieval:
“Enhanced pages achieved +29.6% accuracy improvement for standard RAG and +29.8% for the full agentic pipeline.”
Almost 30% more accuracy. We’re not talking about a marginal improvement — it’s the difference between an answer that includes you with the correct data and an answer that cites you incorrectly or doesn’t cite you at all.
And the crucial point is what “enhanced” means in this context. It’s not better content. It’s content that’s served better: clean formatting, clear structure, fast rendering. The quality of the information may be identical, but if the page delivers it more efficiently, the system uses it with greater precision.
Why format matters more than you think
There’s a data point in the same paper that at first glance seems counterintuitive, but that has direct implications for how you should think about your site:
“The agent provides substantial lift on poorly formatted documents but when document format is optimized, the agent adds negligible accuracy lift.”
In plain terms: when the page is well formatted, the basic RAG system manages to extract the information correctly without needing a sophisticated AI agent to do the dirty work. The agent serves as compensation when the document is a mess — but if the format is already optimized, the agent adds almost nothing.
From this follows an important operational takeaway: a technically optimized page is read correctly by the simplest and fastest system. It doesn’t need the full agentic pipeline, which requires more time and more resources. This means your page is more likely to be included in the answer, because the system can process it within its standard time budget without resorting to additional steps.
If your main text depends on React, Angular, or Vue without server-side rendering, that content might simply not exist for the crawler.
Fewer links, same quality: crawler efficiency
The third data point from Volpini et al.’s paper closes the loop and reveals a behavior you should consider carefully:
“Agents using enhanced pages achieved remarkable efficiency: they follow fewer links while maintaining performance.”
When pages are optimized, AI agents follow fewer links to get the same result. This is the most important signal for understanding how to think about page experience from an AI standpoint.
An AI crawler with a limited time budget also has a limited navigation budget. It can follow a certain number of links before it has to return a result. If your page is well structured and the content is immediately accessible, the crawler gets what it needs on the first attempt. If your page is slow, requires heavy JavaScript rendering, or hides the content behind interactions, the crawler has to work harder — and it might decide it’s not worth the effort.
This isn’t Google’s behavior, which has weeks to crawl and recrawl. This is the behavior of a system that has to respond in real time to a user’s question. The difference in context changes everything.
Think about it: if a user asks Perplexity “which companies offer service X?”, the system has to fetch, read, and synthesize information from multiple sources in a few seconds. It doesn’t have the luxury of coming back tomorrow. If your page takes longer than the others to deliver the content, you’re the one who gets cut — not because your content is worse, but because the system found equivalent answers from faster sources.
The text you want the system to read must be in the initial HTML, not injected afterward.
What really slows down a page for an AI crawler
Here I have to be specific, because the bottlenecks for an AI crawler aren’t exactly the same as those for a human user. You might have a site that loads fine for someone visiting from mobile — but that’s a wall for a bot trying to extract structured text in two seconds.
- Client-side JavaScript rendering. If your main content is loaded by JavaScript after the first render, a traditional crawler might not execute it at all. Googlebot executes JavaScript — but with a delay. A real-time AI crawler might not, or might do so with a much tighter timeout. If your main text depends on React, Angular, or Vue without server-side rendering, that content might simply not exist for the crawler.
- Interstitials and overlays. Full-page cookie banners, subscription pop-ups, promotional overlays. For a user they’re annoying. For a crawler trying to extract the main content, they’re noise that overlaps the signal. In the best case the crawler ignores them. In the worst case, it extracts the interstitial’s text instead of the real content.
- LCP above 2 seconds. Largest Contentful Paint is the moment when the main content is visible. If it exceeds 2 seconds, you’re already in the critical zone for AI crawler timeouts. I’m not talking about an aspirational target — I’m talking about an operational threshold below which the system might decide not to wait for you.
- Unstable layout (high CLS). A high Cumulative Layout Shift means the page moves while it loads. For a system trying to extract content from a specific position in the DOM, a shift can mean extracting the wrong block.
- Redirect chains. Every redirect adds latency. One redirect is tolerable, three are a problem. An AI crawler with an already tight time budget doesn’t want to spend half a second following bounces between URLs before reaching the content. And if the chain includes a redirect from HTTP to HTTPS (I talked about this in the article on the HTTPS protocol), you’re adding a step that could be avoided with a proper server configuration.
The check to do this week
Open PageSpeed Insights and look at your Core Web Vitals — not for the overall score, but for the individual values. LCP under 2 seconds, CLS near zero, no interstitials covering the content in the first few seconds. This is the bare minimum.
Then do a more targeted test: disable JavaScript in your browser and reload your main pages. If the content disappears, you have a serious problem with AI crawlers. The text you want the system to read must be in the initial HTML, not injected afterward.
Finally, try having Perplexity analyze your pages. Ask something specific that is written only on your site. If Perplexity manages to retrieve the information, the crawler reaches it. If it can’t — and the information is clearly present on the page — the problem is technical.
These checks give you a direction, but a complete analysis of how AI crawlers interact with your site requires tools and expertise that go beyond the self-check. Every site has different bottlenecks, and finding them requires a dedicated technical analysis.
The thread that connects everything: from technical credibility to visibility
Page experience for AI isn’t an isolated topic. It’s part of a system of technical signals that AI engines evaluate before they even read your content. If the page doesn’t load in time, it doesn’t matter how good what you wrote is. If it loads but the content is behind JavaScript, the crawler doesn’t see it. If it sees it but the structure is chaotic, the extraction is imprecise.
I’ve written a series of deep dives on these technical signals to help you build a site that the AI can actually read: from the HTTPS protocol as a trust signal, to crawlability for AI bots that determines what the system can reach, all the way to the semantic markup that guides content extraction and the content freshness that tells the model whether your data is still current.
Every piece counts. And page experience is the first filter: if you don’t pass it, the rest isn’t even evaluated.
