SeenandCited

Rendering and JavaScript Pitfalls

Understand how client-side rendering (CSR) and complex JavaScript frameworks hinder AI crawlers and LLM data fetchers from indexing and citing your core content.

12 min read
Foundations

The Visibility Gap in the Age of AI

While traditional search engines like Google have become increasingly adept at rendering JavaScript, the same cannot be assumed for the burgeoning ecosystem of AI models and 'Generative Experience' (GE) crawlers. In the AI Visibility Practitioner's world, we distinguish between standard SEO crawling and AI fetchers used by Large Language Models (LLMs) like GPT-4, Claude, and Perplexity. These bots often operate under strict resource constraints, leading to a 'rendering gap' where your content is physically on the page for a human user but effectively invisible to the AI that needs to cite it.

This lesson explores why relying heavily on Client-Side Rendering (CSR) is a high-risk strategy for AI visibility and how to audit your site to ensure your data is 'scraper-ready' and easily digestible by the engines driving the next generation of search.

The Technical Conflict: Client-Side vs Server-Side

To understand the pitfall, we must look at the delivery mechanism. In a Client-Side Rendering (CSR) environment (common with React, Vue, or Angular), the server sends a nearly empty HTML shell and a massive JavaScript bundle. The browser then executes the script, fetches data from an API, and builds the Document Object Model (DOM) locally.

Why this fails for AI Fetchers:

  1. Execution Timeouts: AI agents often use 'headless' browsers or simple HTTP request libraries. To save on compute costs, they may wait only a fraction of a second for JavaScript to execute. If your meaningful content (e.g., a product review or a technical guide) hasn't rendered within that window, the bot sees a blank page.
  2. Stateless Crawling: Some LLM training pipelines use stateless scrapers that do not handle cookies, local storage, or session-based JS triggers effectively. If your content requires a user interaction (like clicking a 'Read More' button driven by JS), it will never be indexed.
  3. Shadow DOM Complications: If your site uses Web Components and the Shadow DOM, many scrapers struggle to 'pierce' the shadow root to extract text, leading to missing citations even if the page is technically rendered.

Identifying JavaScript Bottlenecks

A central task for a Practitioner is identifying which parts of a client’s site are 'JS-dependent'. If the core value proposition of a page—the data that should be cited—is trapped inside a JSON response that requires script execution, you are invisible to any bot that doesn't fully execute JS.

The 'No-JS' Litmus Test

Using a browser extension or developer tools, disable JavaScript and reload the page. What remains?

  • The Visibility Gold Standard: All textual content, tables, and structured data remain visible and readable.
  • The Risk Zone: The navigation remains, but the main content body disappears (replaced by a loading spinner or a blank div).
  • The Failure State: The page is entirely blank or displays a 'JavaScript Required' error.

Worked Example: The 'Hidden' SaaS Documentation

Imagine a SaaS client whose documentation is built using a Single Page Application (SPA) architecture.

The Setup:

  • The URL is example.com/docs/api-integration.
  • The server sends a file containing <div id="app"></div>.
  • A script fetches the actual documentation text from a private API endpoint.

The AI Result: When an AI bot like OpenAI's GPTBot visits the site, it spends 400ms waiting for the page to load. Because the documentation text only appears at 1200ms (after the API call completes), the bot records the page as 'Empty' or 'Low Content'. Consequently, when a user asks ChatGPT 'How do I integrate Example.com's API?', the AI cannot find the source and either hallucinate or state it doesn't have the information—even though the info is 'live' on the web.

The Fix: Shifting the client to Server-Side Rendering (SSR) or Static Site Generation (SSG). By pre-rendering the documentation text into the HTML on the server, the bot sees the full text immediately upon the first byte transfer, requiring zero JavaScript execution to read the content.

Best Practices for AI Discoverability

To ensure your content is 'Cited', you must move away from 'Fragile Rendering'. Follow these architectural requirements:

1. Hybrid Rendering (Hydration)

Use frameworks like Next.js or Nuxt.js. These pre-render the initial HTML on the server so it is immediately readable by any simple scraper, but then 'hydrate' into a fully interactive JavaScript app for the human user. This provides the best of both worlds.

2. Semantic HTML Over JS Components

Avoid using JS-based UI elements for core data. For example, do not use a custom <js-table> component if a standard HTML <table> can do the job. Bots are trained on semantic HTML patterns; they 'understand' the relationship between <th> and <td> without needing to execute code.

3. Progressive Enhancement

Ensure that the core message of the page is delivered as text/HTML. Enhancement (animations, interactive calculators, dynamic filters) should be the 'icing', not the 'cake'.

Common Pitfalls in Data Fetching

Beyond basic rendering, how you fetch data matters:

  • Scroll-triggered Loading (Infinite Scroll): AI bots generally do not scroll. If your content only appears when a user reaches the bottom of the page, the bot will only see the first few items.
  • Personalised JS Content: If your content changes based on a user's location or browser history via JS, an AI bot (usually crawling from a US-based data centre with no history) may see a generic 'Hello World' page instead of your high-value regional content.

Putting it into Practice

  1. Audit the Core: Select the top 10 pages you want AI to cite. Perform the 'No-JS' test on each. Document any missing content blocks.
  2. Monitor Fetcher Behaviour: Check your server logs for user agents like GPTBot, Claude-Web, or PerplexityBot. Look for 200 status codes vs unexpected 404s or 403s.
  3. Evaluate Document Object Model (DOM) Size: Excessive JS often leads to bloated DOMs. A DOM with 3,000+ nodes can cause AI parsers to truncate the content. Aim for a lean, semantically logical structure.
  4. Implement Schema Markup: Since schema is often injected via JSON-LD in the <head>, it is generally more robust for bots than dynamic DOM content. Ensure your most important data points (Product price, Author name, Steps) are in the source HTML as JSON-LD.

Visual diagram

[ diagram placeholder ]

A workflow diagram showing the 'Path to Invisibility' for CSR where a bot sees only an empty shell, contrasted with the 'Path to Citation' for SSR where a bot instantly receives full text and schema.

Exercise

Navigate to your company's or a client's most important service page. Open 'Developer Tools', disable JavaScript in the settings, and refresh. List three pieces of information that disappeared and explain how their absence would impact an AI's ability to answer a question about that service.

Key takeaways

  • AI fetchers often have shorter execution timeouts for JavaScript than Googlebot.
  • Client-Side Rendering (CSR) creates a high risk of content being 'invisible' to AI models.
  • The 'No-JS' test is the fastest way to identify mission-critical discoverability gaps.
  • Server-Side Rendering (SSR) or Static Site Generation (SSG) is the gold standard for AI visibility.
  • AI bots rarely interact with the page (scrolling, clicking), so content must be 'visible on load'.
  • Shadow DOM components can act as a barrier to simple scrapers used in LLM training.
  • Semantic HTML (tables, lists, headings) is more reliable for AI extraction than JS-driven UI elements.
  • Infinite scroll is an 'AI-killer' because bots generally do not trigger scroll events.
  • Excessively large DOM trees can lead to parsers truncating your content midway through.
  • Hybrid architectures like Next.js satisfy both human UX and AI crawler requirements.

Lesson Quiz

Pass at 70%.

1. What is the primary reason AI fetchers might fail to see content on a React-based SPA?
2. How does 'Hydration' in frameworks like Next.js help AI Visibility?
3. Which of these features is most likely to be 'invisible' to an AI crawler?
4. What is the 'No-JS' litmus test used for?
5. Why is the Shadow DOM a potential pitfall for AI discoverability?
6. If a page uses infinite scroll to show product reviews, how will an AI bot likely react?
7. What is the 'Visibility Gold Standard' mentioned in the lesson?
8. Which HTTP status code might an AI bot receive if it fails to trigger a JS-driven redirect?
9. Why does DOM size (number of nodes) matter for AI Visibility?
10. What should a Practitioner suggest if a client refuses to move away from a CSR architecture?
Create a free account to save progress and earn a certificate.
Robots.txt for AI BotsBack to moduleSitemaps, Feeds and llms.txt