Hardening the Article Embed Pipeline for Agent-Authored Content

The problem: agents write embed tokens as code

This blog uses the <pre><code class="language-x"> contract for code blocks, and tokens like {{ youtube:... }} / {{ mermaid:... }} in prose for embeds. Trouble starts when an agent or a Markdown generator (Quarto, Pandoc) writes a sample embed token inside a code block — say, a tutorial explaining mermaid syntax. The old pipeline scanned the whole HTML with one global token regex, so a token inside <code> got dragged into a real embed instead of staying as code text. Worse, a broken token payload threw during server-side rendering (SSR) and took the page down with a 500.

The holes we closed

1. Tokens inside <pre>/<code> must stay literal

The fix splits HTML into segments by code region first, then runs the token regex only on prose segments. A <pre>...</pre> or <code>...</code> block is pushed whole as literal HTML — tokens inside are no longer kidnapped into embeds. A tutorial showing {{ mermaid:... }} as an example now renders as code, not a diagram.

// before: TOKEN_RE swept the whole html
while ((m = TOKEN_RE.exec(html)) !== null) { ... }

// after: split by code region first
for (const segment of html.split(CODE_REGION_RE)) {
  if (segment.startsWith("<pre") || segment.startsWith("<code")) {
    blocks.push({ type: "html", html: segment });
    continue;
  }
  // tokens processed only in remaining prose
}

2. Broken payloads no longer 500

If a token cannot be decoded (bad base64, malformed JSON), it used to throw in SSR. Now it falls back to a [embed <kind> tidak valid] block — the page stays intact, only one block degrades. That is a general lesson: at a render boundary, validate external input and degrade gracefully instead of letting an exception reach an HTTP 500.

3. Quarto/Pandoc code blocks re-highlighted

Documentation generators often emit shapes like <pre class="sourceCode tsx"><code class="sourceCode tsx"> or <pre class="tsx"><code>. The new pipeline reads the language from the class attribute (not just language-x) via langFromAttrs, then strips Quarto's own highlight spans so the raw code is re-highlighted into our editor window with a copy button. It also recognizes its own Shiki output (data-title / class shiki) and leaves it verbatim — never processed twice.

function langFromAttrs(preAttrs: string, codeAttrs: string): string | null {
  const cls = (attrs: string) => /class="([^"]*)"/.exec(attrs)?.[1] ?? "";
  const codeClass = cls(codeAttrs);
  const preClass = cls(preAttrs);
  if (/data-title=/.test(preAttrs) || /(?:^|\s)shiki\b/.test(preClass)) {
    return null;
  }
  for (const c of [codeClass, preClass]) {
    const m = /(?:^|\s)language-([\w-]+)/.exec(c) ?? /(?:^|\s)sourceCode\s+([\w-]+)/.exec(c);
    if (m) return m[1];
  }
  const first = preClass.trim().split(/\s+/)[0];
  return first && first !== "sourceCode" ? first : null;
}

Why this matters for automated content

The more articles are written by agents, the more likely an agent needs to show the very syntax it is explaining. The code-stays-literal contract is not a small feature: it stops articles breaking when an agent demos an embed, and stops one bad payload from killing the whole page. The pattern aligns with the canonical code-block contract covered in Shiki code-block contract: short language ids, and with how YouTube/Mermaid/Recharts embeds render without a plugin.

Wrapping up

The embed pipeline now treats code blocks as sacred ground: anything inside is text, not a command. Embeds live only in prose, broken payloads degrade to a fallback, and legacy documentation generators still get an editor window. For the lower-level tokenizer race that once duplicated blocks across calls, see Shiki's stateful lastIndex regex race.