Then on June 12, Google Cloud published the Open Knowledge Format (OKF) v0.1, a spec that Google explicitly frames as a formalization of the LLM-wiki pattern. I read it, went back and reread Karpathy's gist, and added it to my growing backlog of things to properly learn.

But here's what hooked me, then and now — it was never the markdown. It's one property: a knowledge base that improves every time you use it. Every question the agent answers can leave the library better than it found it. That's the part worth an article, so this weekend I made an AI agent build me an encyclopedia of microphones to see it work.

Every question starts from zero

Look at how we use LLMs with documents today. You upload files; a RAG pipeline chunks and embeds them; each question retrieves the most similar fragments; the model synthesizes an answer. It works. But watch what happens to that synthesis: it's discarded.

Ask a subtle question that required piecing together five documents, and the model does the piecing. Ask it again next month or a teammate asks it tomorrow and the model does all of it again.

Nothing accumulates. The system's understanding at any moment equals its understanding on day one.

The waste isn't really the compute; it's the discarded comprehension. That's the problem Karpathy was pointing at, and it's the one every IT leader and VP surfaces when they ask: how do we make this agent actually get better over time? Right now, the honest answer is a shrug and a laborious manual process with no clear path to improvement.

A library that learns

Karpathy's answer flips the whole thing around. Instead of the agent re-deriving what it knows every single time a question comes in, it builds a wiki — a collection of plain markdown pages that it writes, owns, and keeps up to date. Each page covers one concept, pages link to each other, there's an index.md that acts as the catalog and a log.md that tracks every change, and every so often the agent runs a check to catch anything that's gone stale or drifted out of sync. When something new comes in, or a question reveals a gap, the agent goes and updates the relevant pages rather than just answering and forgetting. The synthesis doesn't get thrown away anymore it accumulates. Karpathy mentioned his own wiki had grown to around a hundred articles and roughly 400,000 words, and the agent could still navigate it cleanly because the index and summaries kept everything findable. What struck me most though is that he shipped no code at all, just the pattern, described clearly enough that any reasonably capable agent could pick it up and run with it.

Google's OKF defines the minimum conventions that make the pattern portable. The v0.1 spec fits on one page: a bundle is a directory of markdown files with YAML frontmatter, and exactly one field is required — type. Everything else is the producer's choice. Concepts reference each other with ordinary markdown links, which quietly turns the directory into a graph. Optional index.md files enable progressive disclosure; optional log.md files carry history. No SDK, no runtime, no registry. Google calls v0.1 "a starting point, not a finished standard," and shipped two proofs of concept alongside it — a BigQuery producer agent and a bundle visualizer. The framing is data-team-first (metrics, schemas, runbooks), but the format is general.

The build: a microphone encyclopedia

I wanted a domain where the knowledge is real, public, and scattered across websites that don't cooperate. Professional microphones fit perfectly: Shure documents the SM58 on shure.com, Sennheiser documents the e935 on sennheiser.com, and nobody publishes the market as one coherent, machine-readable source.

The build had two artifacts. First, a one-page bundle spec layering my conventions on top of OKF's minimum: a controlled type, a use_case enum, an allowed set of specs keys, a source block recording the exact page fetched and when, and a status lifecycle (draft → verified → stale). Remember, OKF only requires type — everything else here is producer convention, which is exactly what the spec intends.

Second, a producer agent task file for a coding agent with a search tool. Its first ground rule is the one that matters most:

Never write a fact from memory. Every spec value and claim must come from the page fetched in this run. If a value is not on the page, omit the key. An empty field is correct; a guessed field is corruption.

The agent's first job wasn't data it was deterministic tooling: a validator, a tag deriver (tags are a pure function of the other fields, never hand-written), a freshness checker, a graph builder. Only then did it fetch official product pages for twelve microphones across Shure, Sennheiser, Rode, Audio-Technica, and Electro-Voice. It did a genuinely good job, better than I expected. One unit:

---
id: shure-sm58
brand: shure
model: SM58
type: dynamic
use_case: [live-vocal]
tags: [dynamic, cardioid, xlr, live-vocal]
specs:
  polar_pattern: cardioid
  frequency_response_hz: [50, 15000]
  connector: xlr
  phantom_power: false
source:
  url: https://www.shure.com/en-US/shop/microphones/sm58
  last_checked: 2026-07-10
status: verified
---

The SM58 is the default handheld vocal microphone for live
sound — nearly indestructible, forgiving of loud stages, with
a presence lift that helps vocals cut through a mix...

How a question flows through it

There is no retrieval service here. The agent is the retriever, and it navigates rather than reads.

Ask the bundle: "I need a mic for a podcast studio that occasionally records sung vocals." The agent doesn't load twelve documents. It scans the index and frontmatter — a few dozen tokens per unit — and filters by structure: type: dynamic, use_case containing podcast. Three candidates survive. It opens only those bodies, and when one links to a rival ("often compared with the RE20"), it follows the link, reads one more file, and answers. Every fact in the answer traces to a specific file carrying its own source.url.

Cheap scan → targeted read → link traversal. The frontmatter exists so rejection costs thirty tokens instead of three thousand; the context window is the working set, and the bundle is the disk.

So far, though, this is just an efficient way to read. The big deal is what happens next.

The flywheel: every answer can improve the bundle

Here is the property that separates this from every knowledge system I've operated, and the reason I think the pattern had to come out.

Ask the bundle something it doesn't cover: "Does the SM7B need extra gain to drive it?" The frontmatter scan finds the SM7B, but the body says nothing about gain. A RAG system would stop at "not found." A search-equipped agent does better: it searches the web, lands on Shure's official FAQ, and answers the question. And then this is the moment it writes what it learned back into the bundle. A new paragraph in the SM7B unit, a gain note under specs, the FAQ URL appended to source, last_checked touched. On a branch, as a pull request.

The next person who asks gets the answer from the bundle in one hop, zero searches. If you want an engineer's mental model: it's a read-through cache with write-back except it never evicts, and every entry carries its receipt.

Two triggers feed the same pipeline. A question exposes a gap or surfaces new information; a schedule flags staleness the moment today − last_checked exceeds the manifest's refresh window. Either way the flow is identical: deterministic code decides what's due and what changed; the model rewrites only the affected narrative; the write lands as a branch → pull request → validator running as a CI gate → human review → merge, with log.md recording the change and the graph re-projecting itself from the new frontmatter. My standing rule for every agentic system I build applies with no exceptions here:

Deterministic code computes; the model narrates and judges — never the reverse.

And the anti-fabrication rule governs writes exactly as it governed the first build: the agent may only write back facts from pages it fetched this run, recorded as sources. Memory is not a source.

Notice what the questions themselves become: a curation signal. The bundle doesn't grow along axes some architect guessed at it grows along the axes people actually ask about. That's the "ever-improving" part, made concrete: the bundle you query on Friday is smarter than the one you queried on Monday, and git log can show you exactly how, one reviewed diff at a time. Curation becomes code review — blame, history, and rollback for free, no bespoke approval engine required.

RAG's synthesis is exhaust. The wiki's synthesis is inventory.

Where RAG still wins

None of this is a funeral for vector search, whatever the hype threads say. The two answer "what is relevant?" with different machinery:

Ask "warm-sounding mic for rap vocals on a home-studio budget" and vector search earns its complexity — fuzzy semantic matching over language no frontmatter field anticipated. Ask "which dynamic mics compete with the SM7B, and where did each fact come from?" and traversal wins, because the answer is structure plus provenance, which embeddings don't carry. Different physics, not a ranking. Mature systems will run both: a bundle for the curated core, retrieval for the long tail.

"But LLMs already read markdown"

True — agents read markdown all day: AGENTS.md, CLAUDE.md, .github/copilot-instructions.md, SKILL.md. Those files are behavioral configuration: they tell an agent how to act in a workspace. llms.txt is a pointer — a machine-friendly sitemap. An OKF bundle is neither; it's the knowledge itself, under conventions that make it portable: one concept per file, typed frontmatter, links forming a graph, an index, a log. Instruction files configure the reader; OKF standardizes the library. They don't compete — my producer agent was itself driven by a task file (config) whose whole job was to emit a bundle (knowledge).

Browse the bundle

Twelve concepts, five brands, every fact traceable to an official product page:

• Explore it: antonykanagu.github.io/demo-microphone-market-okf — interactive graph, brands as hubs, rival mics linked across brands

• Fork it: github.com/antonykanagu/demo-microphone-market-okf — spec, producer instructions, tooling, and the bundle itself

Clone it and cat a file. That's the entire onboarding.

MCP standardized the verbs. OKF wants the nouns.

When Anthropic released the Model Context Protocol in late 2024, it was a modest proposal for how agents call tools. Within a year it was the default answer to "how does my agent connect to X" — not because it was clever, but because it was neutral, minimal, and required no permission to adopt. OKF is running the same playbook one layer down: MCP standardized actions; OKF is a bid to standardize knowledge at rest. A bundle produced by one team's agent, improved by another vendor's agent, reviewed by a human in between — that interoperability is the part no proprietary catalog could deliver. v0.1 may not be the version that wins. The pattern underneath it very likely is.

What I'm taking away

The story is a knowledge base that participates in its own upkeep, getting measurably better with every question it's asked, under human review the whole way. After a decade of knowledge platforms that promised intelligence and delivered lock-in, the most interesting thing Google shipped here is a format that lets the intelligence accumulate somewhere you own.

Links: Karpathy's llm-wiki gist · Google Cloud OKF announcement · OKF v0.1 spec · llms.txt · Model Context Protocol

Every enterprise AI team knows this story. There's a solid use case, budget gets approved, the team spins up resources and starts building and then six weeks later a $40,000 cloud bill shows up. The POC scope has completely drifted from what was originally discussed and the business stakeholder who championed the whole thing has already moved on to the next priority. The experiment dies not because the idea was bad but because the environment around it was never set up for safe fast experimentation in the first place.

That is exactly the problem AWS Innovation Sandbox is built to solve.

What AWS Innovation Sandbox Gets Right

At its core this is an automated environment lifecycle management solution built on top of AWS Control Tower. What caught my attention as someone who spends a lot of time thinking about IT governance is that it goes after the real structural failure mode in enterprise innovation which is the absence of guardrails that are loose enough to let people actually experiment but tight enough to stop runaway costs and security drift before they become a serious problem.

Developers get real AWS environments, not some watered-down simulation, to test Bedrock models, Strands SDK agents, complex multi-service architectures, whatever the use case demands. And the account comes pre-configured with spending limits, automatic cleanup and audit trails so governance is baked in from day one rather than bolted on later.

For our Gen AI delivery team this changes the economics of experimentation in a pretty significant way. Instead of going through a lengthy governance process every single time someone wants to test a new Bedrock capability or validate an architecture we can provision a governed sandbox, let the team build, measure the outcomes and retire the environment all within a defined budget envelope.

Connecting the Dots: Operational Excellence Enables AI Innovation

The more I look at Innovation Sandbox the more I think it is less of a developer tool and more of a statement about how AWS thinks Operational Excellence should extend into AI experimentation. The principles are not new at all, operations as code, anticipate failure, small reversible changes, but applying them specifically to the POC lifecycle is where the real value is. That is the part of the engineering process that has always been a bit of a governance blind spot in most enterprises including ours.

At Shure where I'm responsible for building out our Gen AI delivery capability while maintaining rigorous cost and security governance, this kind of tooling is not optional. It's what allows us to say yes to business partners faster and deliver more AI POCs per quarter without accumulating technical debt or creating financial exposure.

What's Next

We are evaluating Innovation Sandbox as a core part of our AI governance intake process going forward. The goal is straightforward, every net-new AI POC at Shure gets a governed sandbox by default. Done right this eliminates the ad hoc environment sprawl and the surprise bills and the frustrating conversations where the answer to a business team is that we just cannot prioritize this right now.

The enterprises that win in AI over the next few years will not just be the ones with the boldest ideas. They will be the ones that built the ope

Sharing the learning here partly so the next person Googling "why is my scraper getting 403" finds something useful, and partly because I think the whole experience says something about how the web actually works today.

The thing I was wrong about for the longest time

My mental model going in was: pages are public, a request is a request, just fetch the HTML and parse it. That model is about a decade out of date.

The modern web is built to tell humans and machines apart, and it's genuinely good at it. The defenses stack. Your IP gets scored before you even send a useful request. Your TLS handshake gets fingerprinted. The browser you claim to be gets checked against how a real one actually behaves. And if you're still standing after all that, there's a "Press and Hold" button waiting that no amount of clever code gets you past. Beating one layer doesn't beat the others. I watched a perfectly reasonable automated request get turned away by sites that would happily serve the same page to me in Chrome a second later.

So I did what everyone does. I reached for tools.

The tools all have the same secret

I tried the AI browser agents. Blocked. I tried a few premium scraping service that got past some of the protected sites. It worked where everything else had failed, and once I understood why, the whole problem clicked into place. The hard part, getting past the bot wall, was happening on their infrastructure, through a pool of residential IP addresses: real consumer connections that anti-bot systems don't flag because to those systems the traffic looks like an actual person at home rather than a server in a data center.

That reframed the whole problem for me. The reason commercial scraping services work is not better code. It's that their traffic comes from IPs that look like real people in real homes, and yours comes from a data center. Everything else is detail.

What I'd actually tell someone starting this

A few things that are true regardless of which tool you pick.

Test empirically, early, from the environment you'll actually run in. A request that works from your laptop on the office network tells you almost nothing about how it'll behave from a data center IP at volume. Find that out on day one, not after you've built the pipeline.

Don't extract with brittle pattern-matching if you don't have to. I burned time writing rules to find the exact field I wanted in the HTML and they kept latching onto the wrong thing. The page was full of near-misses that matched my patterns but weren't what I was after. Converting the page to clean text and letting a language model read it was both simpler and more accurate. Let the machine that's good at reading do the reading.

The strategic one, the one that actually matters for anyone thinking about this for real work: a scraping approach that works today works at the goodwill of the sites you're hitting. They can turn on protection overnight, redesign a page, change a URL. When they do, your working solution silently starts producing gaps or nonsense and there's no one to call. For a weekend project, fine. For something a business is going to make decisions on, the honest answer is usually a licensed data feed or a paid service with a contract behind it. Not because you can't get the free version working, but because working and reliable are different words.

I spent a few days building a small multi-agent pipeline — Amazon Bedrock (Claude Sonnet) as the model, Strands for orchestration, MCP for the tool interface and used it to answer one simple question:

"For each item on a list, read a specific value off three different third-party websites."

Then I ran the same pipeline through two very different retrieval backends and compared them head to head:

• Backend A — a page-fetching scraping service (called via MCP): launches a headless browser, waits for JavaScript, and reads the target value from the rendered DOM.

• Backend B — Amazon Bedrock AgentCore Web Search (also via MCP): asks a search index for the URL and reads the target value out of the cached preview snippet — no page fetch.

Same LLM. Same prompts. Same 40-item catalog. Same three source categories (call them A, B, C). Same pinned site per category. The only variable was the tool at the end of the MCP call.

Here is what fell out of the data, and what I would carry into any project that has an LLM reaching for the web.

1. The most interesting number wasn't cost. It was silent coverage loss.

Headline numbers, over 29 items where both pipelines returned a result (scraping service vs Web Search (AWS Bedrock AgentCore)):

• Wall time per query: 64 s vs 46 s

• AI tokens per query: 330k vs 90k

• Sources returning a value (of 3): 2.72 vs 1.76

• Bedrock spend across the batch: ~$55 vs ~$9

AgentCore was 30% faster and used roughly 4× fewer tokens. On paper, a win.

But it returned values for 1.76 sources out of 3, versus 2.72 for the scraper. Nearly the entire gap was one source category: AgentCore returned zero values from category C across 29 items. The scraper got 26 of them.

Not because AgentCore is bad at that category. Because the site we pinned there renders its target value in a JavaScript element that never makes it into the search-index snippet. Snippet-based tools cannot see values that aren't in the snippet — full stop. It's not a bug you can prompt your way out of.

Takeaway. When you evaluate an LLM's "web tool", the interesting question isn't how fast it is. It's what fraction of the answers it silently fails to see. Silent nulls are far more expensive than slow completions, because your downstream consumer usually can't tell the difference between "no result" and "no such value".

2. "Faster" is misleading if you don't normalize by coverage

AgentCore ran in 46 s per query, the scraper in 64 s. Sounds like AgentCore wins.

Now divide by values actually returned:

• Scraper: 64 s ÷ 2.72 values = 23.6 s per value

• AgentCore: 46 s ÷ 1.76 values = 25.9 s per value

Once each tool is charged for the answers it actually delivered, the speed advantage evaporates.

Takeaway. Track latency per successful data point, not latency per query. The two numbers can point in opposite directions.

3. Snippet caches lag live pages

Both pipelines picked the same URL for many items. In roughly a third of those matches, the two pipelines returned different values from that same URL. The most striking example was a ~27% gap on a single page: the live-fetch backend read one number, the snippet backend read another, older one.

The scraper reads what the page displays right now. The snippet-based tool reads what the search index cached the last time its crawler visited. Updates to the page don't propagate into snippets until the crawler comes back.

Takeaway. If your use case cares about "the value a human sees on the page today," the tool that actually renders the page is not optional. Snippet-based retrieval is fine for stable text (descriptions, identifiers, static metadata); it's dangerous for anything that changes on a schedule shorter than the crawler's.

4. Prompt pinning is what let me draw any conclusions at all

My first run of this comparison was garbage. Both agents were free to pick "any relevant site" per category, and they picked different ones — so I couldn't tell whether a value gap came from the tool or from the site.

I locked the researcher-agent's prompt to a fixed site per category and re-ran. Every single successful query obeyed. Site choice stopped being a variable. What was left was purely the retrieval backend.

Takeaway. Every degree of freedom you leave in an agent prompt is a source of noise between runs. When you're benchmarking anything, pin down every choice that isn't the thing you're actually measuring. This is the "vary one thing at a time" principle from experiment design, applied to prompts.

5. A separate verifier agent kept the comparison honest

The pipeline is three agents wired together in a Strands GraphBuilder: Researcher → Extractor → Verifier.

The Verifier is the same Claude Sonnet, given the upstream agents' outputs plus a fixed checklist: does the returned value look well-formed, does the URL's domain match the pinned site for its category, is the value in a plausible range for this item, and so on. It emits a boolean verified flag and a list of issues.

I did not use the verifier to gate the final answer. I let it run on both pipelines' outputs and compared its issue counts. Result: it flagged 0.5 issues per run for the scraper and 2.6 for the snippet-based tool — a 5× gap, from the same judge, against the same rubric, blind to which pipeline it was grading.

Takeaway. LLM-as-judge isn't ground truth, but it's a cheap and remarkably useful second opinion — as long as the judge doesn't know which condition it's judging. The absolute score isn't what you trust; the ratio between conditions is.

6. The cost model isn't what most people put on the whiteboard

Teams tend to compare retrieval tools by their per-call price. That's the wrong axis by an order of magnitude. On this run:

• Retrieval-service fee per query: cents.

• Bedrock LLM token bill per query: dollars.

Full-page scrapes push tens of thousands of tokens of page HTML into the model's context window. Snippets push a few hundred. That's where the 4× token-cost gap really came from — not from the tool's own pricing.

The corollary is a little uncomfortable: the "cheap" retrieval backend is only cheap because it's feeding the model less to think about. Sometimes that trade is right. Sometimes what got trimmed was the part you needed.

Takeaway. Cost of an agent ≈ tool fee + LLM tokens the tool feeds the model. The second term dominates. Budget by "tokens my agent will consume," not by "cents per API call."

7. The best answer was a hybrid

Once I had the coverage-per-category numbers, the pragmatic answer wrote itself:

• Category A → snippet-based retrieval (28/29 coverage, 4× cheaper)

• Category B → snippet-based retrieval (23/29 coverage, 4× cheaper)

• Category C → scraping service (the only tool that returns anything)

Same agent, tool picked per category. Rough savings versus scrape-everything: ~40%, with no measurable coverage loss.

Takeaway. In agentic systems, tool choice isn't a single decision at design time. It's a function of the query. Route each sub-question to the tool that answers it best, and design your framework so that swap is cheap.

What I'd tell a team starting today

• Instrument for silent failures. Track "percent of asks that returned a real answer," not "percent of runs that didn't crash." Those numbers can be very far apart.

• Pin what you're not measuring. If your LLM has three degrees of freedom and you change one thing, the other two will confound the result.

• Read the LLM bill, not the tool bill. The tool bill is a rounding error next to the tokens the tool feeds the model.

• Add a verifier, and let it grade both variants blind. It's cheap truth-in-labelling for agent output.

• Assume the "cheap and fast" retrieval tool is quietly dropping data. Prove otherwise before you standardize on it.

• Design for hybrid from day one. Different sub-queries deserve different tools. Your framework should make that swap a config change, not a rewrite.

The stack — Strands + Bedrock + MCP — was the easy part. The harder part was noticing that a 30% speed win was actually a 35% coverage loss in disguise, and having the data to prove it. If you're wiring agentic retrieval into anything real, do the boring benchmark first. It will change what you ship.