Just-in-Time Retrieval

Two ways to feed an agent

An agent that needs information from a large corpus — a codebase, a document store, a wiki — can get it two ways:

Pre-inference retrieval: before the model runs, a retrieval system (typically embedding-based semantic search, which indexes documents as vectors and finds ones "similar" to the query) selects chunks and stuffs them into the prompt. This is classic RAG — retrieval-augmented generation.

Just-in-time (JIT) retrieval: the context holds only lightweight identifiers — file paths, links, queries that could be run — and the agent loads actual data at runtime through tools, as the need arises.

Pre-inference retrieval is fast and predictable, but it must guess relevance before the task unfolds — so it risks stale chunks, irrelevant inclusions, and missing the thing the agent turns out to need at step 7. JIT trades some speed for adaptivity: the agent fetches exactly what the task actually demands, informed by everything it has learned so far.

The human analogy that makes it click

You don't memorize your company's entire wiki. You remember that the deploy runbook lives in the infra space, that Slack #releases has the latest cut, that the API contract is in a file called openapi.yaml. You hold pointers, and you dereference them when needed.

JIT retrieval is agents doing the same — and the organization of the external store does real work. Metadata signals behavior before a single byte is read: a file named test_auth.py in a tests/ folder announces its purpose; a legacy/ directory whispers "prefer the non-legacy version." Naming conventions and folder hierarchy are context engineering, whether you meant them to be or not.

Progressive disclosure: each read informs the next

The deep advantage of JIT is that retrieval becomes sequential and self-correcting. Watch an agent answer "why does login fail after session timeout?":

1. grep "session timeout"        -> 3 files match
2. read auth/session.ts          -> sees refreshToken() called on expiry
3. grep "refreshToken"           -> finds auth/refresh.ts
4. read auth/refresh.ts          -> finds the bug: expired refresh
                                    tokens aren't cleared

Step 3 exists only because of what step 2 revealed. A pre-inference retriever had one shot to guess all four files from the initial question — before anyone knew refreshToken() was involved. And the window ends up holding only two relevant files, not ten chunks of maybe-relevant embedding matches: less rot, higher signal.

This is progressive disclosure — the same principle behind skills that load on demand and tool definitions discovered from a filesystem: the agent discovers context incrementally, keeping only relevant slices in the window.

The trade-off, honestly

JIT is not free:

• Latency. Every runtime fetch is a tool round-trip. Pre-computed retrieval delivers context in zero extra steps.
• Navigation risk. The agent might grep badly or read the wrong file — exploration quality depends on tool quality and corpus organization.
• Cost shape. Many small reads across many steps, versus one bulk stuffing upfront.

The practical sequencing rule from the Agent SDK guidance: start with agentic search (grep, glob, file reads — simple, transparent, no infrastructure), and add semantic search only when measured performance demands it — corpora too large or too unstructured to navigate, or latency budgets too tight for exploration.

Hybrid: the strategy most real systems land on

Front-load the small set of things always needed; let the agent explore for the rest. Claude Code is the canonical example:

CLAUDE.md earns its permanent slot because it's small, high-signal, and needed in essentially every task. The other 100,000 files of the repo stay on disk until a specific task summons a specific slice.

Designing a hybrid split comes down to one question per item: is this needed on virtually every task? Yes → load it upfront and keep it lean. No → leave it findable — well-named, well-organized, one grep away.

What this means for systems you build

1. Give agents navigation tools (search, list, read) rather than only bulk-dump tools.
2. Invest in the corpus's organization — naming and hierarchy are retrieval infrastructure.
3. Keep the always-loaded layer ruthlessly small; every permanent token is paid on every single call.
4. Add embeddings when measurement says so, not by default.

The unifying idea: context is fetched on the task's schedule, not pre-committed on a guess.