Serverless RAG on S3: a feasibility build

The bet

A RAG system stores 4 things: source documents, the chunks it retrieves, the embeddings it searches, and a queryable table for analysis. The usual stack puts each in a separate service — object storage, a vector database, sometimes OpenSearch, sometimes a warehouse. All of them run continuously whether or not anyone queries, and each adds cost and operational surface.

Two recent S3 features remove most of that stack. S3 Vectors (GA 2025-12-02) is a native vector index inside S3. S3 Annotations (GA 2026-06-16) is mutable, queryable metadata attached to objects. Add S3 Metadata's Iceberg table and the entire stateful layer of a RAG system lives in S3: no vector database, no VPC, no compute running at zero traffic.

This matters most for a fleet of low-traffic tools: dozens of internal RAG systems, one per team, each bursty and idle most of the day. On a vector database or managed search cluster, each one carries either a standing idle cost or a cold-start delay, plus its own operational surface. Near-zero idle cost and near-zero per-tool ops change whether a fleet like that is worth building at all.

So I built the system end to end on a real corpus, and fixed 4 pass/fail thresholds before running anything so the numbers couldn't be tuned after the fact. Both features are new enough that little has been written on running them at this scale, so this doubles as a report on what they're like to use.

The approach

Every piece of state lives in S3:

• Source text — plain S3 objects, one per source document
• Chunk text — S3 Annotations attached to those objects
• Embeddings — S3 Vectors, a native vector index inside S3
• A free analytics table — S3 Metadata's annotation table, Iceberg format, populated automatically from the annotations, no separate ETL pipeline

Lambda is the natural pairing for compute. It can live outside a VPC and side-step a NAT gateway's impact on idle cost. Orchestration is handled by a Step Functions Distributed Map, one iteration per source document. The embedder (bge-small-en-v1.5, int8 ONNX) runs in-process in the Lambda container. Claude handles generation.

The setup

The architecture is agnostic to the corpus, but I chose BEIR Natural Questions. 2.68 million passages, 108,593 Wikipedia pages, 3,452 test queries with relevance judgments. Anything that decomposes into documents and chunks slots in; the S3 surfaces, Lambda functions, and Step Functions orchestration are unchanged. BEIR NQ is just a well-benchmarked corpus at a realistic size.

A chunk here is one NQ passage, used exactly as the dataset ships it, with no custom splitter. Passages are the unit ground truth is computed over for both the brute-force answer key and BEIR's own relevance judgments. Keeping the retrieved chunk identical to that passage is what makes recall@10 a fair comparison. Re-chunking is outside the scope of this test, as it would turn the chunker into a variable.

Each passage is small: mean 473 bytes, up to 17 KB at the extreme, well within S3's 1 MiB per-annotation limit. A chunk lives in two places: its text as an annotation, and its 384-dimension embedding as a vector. This is exactly the path a query walks: embed the question, search the vectors, then hydrate the matching annotations for their text. The embedder truncates at 512 tokens, which only the longest passages reach.

Scoring recall@10 needs an exact answer key, which the architecture doesn't produce while serving queries. So every ingest invocation writes one extra artifact alongside its annotations: a Parquet shard of its own embeddings (passage_id, vector). It exists for building ground truth later, and plays no part in serving a query; the goal is just to avoid re-embedding 2.68 million passages.

The 4 thresholds, in full:

All 4 pass; the build and the numbers follow.

The build

The full build is on GitHub: jacquardlabs/serverless-rag.

The Distributed Map's work list is a manifest file in S3: one line per page, one Lambda invocation per line, 108,610 in total. That's slightly more than the 108,593 pages because a few oversized pages split across multiple objects.

Two of the findings below depend on how chunks cluster per page. Most pages are small (mean 24.7 chunks, median 15), but the tail is long: p99 is 134 chunks, and one page has 1,137.

Read-after-write on a managed vector index

I expected freshness lag to be the weakest number. Managed search indexes are usually eventually consistent, so a wait window between a write and its appearing in results seemed likely.

20 canary writes, each followed immediately by a poll for the exact key: every one was visible on the first check, no retry loop needed. The reported "lag" (p50 452.8ms, p95 597.0ms) is just PutVectors + GetVectors round-trip time. The test never found a window where the data existed but wasn't queryable yet. It ran at idle, though, so consistency under concurrent ingest load (the case that matters most for running both together) is untested here.

Per-object annotation writes throttle under load

The largest page has 1,137 chunks. That's more annotations than S3 allows on one object (the cap is 1,000), so the page splits across objects, and the worst-case object carries the full 1,000. Writing those 1,000 as parallel PutObjectAnnotation calls (pool size 32) took 73.8 seconds with 24 retries. That's per-object write throttling. It only appears under heavy concurrent writes to one key, so a preliminary test of 10 parallel writes across 3 objects never surfaced it. Worst-case invocation time was around 88 seconds against a 120-second timeout. That's only 27% margin on a clean single test, with concurrent load still ahead. Widening the timeout to 240 seconds was enough; tuning the write pattern with backoff or pool sizing wasn't needed to clear the threshold.

It recurred at scale: 5 of 108,610 ingest invocations timed out near 240 seconds, worse under sustained multi-hour concurrency than any single-invocation test had shown. All 5 succeeded on a low-concurrency retry. Annotations on a single object are still the right data model here, but writing many to one key under sustained load needs an explicit timeout budget that a small-scale test won't reveal.

This is also a cost driver. A Lambda is billed at full memory while it waits on throttled writes, which made this throttling the largest single contributor to ingest cost (see the numbers below).

A K-pop rhythm game slowed embedding down 13x

This one is a tokenizer detail rather than a service finding, but it applies to any embedder. That same 1,137-chunk page is "List of Pump It Up songs", a rhythm game that mixes short Korean song titles with long English passages. One batch from it measured 175ms/item, against the 13ms/item a smaller benchmark had clocked.

The tokenizer pads every sequence in a batch to the batch's longest token count, not its longest character count. Korean tokenizes much denser than its character length suggests: an 8-character Korean string came out to 17 tokens, more than a 71-character English sentence at 21. A single 238-token outlier forces every other item in its batch to pad out to 238 tokens, about 8× the batch's ~30-token average. The linear cost scales with that, attention's quadratic term adds more, and the measured slowdown landed at 13×.

The fix is to sort by token length and shrink the batch, so a rare long chunk lands in a small batch of its own instead of dragging a full one up to its length. One trap: sorting by character length first did nothing, because the batch was large enough to hold every chunk regardless of order. Shrinking the batch is what made the sort matter. Back to 12.2ms/item.

10 Wikipedia titles collapsed into one S3 key

Chunk text is stored as annotations on one S3 object per page, keyed by a slugified title: re.sub(r"[^a-z0-9]+", "-", title.lower()). 10 Wikipedia pages are named after single accented letters: "Ø", "Ü", "Æ", "Ñ", "ß", "Å", "Ö", "Ä", "É", "Ë". Each strips to an empty string. All 10 collapsed onto the key pages/.txt, and the last one written silently overwrote the other 9.

That lost 1,792 of 2,681,468 passages, 0.067% of the corpus. Small, but it's a real collision in a corpus of only 108,593 pages, and the count grows with scale. "One S3 object per page" holds up until two pages want the same object.

It doesn't bias the recall numbers. Ground truth is built from the same Parquet shards the live index reflects, so both sides of every recall comparison exclude the same missing passages. Threshold #2 measures retrieval quality over the passages that made it in.

The fix is a collision-resistant key: append a short SHA-256 of the full title, so the 10 empty-slug titles hash to 10 distinct keys and no two pages can share an object. It's in the code now, but these numbers come from the run before it landed. So the 0.067% is real here, and a fresh ingest wouldn't lose it.

The results

Recall@100 came in at 0.8944, just below recall@10 (0.9105). That ordering is expected once the metric is clear: recall@k here is overlap with exact brute-force search, the fraction of the true top-k nearest passages the index returns in its own top-k. The index reproduces the nearest handful more faithfully than the full hundred, so k=100 scores slightly below k=10.

nDCG@10 was 0.324, well below bge-small's published 0.502 on NQ. That gap is upstream of the S3 layer. Recall measures how faithfully the index reproduces this embedder's own exact search, not absolute retrieval quality. Re-running exact search locally splits the rest: adding bge's query prefix lifts nDCG from 0.348 to 0.413, the ANN approximation costs about two points, and the remainder is int8 quantization plus harness differences. Dropping the prefix (one embed path, no query special-casing) and int8 (to fit the embedder in a Lambda) are tradeoffs this run measures or bounds rather than assumes.

Retrieval total (embed + QueryVectors + hydration) is the gate metric, highlighted above, over 120 warm queries. It clears the 500ms threshold with headroom even at p99 (356.6ms). At p50 the split is embed 7.5ms, QueryVectors 68ms, hydration 100.5ms: hydration leads QueryVectors there, and embed is negligible next to both. Cold starts, when they happen, run 1.76–2.67 seconds.

At-rest cost: $0.35–0.55/month, storage only. Zero traffic means zero Lambda and zero Step Functions charges. The low end, $0.35/mo, is what's needed to keep serving: the vector index, annotations, page objects, and annotation table. The high end, $0.55/mo, also counts the Parquet embedding shards and ingest manifest, kept only as reproducibility artifacts that no query touches.

Building the index is a different number. The full corpus cost $60 in actual AWS charges (from the bill, not estimated): $31 Lambda, $19 S3, $5 Step Functions, plus tax. Lambda dominates because of the annotation throttling above: 108,610 invocations at 3GB each, many billed while idle-waiting on throttled writes. So there are two numbers: ~$60 once to build, ~$0.50/month to keep. Idle is the thesis, build is the one worth scrutinizing, and $60 is still cheap for indexing 2.68 million passages.

The obvious comparison is OpenSearch Serverless. The reason a project like this looks worth building is its old reputation: an always-on floor around $700/month regardless of traffic. I checked the current pricing instead of trusting that, and it's stale. Classic collections still have a mandatory floor: 2 OCUs (OpenSearch Compute Units) minimum, always billed, $350.40/month at $0.24/OCU-hour. But the newer NextGen collection type (GA May 28, 2026, a month before this build) scales indexing and search compute to zero after 10 minutes idle, with no minimum. Storage bills separately, about $0.14/month for a corpus this size.

What actually holds up is simpler: this architecture never pays for standing compute at any traffic level. NextGen's scale-to-zero has a 10-minute idle grace period; Lambda's is zero.

The speed story is the same. NextGen autoscales up to 20x faster than Classic: extra OCUs provision in seconds rather than minutes, and a scale-from-zero request queues about 10 seconds instead of dropping. So "OpenSearch scales slowly" is stale the same way the price story is. Both describe Classic, not what AWS ships now. The honest difference is narrower than pool-versus-no-pool. Lambda has an account-level concurrency limit too, and this build's ingest ran under one. But it's high and raisable, there's nothing to size per tool, and a cold tool's first request doesn't wait about 10 seconds for capacity to come back. That last part is the real edge at fleet scale.

S3 isn't limitless either. The per-object annotation throttling above is a limit this build hit directly, and the S3 services still throttle under sustained load like anything else at scale.

Before and after a negative control

An early smoke test, with only 100 curated pages ingested, asked a question deliberately outside that set: "who plays the joker in the dark knight." The system found nothing relevant and said so, instead of guessing.

The same question, after the full 2.68-million-passage corpus went in:

Based on the sources, the Joker in The Dark Knight (2008) was played by Heath Ledger, a late Australian actor [1][5]. His performance won him the Academy Award for Best Supporting Actor posthumously, as he passed away on January 22, 2008, before the film's release [10].

Same grounding instruction, same code, just a complete corpus instead of a hand-picked slice.

The verdict

This architecture fits one regime well, and it's a common one for internal platforms: a fleet of low-traffic tools. Dozens of them, one per team, each serving a few thousand queries a day against its own corpus and idle the rest of the time. There, per-tool idle cost and per-tool ops are what matter, and S3-native wins on both: near-zero at rest, and a new tool is a manifest plus a Lambda rather than another cluster to run.

OpenSearch Serverless has closed part of the gap (single-system numbers above). Its NextGen tier scales a collection's compute to zero when idle, so a fleet of quiet collections isn't a fleet of always-on bills the way Classic was. The remaining gap at fleet scale is operational. Capacity limits are account-wide, not per-collection, so a fleet shares one scaling budget and one busy tool draws down what the rest can scale into. Scale-from-zero also isn't free: an idle NextGen collection takes about 10 seconds to restore capacity on the first request, and bursty tools are idle most of the time, so that delay hits many real queries. Lambda shares the first limit (its concurrency is account-level too, and raisable) but not the second: an idle tool cold-starts in ~2–2.5 seconds instead of waiting ~10 for capacity, then serves warm at ~300ms.

It's the wrong tool at the other end. High, steady QPS pays for compute continuously anyway, so the idle-cost edge disappears, and you'd want OpenSearch's scaling compute and richer query surface (hybrid search, aggregations) over S3 Vectors' vector-plus-filter lookup. The p95 here is retrieval-only, too. Generation adds seconds, so anything latency-critical end to end is a separate question.

The backlog

Four things would land on your backlog past a prototype:

• Add bge's query prefix to the query path. The query Lambda embeds questions with no retrieval instruction; adding bge's prefix is a one-line change worth a measured +0.065 nDCG@10. Re-score ground truth with the same prefix so recall stays a clean comparison.
• Wire the analytics table up for Athena. The annotation table backfills automatically and reports ACTIVE, but querying it from Athena needs a one-time Glue Data Catalog integration this build skips. The Iceberg byproduct exists; making it queryable is the step to add.
• Move the annotation writes off the embedder Lambda. They're pure I/O, but they run inside the same 3GB embedder container, so a worker stays billed at full memory while it backs off on throttled writes. That's the single biggest line in the build cost. Decouple the memory-heavy embed from the I/O-heavy writes (a lighter write Lambda, or a queue drained by cheap consumers) so the wait bills on small compute; backoff tuning wouldn't move it. Batching a page's chunks into one annotation instead of up to 1,000 separate writes would also cut the per-object fan-out that triggers the throttling.
• Size the traffic cost, not just idle. The ~$0.50/month is at-rest only; a live deployment adds Lambda, S3 Vectors query, and generation cost at your query volume, and the index cost ~$60 to build once. The number to budget for is traffic; the idle figure shouldn't stand in for all three.

The takeaway

4 thresholds, fixed in advance, all 4 cleared on the full corpus, rough edges left in. For the workload it's built for, I'd now start from an all-S3 stateful layer by default and add a separate service only when a threshold forces it.