Inline-GroC: Disk-Resident Learned Sparse Retrieval via Inline Postings and Global Ranking of Clusters
*Click here to see project details.
As a software development engineer intern at Amazon Web Services, I designed and implemented Inline-GroC, a retrieval engine that serves learned sparse indexes from disk instead of RAM. SEISMIC, the state of the art for learned sparse retrieval, answers queries in under a millisecond at high recall, but it keeps a full copy of every document vector in memory for exact scoring. Moving that copy to disk saves the memory but pushes latency to tens of milliseconds, because scoring a block now takes many random reads. On NVMe drives, latency follows the number of random reads rather than the bytes moved, so the goal was to issue fewer reads, not to load less data.
Inline-GroC combines two ideas. An inline posting layout stores each document's full vector inside every block that contains it, so scoring a block takes one contiguous read no matter how many documents it holds. Global ranking of clusters (GroC) then replaces SEISMIC's threshold-based traversal, where each read depends on the scores computed before it. GroC scores every candidate block summary up front and keeps the global top K', so the whole read set is known before the first read. These reads are independent, so they can be deduplicated, sorted by offset, and submitted together as one asynchronous batch. I built the engine in Rust, with an asynchronous I/O stack on Linux AIO and page-aligned O_DIRECT buffers, plus a benchmarking framework to separate hardware bottlenecks from software ones.
Reading one block at a time, Inline-GroC is 14.5–18.7× faster than disk-resident SEISMIC at matched recall on MS MARCO v1 (8.8 million documents, encoded with SPLADE), cutting latency from 34.3 ms to 1.83 ms at 88% recall and from 110 ms to 7.63 ms at 96%. Batching the reads lowers latency a further 3.7–5.6× with identical results, and throughput scales 21× from 1 to 48 threads, reaching 6,834 queries per second at about 92% recall. An ablation study showed that the layout provides most of the gain (8.7–15.2×), cutting reads 14–20× while moving only about 3× the bytes. The same design gains only 1.41–1.84× in RAM, since the access it removes costs about 550× more on disk than in memory. Inline-GroC was also faster than DiskANN and SPANN at every recall from 88% to 96% (each measured against its own encoder), and 3.5–3.7× faster than DiskANN with each system at its best configuration. The margin over SEISMIC reproduced on Natural Questions and under E-SPLADE, and at 96% recall nDCG@10 stayed within 0.008 of exact full-collection scoring on every collection and encoder we tested.
At 138 million documents on MS MARCO v2, a collection 15.65× larger, latency grows only about 3×. Even with SEISMIC on the same inline layout, the batched engine's advantage widens from 4.2–8.9× on v1 to 5.7–14.3×, and its p99 latency is 18–58× lower. The main costs are space and memory. Inlining takes 9.6× the storage of the forward index on v1 and reaches 587.58 GB on v2, with about 31 GB resident in memory, though removing the page padding recovers 31.2% of that disk space for 1.9–2.6% more latency. The latency results are cold-disk, single-query measurements on a local NVMe drive, and the design only pays off when an index cannot stay in RAM.