Seven More Hypotheses, One That Survived
The first record in this series established the budget. The second recorded the five hypotheses on the compression of the vector tier and their falsification. The third documented the substrate that supported the work between the gates. This record covers the period from February to April 2026, during which seven further hypotheses were tested. Six were falsified. One was confirmed. The confirmed hypothesis is the load-bearing element of the system as shipped.
The five falsifications of the previous post in this series shared a common subject: the per-vector storage of the proxy tier could not be compressed below eight bits without breaking the greedy walk. The structural floor of the tier was therefore one kilobyte per vector at dimension 1024, and one gigabyte at one million vectors. The constraint envelope demanded a resident set in the low hundreds of megabytes. The reduction had to come from elsewhere.
Three directions emerged from this position. The first was that the access pattern of the walk might be made local, so that a small cache could hold the working set and the rest of the tier could remain on disk. The second was that the walk itself might be accelerated without touching the tier at all, by exploiting structural properties of the graph or of the hardware. The third was that the quantisation might be improved by a different mathematical construction, one not yet tested in the first round. Each direction produced hypotheses. Each hypothesis was tested against a pre-registered gate. The outcomes are recorded below.
Hypothesis 6: Graph reordering for cache locality
The previous record established that a least-recently-used cache, applied to the tier paged on disk under the insertion-order layout, returned a hit rate equal to the cache size ratio. The diagnosis was that the layout placed graph-adjacent nodes on unrelated pages. A natural amendment is to reorder the nodes before laying them out, so that adjacent nodes in the graph fall on adjacent pages.
The technique is established in the literature on sparse matrix reordering. Cuthill and McKee, in 1969 [1], proposed an algorithm that produces an ordering of the vertices of a graph in which the adjacency matrix has minimum bandwidth. The reverse Cuthill-McKee variant, introduced shortly after, is the standard tool for cache-aware layouts of sparse structures. The hypothesis was that the same technique, applied to the Vamana graph, would convert the random access pattern of the walk into a banded access pattern whose locality the cache could exploit.
The gate was specified as a trace-driven simulation. The graph was rebuilt with a reverse Cuthill-McKee ordering applied at the end of construction. The walk was re-instrumented to emit page accesses. The simulated LRU cache, at one per cent of the corpus size, was required to achieve a hit rate of at least 0.85. The measurement was conducted on a 10K corpus to keep the simulation tractable.
The measurement returned a hit rate of 0.21. The improvement over the insertion-order layout was real but insufficient. The diagnosis: graph reordering reduces bandwidth in the matrix sense, which corresponds to the maximum displacement between any two adjacent nodes in the linear order. The walk, however, does not traverse adjacent nodes uniformly. It traverses a small number of nodes per step, selected by the proxy distance, and these are distributed across the bandwidth window. The expected page accesses per walk step are not bounded by the bandwidth of the reordering; they are bounded by the in-degree of the graph, which is configured to 64. A cache of one per cent of the corpus, at the test scale, holds 100 pages, which is below the per-step expected access count of the walk. The technique had no headroom to deliver locality.
The hypothesis was archived. A secondary observation was recorded. At the 100K scale, with a cache fixed at one per cent of the tier and a graph of one million pages, the same simulation would produce a hit rate lower than 0.21, because the working set scales linearly with corpus while the cache fraction is constant. The technique therefore not only failed at the test scale but would have failed worse at the deployment scale. Graph reordering, as a route to cache locality on the Vamana walk under isotropic embeddings, was closed.
Hypothesis 7: Hot-cold node separation
A related approach starts from a different premise. Rather than reorder the entire graph, separate the nodes into two classes: those that are visited frequently across all queries, and those that are visited rarely. The hot class, smaller than the full tier, would be held in RAM. The cold class would be paged from disk. The technique is standard in database storage and assumes a power-law distribution of access frequencies.
The hypothesis was that a small fraction of the nodes, perhaps five to ten per cent, would absorb the majority of walk visits across a representative query workload. The system would then run with a resident tier of fifty to one hundred megabytes per million vectors, with disk paging for the long tail.
The gate was the measurement of the access distribution. A workload of one thousand queries was issued against a 100K-vector graph. Per-node visit counts were recorded. The distribution was plotted against a power-law fit and against the access frequency of a uniform random walk on the same graph.
The measurement returned a near-uniform distribution. The most-visited node received approximately three times the access count of the median node. The least-visited node received approximately one quarter. The Gini coefficient of the distribution was 0.18, which corresponds to a near-flat access pattern. The medoid, the fixed entry point of every walk, was visited by every query and accounted for the upper tail of the distribution; beyond the medoid, the distribution was flat.
The diagnosis is structural. The Vamana graph on isotropic embeddings has a near-uniform degree distribution after RobustPrune, and the walk descends through the graph in a small number of steps from any starting node. The expected visit count per node, over a uniform query distribution, is therefore approximately constant. A power-law distribution would require either a graph with heavy-tailed connectivity or a query distribution with heavy-tailed semantic locality. Neither obtains under the assumptions of the system. The hypothesis was archived.
Hypothesis 8: Software prefetch hints
The cost of a graph walk on the proxy tier is dominated, at scale, by the cache misses incurred when the next node visited is not resident in the L2 cache of the processor. The cache architecture of the Apple Silicon M1 includes a hardware prefetcher that detects sequential and strided access patterns and pre-loads the corresponding cache lines. The walk, however, is a pointer-chasing operation, and the access pattern is not detected by the hardware prefetcher.
The hypothesis was that software prefetch hints, inserted at the point where the walk fetches the neighbour list of the current node, would issue cache loads for the neighbours’ tier entries in parallel with the distance computation of the current step. The hint instruction on aarch64 is prfm, and the Rust intrinsic exposes it under the unstable core::intrinsics::prefetch_read_data feature.
The gate was specified as a 15 per cent reduction in walk latency at 100K corpus, with the same recall as the baseline. The implementation was straightforward: the walk loop was modified to issue prfm instructions on the neighbour identifiers immediately after fetching the neighbour list, before the distance loop.
The measurement returned no improvement. Walk latency varied within the noise floor of the bench harness, which is approximately 2 per cent at the configured number of repetitions. The recall was identical, as expected, since prefetch hints do not affect correctness.
The diagnosis required reading the M1 documentation more carefully than I had read it on the first pass. The M1 hardware prefetcher, unlike the prefetchers of x86 processors, is reported to identify access patterns through pointer-following sequences as well as through stride detection. The published behaviour of the Apple Silicon performance cores is that software prefetch hints have negligible effect on workloads where the hardware prefetcher is already engaged, because the hint either duplicates a load already in flight or is dispatched too late to mask the latency. The literature on Apple Silicon microarchitecture is sparse, but the empirical result is consistent across multiple workloads in the published benchmark suite of the project. The hypothesis was archived. Software prefetch, on this hardware, is not a tool I have access to.
Hypothesis 9: Adaptive entry point
The greedy walk on the Vamana graph begins at a fixed entry point, the medoid of the corpus. The medoid is the vector whose maximum distance to any other vector is minimised. It is the starting point, approximately equidistant from all queries. The cost of beginning at the medoid is the number of expansions required to traverse the graph from the medoid to the neighbourhood of the query.
The hypothesis was that an entry point selected as a function of the query, rather than fixed across all queries, would reduce the path length and therefore the walk latency. A small auxiliary index would map a coarse hash of the query to one of a small set of candidate entry points, each pre-computed to lie near a region of the embedding space. The walk would then begin from the candidate closest to the query rather than from the medoid.
The gate was specified as a 20 per cent reduction in walk latency, with recall@10 not decreased by more than 0.005 absolute, on a 100K corpus.
Two variants were implemented. The first selected the entry point through a coarse-grained k-means clustering of the corpus at k=8. The second selected the entry point through a learned projection of the query onto a small set of axes, computed offline. Both variants were tested on the mxbai-embed-large-v1 corpus and on the all-MiniLM-L6-v2 corpus.
The measurement returned no improvement on either variant on either dataset. The walk latency varied within the noise floor. The recall, in the case of the k-means variant, decreased by 0.011 absolute, which exceeded the recall budget of the gate. The diagnosis is that the medoid, in a graph constructed by Vamana with the long-range edges introduced by the pass, is already a near-optimal entry point. The long-range edges from the medoid reach any region of the graph in two to three steps, which is a small fraction of the total walk length. Replacing the medoid with an entry point closer to the query saves the cost of the first two steps but loses the structural advantage of the medoid’s connectivity, which exceeds the local connectivity of any non-medoid node.
The hypothesis was archived. The medoid remains the entry point of the shipped system.
Hypothesis 10: Beam early termination
The walk maintains a beam of candidate nodes during the search, sorted by proxy distance. The beam is bounded in size, typically to 100, and is updated as the walk expands new neighbours. The walk terminates when no node in the frontier improves the proxy distance below the worst node currently in the beam.
The hypothesis was that the walk could be terminated earlier, before the formal stopping condition, with a small loss of recall and a substantial gain in latency. The technique is a standard heuristic in graph search and is documented in approximate nearest neighbour systems as the early-termination criterion. The proposed criterion was: terminate the walk when the proxy distance of the best candidate in the beam has not improved by more than a threshold over the last steps.
The gate was specified as a 30 per cent reduction in walk latency, with recall@10 not decreased by more than 0.005 absolute, on both the mxbai and the MiniLM corpora.
The measurement returned a partial result. On the mxbai corpus, the technique achieved a 28 per cent latency reduction with a recall decrease of 0.003. On the MiniLM corpus, the technique achieved a 31 per cent latency reduction with a recall decrease of 0.012. The gate as specified required both corpora to pass. The MiniLM result failed the recall constraint.
This is the only hypothesis in the series that returned a non-binary outcome. The technique is correct in principle, the implementation is correct in execution, and the gate failed only on one of the two corpora used to evaluate the system. The decision was to record the technique as a falsification of the gate as specified, but to ship it as an optional configuration. The walk supports a --speed flag that enables the early-termination heuristic with the chosen and . The flag is off by default. The recall behaviour on a workload not yet tested is the responsibility of the operator who enables it. The default configuration of the system retains the formal stopping condition of the Vamana walk.
The partial outcome is recorded here because the alternative would have been to ship the technique as default and silently accept the recall loss on workloads that resemble the MiniLM distribution. The principle of the gate is that a falsification is a falsification regardless of how close the measurement was to passing. The handling of the partial outcome is a separate decision and is recorded as such.
Hypothesis 11: Page-aware graph construction at scale
A separate line of work, distinct from the earlier hypotheses on cache locality, asked whether the construction of the graph could be modified to produce a layout that the walk would access locally. The technique is described in the DiskANN paper [2] as a page-aware construction, in which the RobustPrune step at construction time considers, alongside the distance criterion, the page on which each candidate neighbour resides. The construction is biased to select neighbours on the same page when their distance is within a tolerance of the closest candidate. The expected effect is that walk steps stay within the page until they are forced to leave it.
The hypothesis was that page-aware construction on a 100K corpus would convert the linear-with-cache-size hit rate observed in Hypothesis 5 into a saturating hit rate that approached one for cache sizes above a small fraction of the tier.
The gate was the trace-driven simulation, identical to that of Hypothesis 5 but with the page-aware construction substituted for the insertion-order construction. The gate required a hit rate of at least 0.85 at a cache size of one per cent of the tier.
The measurement returned a hit rate of 0.34. The result is the highest observed across all locality-oriented hypotheses in this series. It does not pass the gate. The diagnosis is twofold. First, the page-aware construction is constrained by the distance criterion of RobustPrune, which dominates the page criterion when the distance tolerance is narrow. Widening the tolerance to favour the page criterion produces a graph with lower recall, because the neighbours selected are no longer the closest available. Second, even when the construction succeeds in placing graph-adjacent nodes on the same page, the walk visits only a small number of nodes per page before its trajectory leaves the page, because the local connectivity of a page is bounded by the degree of the graph multiplied by the typical fraction of neighbours on the same page.
The hypothesis was archived. Across hypotheses 5, 6, and 11, the cumulative observation is that the Vamana walk on isotropic embeddings does not admit a page-locality structure under any of the standard layout techniques. The proxy tier must therefore be held in RAM in its entirety, or its size must be reduced below the budget through a different route.
Hypothesis 12: Data-oblivious random rotation with optimal scalar quantisation
The hypothesis that survived was the twelfth. It was tested in early April 2026, three months after the start of the gate sequence. The construction it proposed was the technique introduced under the name TurboQuant by Zandieh, Daliri, Hadian, and Mirrokni in their paper accepted for ICLR 2026 [3].
The construction is as follows. Let be a vector to be quantised. Let be a random orthogonal matrix of dimension . Compute the rotated vector . The key theoretical property, which derives from the concentration of measure on the high-dimensional sphere, is that the coordinates of are approximately distributed according to a Beta distribution on , and the distinct coordinates are approximately independent. This property holds for any input vector , regardless of its provenance. The rotation is data-oblivious: no training, no calibration, no fitting against the corpus is required.
Given this property, the optimal quantisation of proceeds coordinate by coordinate. For each coordinate , a scalar quantiser at bits per coordinate is applied. The quantiser is a Lloyd-Max codebook designed for the standard Beta distribution, which can be precomputed once and shared across all coordinates and all datasets. The quantised representation thus requires bits per vector. The compression ratio against single-precision float is .
The distance between a query and a stored vector is computed in the rotated space. The query is rotated as and quantised against the same codebook. The distance estimate is obtained by reconstructing the dot product from the quantised codes, using a lookup of the centroid values from the Lloyd-Max table. The estimator is unbiased for inner product distortion in the formulation of Zandieh and colleagues [3], and achieves near-optimal distortion rate in the sense of Shannon’s lower bound.
The structural property that distinguishes TurboQuant from the RaBitQ construction of Hypothesis 2 is the following. RaBitQ uses one bit per coordinate, with the sign of the rotated coordinate as the code. TurboQuant uses bits per coordinate, with an optimal scalar quantiser for the Beta distribution. At , the construction achieves a compression ratio of 16 against single-precision float, or a factor of 4 against the 8-bit tier. At , the ratio is 8 against float, or 2 against the 8-bit tier. The compression ratio is selected at deployment time and trades off resident set size against recall.
The hypothesis as registered was that TurboQuant at would achieve recall@10 at least 0.95 after re-rank on the mxbai-embed-large-v1 corpus at 100K, with a beam of size 100, against the brute-force ground truth. The gate further required that the build time of the rotated and quantised tier remain within a factor of two of the 8-bit tier build time, and that the per-query walk latency remain within a factor of two of the 8-bit tier walk latency.
The measurement passed all three components of the gate. The recall after re-rank exceeded the 0.95 threshold by a margin that held across multiple runs. The build time was within the factor-of-two budget. The walk latency was within the factor-of-two budget. The compression of the tier was a factor of 4 against the 8-bit baseline, which reduces the per-vector storage from one kilobyte to 256 bytes. The full benchmark of the shipped configuration, at multiple corpus scales and on multiple embedding distributions, is recorded in the next post in this series.
The gate passed. The technique was promoted to the default configuration for the multi-tenant deployment, and remains an option alongside Product Quantisation for the single-tenant default. The mathematical property that made the difference between Hypothesis 12 and Hypothesis 2 is the rate of distortion as a function of the bit budget. RaBitQ at 1 bit per coordinate operates near the floor of admissible distortion for the greedy walk on the Vamana graph; the proxy is too coarse to direct the walk reliably, as the second post in this series recorded. TurboQuant at 2 bits per coordinate operates above the floor, with sufficient resolution to preserve the local ordering required by the walk, and below the resolution of the 8-bit tier, with substantial compression. The two-bit point is the operating point that the constraint envelope had been seeking across all the falsified hypotheses of the preceding three months.
The technique is data-oblivious. The same rotation matrix and the same Lloyd-Max codebook work on any embedding distribution. The construction adds no calibration step to the build pipeline, no per-dataset training, no maintenance burden in operation. These properties matter for a system that targets personal AI hardware where the dataset is whatever the user happens to have, and the operator is the user. A technique that required a calibration step against a representative sample would have failed a different gate, the gate of operational simplicity, which is not formally registered but which has been the implicit criterion of every architectural decision in this project.
Summary at the threshold
The constraint envelope of the first post in this series demanded a resident set in the low hundreds of megabytes per million vectors at dimension 1024. The substrate of the third post produced a resident set of 1.3 gigabytes. Eleven hypotheses were tested across the four months that followed. Six addressed the compression of the proxy tier directly. Three addressed the locality of access. Two addressed the walk itself. The twelfth, TurboQuant, addressed the compression of the proxy tier with a construction not yet available at the start of the gate sequence.
The shipped system uses TurboQuant at 2 bits per coordinate as one of two default proxy tiers, alongside Product Quantisation. The resident set at one million vectors falls within the envelope target, with measurements recorded in the next and final post in this series. The substrate of milestones zero through seven remains as documented. The eleven falsifications produced, in the negative, the architecture that is.
References
[1] Cuthill, Elizabeth; McKee, James. “Reducing the bandwidth of sparse symmetric matrices.” Proceedings of the 1969 24th National Conference of the Association for Computing Machinery (ACM ‘69), 1969, pp. 157–172.
[2] Subramanya, Suhas Jayaram; Devvrit; Kadekodi, Rohan; Krishaswamy, Ravishankar; Simhadri, Harsha Vardhan. “DiskANN: Fast Accurate Billion-point Nearest Neighbor Search on a Single Node.” Advances in Neural Information Processing Systems 32, NeurIPS 2019. Section 3 describes the page-aware construction.
[3] Zandieh, Amir; Daliri, Majid; Hadian, Majid; Mirrokni, Vahab. “TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate.” International Conference on Learning Representations (ICLR), 2026. arXiv
.19874.