The Method of Pre-Registered Gates: Five Hypotheses That Did Not Survive
The previous record in this series established the constraint envelope: a laptop with 16 gigabytes of memory, a 5 gigabyte language model resident, and a target footprint in the low hundreds of megabytes for the retrieval substrate. The constraint envelope is small. The design space it defines is not obvious. Locating a viable design within it is the subject of this record.
I worked alone. The time budget was eight months at moderate intensity, in parallel with other projects. The cost of a wrong direction was bounded only by my ability to detect it. A direction held for three weeks costs three weeks; a direction held for three months costs three months and forecloses three months of alternatives. Under these conditions, the determining variable is not the speed at which one builds, but the speed at which one decides to stop building. I formulated this as the operational principle for the duration: implement no architectural decision until a falsifiable gate has been passed.
The method is older than software. Popper formalised it for the empirical sciences in 1934 [1]. A theory is admissible to consideration only if it specifies, in advance, the conditions under which it would be rejected. Without such conditions, the theory cannot be tested; it can only be elaborated. Elaboration is not progress.
The adaptation to engineering is direct. Before adopting a structural decision, write down the prediction it makes. Define the measurement that would confirm or refute the prediction. Specify the magnitude of effect that constitutes confirmation, in advance. Run the measurement on a scale sufficient to detect the effect. Decide on the outcome. Implement only after the gate passes. If the gate fails, archive the direction with the measurement that closed it, and proceed to the next hypothesis.
The discipline is uncomfortable. Each hypothesis represents a candidate design that, if it survives, becomes the load-bearing element of the system. Each falsification removes a candidate from a design space already small. By the fifth or sixth failed hypothesis, the question whether the design space contains any viable candidate becomes legitimate. I will record here that the question presented itself more than once, and that no methodological remedy exists for it. The method does not protect against the possibility that no solution exists. It protects against the cost of failing to discover that no solution exists.
This record covers the first five hypotheses. They were tested in the November-to-January window, before the system had a vector index and after the storage layer was stable. The hypotheses share a common subject: how the resident set of the retrieval substrate is to be reduced below the structural floor of the dominant techniques in the literature.
The structural floor of the dominant techniques is approximately 1 gigabyte per million vectors at typical embedding dimensions. The graph index occupies roughly 250 megabytes per million nodes. The vector tier, stored as 8-bit quantised codes for distance computation, occupies 1 kilobyte per vector. At one million vectors of dimension 1024, the resident set is on the order of 1.25 gigabytes. The constraint envelope permits this in principle, but only if the language model and the operating system are absent. They are not absent. The envelope demands a reduction of at least a factor of three. Each of the five hypotheses below proposed a route to that reduction.
Hypothesis 1: Quantised prefilter during graph construction
The Vamana graph [2] is constructed in two passes. Each pass examines, for every node, the set of candidate neighbours and selects a bounded subset by distance. The distance computation is the dominant cost of construction. At one million vectors, construction requires on the order of distance evaluations.
The hypothesis was that construction time and memory could be reduced by using a quantised representation of the vectors during the selection step. A short integer representation, at 8 bits per coordinate, would reduce the per-vector memory by a factor of four relative to single-precision float, and would permit the distance computation to operate on integer arithmetic. The final graph would retain its original full-precision vectors on disk for serving.
The gate was specified as follows: on a 100K corpus of the mxbai-embed-large-v1 embedding [3], at dimension 1024, construction with a quantised prefilter would produce a graph whose serving recall@10 against a brute-force ground truth differed from the float-built graph by no more than 0.005 absolute. Construction memory would decrease by at least a factor of three. Construction wall-clock would decrease by at least 30 per cent.
The empirical outcome diverged from the prediction. The construction wall-clock decreased by 11 per cent, while construction memory increased by approximately 1 gigabyte because the integer representation was held in addition to the float representation rather than in substitution. The recall delta fell within the gate, but the structural failure was the memory increase. A technique that expands memory during construction, in a project whose declared concern is memory limitation, cannot be adopted on the basis of an 11 per cent speedup.
The hypothesis was archived. A secondary architectural finding emerged: the NEON intrinsic vdotq_s32, which would have permitted hardware-accelerated 8-bit integer dot products on Apple Silicon, was not yet stable in the Rust standard library at the time of testing. The instruction had been available unstably since 2023 under the stdarch_neon_dotprod feature gate, tracked by rust-lang/rust issue 117224 [4]. The Lance project encountered the same obstacle [5]. The instruction remained accessible only through unsafe nightly bindings or external assembly, which excluded it from a project targeting stable Rust. Integer quantisation as a construction-time technique therefore remained contingent on the maturation of the NEON dotprod path. Development of skeg proceeded with float-precision construction.
This second-order obstacle was sufficiently recurrent across the implementation work that, in May 2026, I submitted a stabilisation pull request to rust-lang/stdarch [6]. The change is attribute-only: the dotprod target feature and the corresponding is_aarch64_feature_detected!("dotprod") macro are already stable, so the intrinsics under the gate can be promoted without further implementation work. The proposal awaits Final Comment Period at the time of writing. The relevance to skeg is incidental, since the project has now committed to a different quantiser, recorded in the fourth post in this series. The relevance to the broader ecosystem of vector search on Apple Silicon is not incidental.
Hypothesis 2: 1-bit quantisation with random rotation
RaBitQ, introduced by Gao and Long [7], proposes that a vector can be represented by a single bit per dimension while preserving sufficient information for nearest neighbour search. The construction is as follows: let be a random orthogonal matrix of dimension . Define and . The code of is the bit string where if and otherwise. Distance computation between a query and a stored vector proceeds through the rotated query and an unbiased estimator that combines the Hamming distance between the codes with a stored normalisation factor.
The theoretical foundation relies on the concentration of measure on the high-dimensional sphere. The random rotation distributes the energy of the vector uniformly across coordinates, conditional on which the sign pattern carries asymptotically optimal information. The compression ratio against single-precision float is 32 to 1.
The hypothesis was that RaBitQ could substitute for the 8-bit integer tier as the proxy distance metric used during the greedy graph walk. The tier memory would decrease from 1 kilobyte per vector to 128 bytes per vector at dimension 1024. The walk would terminate with a candidate set that, after re-ranking against the original float vectors held on disk, would recover the recall of the 8-bit tier.
The gate was specified as recall@10 of at least 0.95 after re-rank on a top- candidate set of size 100, measured against brute-force ground truth on the same 100K mxbai corpus.
The walk failed to clear the gate. Recall on the top- candidate set, measured before re-rank, was 0.748. The re-rank stage cannot recover candidates that the walk did not visit; the Hamming proxy proved too coarse to direct the greedy traversal toward the true neighbourhood. The walk consistently terminated in a region of the graph displaced from the query.
The variance of the estimator explains the failure. The published RaBitQ results were obtained on flat scans, not on graph traversals. In a flat scan, the proxy is computed against every vector in the corpus and the top- is selected by sorting. In a graph traversal, the proxy is computed only against the candidates visited by the walk, and the walk decides which paths to exploit based on the proxy itself. The proxy thus has a dual role: it estimates distances, and it directs the search. A proxy that is statistically unbiased over the corpus can still produce systematically misdirected walks if its variance is large at the scale of the local neighbourhood. The variance of the 1-bit estimator is large by construction. The walk steps according to a proxy whose confidence interval, at the local scale, encompasses many nodes whose true distance ordering differs from the proxy ordering.
The structural memory floor of the vector tier is not 1 bit per coordinate. It cannot be reduced below the resolution required to direct a greedy walk on the local graph topology. The hypothesis was archived.
Hypothesis 3: 1-bit tier without rotation
A related hypothesis followed immediately. The 1-bit representation, used in the manner already implemented within the system as the QuantKind::Binary variant, does not require a random rotation matrix. The bit is simply the sign of the coordinate . The compression ratio is identical to RaBitQ. The implementation cost is zero, since the variant already existed for use in flat scan.
The hypothesis was that the binary variant, used as the proxy for the graph walk, would produce a candidate set sufficient for re-ranking.
The gate was identical to that of Hypothesis 2: recall@10 at least 0.95 after re-rank.
The measurement returned 0.748 on the top- candidate set before re-rank, indistinguishable from the RaBitQ result within experimental noise. The random rotation does not improve the proxy enough to alter the outcome. The architectural conclusion of Hypothesis 2 generalises to all 1-bit tiers, with or without rotation.
The breakdown is structural. The greedy walk on the Vamana graph performs, at each step, a comparison among the candidate neighbours of the current node. The decision is taken on the basis of relative ordering, not absolute distance. A proxy at 1 bit per coordinate produces an estimator whose noise floor exceeds the typical separation between adjacent nodes in the proxy distance. Relative ordering becomes unreliable, and the walk terminates in a neighbourhood whose true distance to the query is not the minimum reachable.
Hypothesis 4: 4-bit tier
The natural next step was an intermediate resolution. Four bits per coordinate, stored as packed nibbles, produces 64-byte codes at dimension 1024 if the encoding is direct, or 512-byte codes if a coarse representation is preserved alongside a scalar correction. This represents a factor of two compression against 8-bit, and a factor of eight against float.
The hypothesis was that 4 bits would suffice where 1 bit had failed, resolving the local ordering ambiguity that the binary proxy had introduced.
The gate remained identical to the previous two hypotheses.
The walk produced a top- recall of 0.689, which was unexpectedly lower than the 1-bit case. Profiling the 4-bit quantiser revealed the underlying flaw: a naive uniform binning of the coordinate range, without rotation, concentrates the quantisation cells in the dimensions of largest variance. For isotropic embeddings such as mxbai-embed-large-v1, the variance is approximately uniform across dimensions, but the per-coordinate distribution is heavy-tailed. Uniform binning under-resolves the body of the distribution while wasting cells on the tails. The local ordering becomes worse than at 1 bit, where the sign alone is preserved with zero quantisation error within each half-space.
The architectural conclusion was recorded with greater force: neither 1-bit nor 4-bit, in their straightforward forms, suffices for the role of walk proxy on the Vamana graph. The 8-bit tier, at 1024 bytes per vector at dimension 1024, represents the structural minimum. This was a significant negative finding. The factor of three reduction demanded by the constraint envelope could not be obtained by reducing the per-vector storage of the tier alone. The reduction had to come from elsewhere.
Hypothesis 5: The tier fits on disk
The fifth hypothesis addressed the consequence of the four preceding failures. If the per-vector storage cannot be reduced below 8 bits and the tier at one million vectors occupies 1 gigabyte, the only remaining route to a low resident set is to store the tier on disk and access it through paging.
The technique is documented in the DiskANN paper [2]. The graph and the vectors are laid out on persistent storage, and a bounded in-memory cache holds the pages most recently accessed. The cost of a graph walk becomes a sequence of page reads, each subject to the latency of the storage device. The compensating assumption is that the access pattern of the walk is sufficiently local; adjacent nodes in the graph tend to be accessed in sequence, retaining those pages in cache.
The hypothesis was that a least-recently-used (LRU) cache of bounded size, perhaps tens of megabytes, would achieve a hit rate above 85 per cent on realistic queries. The resident set of the tier would collapse to the cache size, independent of the corpus scale.
The gate was specified as a trace-driven simulation. The greedy walk on a 10K corpus was instrumented to emit, for every distance computation, the identifier of the node accessed. The nodes were mapped to 16-kilobyte pages using a node-to-page assignment fixed at build time, yielding approximately 63 nodes per page. The trace was replayed against a simulated LRU cache of varying size and the hit rate was measured. The gate required a hit rate of at least 85 per cent at a cache size corresponding to 1 per cent of the total tier size.
The empirical results tracked a linear relationship: a cache of 1 per cent of the corpus achieved a hit rate of approximately 1 per cent; a cache of 10 per cent achieved 10 per cent. The pages were, for all practical purposes, accessed uniformly at random.
The cause is layout mismatch. The Vamana graph on the mxbai embedding does not place adjacent nodes on adjacent pages because the construction does not optimise for any spatial layout. The graph follows the embedding topology, which on isotropic high-dimensional spheres has no exploitable locality. The greedy walk visits a node, then its neighbours, but the neighbours are scattered across the file by node identifier, which is assigned in insertion order. Pages thus contain unrelated nodes from the perspective of the walk.
A natural amendment suggested itself: reorder the nodes so that graph-adjacent nodes share pages. The amendment was tested as a separate hypothesis, recorded in the next post in this series, and was itself falsified. For the moment, the simple LRU-on-fixed-layout result was sufficient. The architectural conclusion of Hypothesis 5 was that paging the tier with a small cache does not reduce the working set. The tier resident set scales with the cache size, not with the cache hit rate, because the hit rate equals the cache size ratio.
The cumulative state after five falsifications was clear. The tier could not be compressed below 8 bits, and neither the tier nor the graph could be paged effectively using naive strategies. The resident set remained structurally linear in the corpus size at approximately 1.25 gigabytes per million vectors. The constraint envelope established in the opening record was, by these measurements, not satisfiable on the chosen architecture. The next post records the seven further hypotheses, their outcomes, and the single surviving direction that resolved the situation.
References
[1] Popper, Karl. Logik der Forschung. Julius Springer, Vienna, 1934. English edition: The Logic of Scientific Discovery. Hutchinson, London, 1959. Chapters I and IV.
[2] Subramanya, Suhas Jayaram; Devvrit; Kadekodi, Rohan; Krishaswamy, Ravishankar; Simhadri, Harsha Vardhan. “DiskANN: Fast Accurate Billion-point Nearest Neighbor Search on a Single Node.” Advances in Neural Information Processing Systems 32, NeurIPS 2019. Sections 2 and 3.
[3] Lee, Sean; Shakir, Aamir; Koenig, Darius; Lipp, Julius. “Open Source Strikes Bread - New Fluffy Embeddings Model.” Mixedbread.ai technical report, 2024.
[4] Rust language project, “Tracking Issue for NEON dot product intrinsics.” rust-lang/rust issue 117224. Opened 26 October 2023. https://github.com/rust-lang/rust/issues/117224
[5] LanceDB project, “NEON dotprod intrinsic stabilisation tracking.” lancedb/lance issue 6551. 2024.
[6] Scaratti, Daniele. “Stabilize the stdarch_neon_dotprod feature.” rust-lang/stdarch pull request 2120. 18 May 2026. https://github.com/rust-lang/stdarch/pull/2120
[7] Gao, Jianyang; Long, Cheng. “RaBitQ: Quantizing High-Dimensional Vectors with a Theoretical Error Bound for Approximate Nearest Neighbor Search.” Proceedings of the ACM on Management of Data, vol. 2, no. 3, article 167, June 2024. Sections 3 and 4.