Quantization and the RAM bill

18 May 2026

Binary quantization is how a billion-vector collection stays resident. It is also how a bad shortlist happens if you never rescore.

Keep the full vector on disk. Search the quantized form. Rescore the top few dozen at full precision before you return them to the model.

The break-even is boring: if the working set fits in RAM uncompressed, do not quantize it to save a diagram.