You train the model with the loss applied at several prefix lengths at once: the first 64 dimensions, the first 128, and so on. This teaches it to pack the most important information at the start of the vector, like a set of nested matryoshka dolls.
Compression/loss, and the opposite - scaling up and oversmoothing - are fascinating in that any time even the tiniest innovation happens in those areas, all this other technology improves overnight, and a bunch of new technology becomes possible.
> You can push this further by combining quantization with truncation or PCA. The resulting vectors can be dramatically smaller while still preserving a surprising amount of retrieval quality.
Counterintuitively - quantisation can also be combined with a random rotation step before the quantisation. A random rotation spreads information across more dimensions, allowing more aggressive quantisation without losing accuracy. Ironically - almost the opposite of a PCA.
I do wonder if relevant here though. It relies on the embeddings having "structure", i.e. that principal components point along basis vectors, which may not be the case with text embeddings.
Nice! Thank you for doing this. I am not surprised PCA holds up. It comes with some pretty strong theoretical guarantees. The problem with it has always been scalability. The O(n^3) complexity makes it infeasible to use for massive datasets.
That is where something like Matryoshka embeddings has appeal. You trade a little bit of performance for a guarantee of training + validation set coverage.
Nice! I’ve been working on something similar and found similar results.
In my experiments, I used lots of embedding models and the results were not nearly as uniform as this curve, just FYI. I didn’t use any of the API-based models though
PCA is applied after the model, so there should be no difference in embedding throughput. Lookups in the index should be faster, but that speedup also applies equally to MRL.
Thanks for doing this benchmarking Dylan. I wanted to teach people PCA in my original article, but had no idea it would stack up this well against Matroyshka!
Feels like a “just use logistic regression” moment :)
This is fascinating if it works as well as the experiments make it seem. For example, how does it compare to classic image resize algorithms like Seam Carving or Inpainting: https://en.wikipedia.org/wiki/Seam_carving, https://en.wikipedia.org/wiki/Inpainting
(can they be compared?)
Compression/loss, and the opposite - scaling up and oversmoothing - are fascinating in that any time even the tiniest innovation happens in those areas, all this other technology improves overnight, and a bunch of new technology becomes possible.
> You can push this further by combining quantization with truncation or PCA. The resulting vectors can be dramatically smaller while still preserving a surprising amount of retrieval quality.
Counterintuitively - quantisation can also be combined with a random rotation step before the quantisation. A random rotation spreads information across more dimensions, allowing more aggressive quantisation without losing accuracy. Ironically - almost the opposite of a PCA.
I do wonder if relevant here though. It relies on the embeddings having "structure", i.e. that principal components point along basis vectors, which may not be the case with text embeddings.
Source: https://research.google/blog/turboquant-redefining-ai-effici...
That is where something like Matryoshka embeddings has appeal. You trade a little bit of performance for a guarantee of training + validation set coverage.
In my experiments, I used lots of embedding models and the results were not nearly as uniform as this curve, just FYI. I didn’t use any of the API-based models though
I also wrote about this exact comparison when using PCA and MRL to quantize static models, see: https://stephantul.github.io/blog/mrl-pca/
I couldn't find much when I first looked into this, which is why I ended up writing the article.
So I guess the answer is: no
Feels like a “just use logistic regression” moment :)