SuperEMeans: Making e-means 3× faster
Not investment advice. Muhabbit scores how much a story may matter, not which way to trade.
CWI's database research group released SuperEMeans, a fusion of its SuperKMeans library with the recently published e-means clustering method, delivering clustering of high-dimensional vectors up to 3× faster than standard k-means with no loss in quality, according to the team's benchmarks. The open-source code is available on GitHub. In tests on 5 million 1,536-dimensional OpenAI embeddings and 10 million Cohere-embedded MS MARCO passages, SuperEMeans built IVF indexes with roughly 20,000 and 40,000 centroids respectively and ran up to 1.8× faster than e-means alone on a single 16-core Zen 5 machine. The work matters because clustering is the bottleneck users wait on before building large-scale vector indexes such as IVF and ScANN, and SuperKMeans has already been adopted by ParadeDB, Zilliz and FAISS. The authors note that SuperKMeans' dimension-pruning approach is orthogonal to e-means' two-epoch method, which is why combining them works. Next, watch whether vector database vendors fold SuperEMeans into production indexing pipelines and how the combined method holds up on larger and non-English embedding workloads.
Muhabbit's AI wrote this summary in its own words from Hacker News's report. It can get details wrong, so check the source.
Named in this story
SuperKMeans, SuperEMeans, e-means, CWI, FAISS, ParadeDB, Zilliz