Memorilla: Latent Semantic Memory for LLMs

Researchers introduced Memorilla, a memory system that compresses an entire document collection into 16 query-specific "memory tokens" for frozen large language models, encoding documents once into reusable embeddings and learning to place relevant information directly in the model's input embedding space. Using only those 16 tokens, Memorilla outperformed baselines including Mem0 and MemoRAG on five of eight benchmarks covering personalization, long-document and knowledge-intensive question answering. On the FactKG benchmark it reached 84.33% accuracy versus 80.43% for full-context inference while using 108x fewer tokens to represent source documents. The work, posted to arXiv by lead author Vignesh Kothapalli, targets continual learning for frozen LLMs across long interactions. Next, watch whether the approach scales to larger document collections and production agent deployments beyond the eight benchmarks tested.