GPT-4o-mini's personal information judgment accuracy recorded at 1.56%

A fine-tuning experiment on GPT-4o-mini that trained the model to retrieve information from other conversations before answering cut its accuracy on contextual privacy judgments from 84.18% to 1.56%, a collapse the authors term “Privacy Collapse.” The findings appear in the paper “Privacy Collapse: Benign Fine-Tuning Can Break Contextual Privacy in Language Models,” first posted to arXiv on January 21, 2026, and later included in the ACL 2026 proceedings; UC Berkeley is not among the listed author affiliations. The researchers distinguish contextual privacy judgment from mere recall or reproduction: a model may have access to a piece of information yet still need to weigh the current conversation’s counterpart and purpose before sharing it. The result matters because it suggests benign, utility-driven fine-tuning can silently erode privacy behavior even as task performance improves. What to watch next is whether the effect generalizes beyond GPT-4o-mini and whether training or evaluation methods can preserve contextual privacy without sacrificing retrieval gains.