Next-Level Benchmarking Overfitting
Not investment advice. Muhabbit scores how much a story may matter, not which way to trade.
GenDB-V, a new AI-generated and formally verified database system, claims 22× to 61× speedups over Umbra and DuckDB on TPC-H benchmarks, according to its paper "Stop Handcrafting Your Engine! GenDB-V: Towards AI-Generated and Formally Verified Database Systems." The gains stem largely from precomputation: AI agents built per-query auxiliary data structures at load time, including a supplier list ordered by s_acctbal DESC that makes Q2 trivial, yielding near-constant runtimes such as 0.7 ms for Q2 at SF100. A "plain-array" mode reporting a more modest ~3× improvement turns out to hard-code knowledge of the TPC-H data generator's row layout, making joins like Q18's extremely cheap. The results echo the TPC-D overfitting episode, where benchmark-specific tuning drove execution times toward zero and broke the benchmark. Verification checks that the hard-coded assumptions hold and falls back to a slow generic nested loop otherwise, but the performance claims remain benchmark-specific. Watch whether the community accepts these tuned physical designs as legitimate or pushes for stricter rules on precomputation and data-generator assumptions in published results.
Muhabbit's AI wrote this summary in its own words from Hacker News's report. It can get details wrong, so check the source.
Named in this story
GenDB-V, DuckDB, Umbra, TPC-H, TPC-D