Story

On air. The free wire runs 15 minutes behind live. See pricing

World, crypto, finance and tech news, scored for impact.

TechHacker NewsFiled 2 days ago

Test/verification technique effectiveness with Astra

Open the sourceReported by Hacker News, danluu.com, 16 November 2026

Not investment advice. Muhabbit scores how much a story may matter, not which way to trade.

GPT-6 Astra, the successor to GPT-5.6 Sol, has improved enough on test and verification techniques that it saturates the Zstd RFC implementation benchmark, achieving full holdout test correctness over 99% of the time at medium effort for $1 per run, versus 5.6 Sol's 63% correctness at $3.50 on medium and 77% at $5.50 on xhigh. On the harder IMAP RFC task, which 5.6 Sol could not handle, Astra completes 80 runs per condition with Default testing again performing well and some techniques yielding modest correctness gains at higher cost. Astra also uses techniques more substantively: fuzzing now exercises real code paths and surfaces non-trivial bugs rather than generating redundant random inputs. The results matter because cheaper, more reliable verification could lower the cost of agent-built software, but the Zstd saturation means the benchmark no longer discriminates between techniques. Next, the team plans to run the larger Pandoc eval, where prior work showed models behave differently on bigger tasks and where C and C++ implementations were plagued by memory safety issues.

Muhabbit's AI wrote this summary in its own words from Hacker News's report. It can get details wrong, so check the source.

Named in this story

GPT-5.6 Sol, GPT-6 Astra, Zstd, IMAP