HighWalk AI Benchmark
- Which model, which settings, which harness for your tasks? I'll measure it.
- Pilot case: 80+ configurations, 2 months of runs, cost divided by 100 at equal quality.
Genesis of HighWalk
On a project that had moved into its run phase, I automated updates to the technical specifications with a dedicated Skill, tasked with understanding the real impact of dozens of commits and then integrating that impact at the right level of abstraction into a document read by humans.
Models that excel on coding benchmarks struggled with this exercise. I therefore built HighWalk, a reproducible testing and scoring protocol, which I ran for two months on this task.
Why HighWalk? Because the pilot case measured the models' ability to see the big picture, in overall understanding and documentation, beyond just writing code.
Key takeaways
- A higher reasoning effort does not mean a better result. On this task, the medium level was often the right setting.
- Public benchmarks do not predict your results. The ranking changes with the task, and it can change fast following new releases.
- Operational efficiency rules out configurations from the start. Some leading models are unusable in production because of their cost or processing time.
A benchmark on your tasks
I apply the same method to your AI workflows, to choose the right configuration for the task.
- Scoping: choice of a real, representative task, definition of weighted quality criteria and constraints (cost, time, confidentiality, hosting).
- Runs: tests of several models, effort levels, harnesses and skills, with a controlled protocol and repeated runs.
- Deliverable: a quality, cost and duration ranking, the recommended configuration, the projected operating cost, and recommendations to optimize the harness.
Indicative duration: 3 weeks. Can be re-run as new models and harnesses are released.
Benchmark details
Protocol, criteria, and ranking
Task: Update an existing specification document from 46 commits on a Laravel project (95 modified files, +4,587 / −945 lines).
Environment: Runs performed with OpenCode and OpenRouter, with locked providers (the model publisher's provider when it does not train on prompts, NovitaAI otherwise). Models that support reasoning levels were evaluated at multiple effort levels.
Raw quality: Weighted functional evaluation across 11 criteria. It measures the ability to identify relevant changes, abstract them at the right level, and integrate them consistently. Some criteria do not require an addition: they can involve identifying a change as irrelevant. Omissions and unjustified additions are penalized.
Operational efficiency: Duration and cost, weighted equally. The 10-minute and $1.50 caps reflect a practical usage constraint. Exceeding both results in a zero score.
Overall score: 80% raw quality, 20% efficiency. Each configuration was run twice, and the best run was kept.
July 28: first snapshot
At publication, proprietary frontier models hold the top of the ranking. Notable among them are Grok 4.5 (best overall), Claude Opus 5 (best on raw quality), and GPT 5.6 Terra. GPT 5.6 Sol achieves a strong raw-quality score, but its operational efficiency drops to zero. On the open-weight side, only GLM 5.2 and DeepSeek V4 Pro make it into the Top 10.
From this first set of runs, a higher reasoning effort does not seem to guarantee a better result on this task. Operational efficiency also proves fundamental, and it rules out economically unviable configurations from the start.
August 14: Qwen 3.8 Max and effort level
Four models enter the ranking, including Qwen 3.8 Max, which takes first place overall, the first open-weight model to reach that level. Claude Opus 5 remains ahead on raw quality and on 100% criteria coverage. Grok 4.6 stays behind Grok 4.5 on both quality and operational efficiency.
The medium effort starts to look like the right setting for many models. A new iteration of a model also does not necessarily mean a quality gain.
August 23: GLM 5.3, the open-weight shift
Qwen 3.8 Max led the open-weight models overall for only a few days; GLM 5.3 changes that. First overall, first on raw quality, and 100% criteria coverage. It's the first model, open-weight or not, to lead all three at once, at a single effort level.
The reversal of the balance of power between open-weight and proprietary models is starting to take shape. Coverage rate also becomes a differentiating factor.
September 22: a significant efficiency gain
Two months after the first run, open-weight models have taken the lead. Among proprietary models, only Grok 4.7 still holds a place at the top of the ranking. GLM 5.3 stays first, but MiMo V2.6 is the real surprise. Its Flash version reaches 90.91% raw quality and 100% coverage at $0.02, the lowest cost ever observed at that quality level.
The verdict is clear: the top of the ranking is now dominated by open-weight models. It took only two months for the trend to reverse. One model stands out for this task and lets us reach a 100× cost reduction at equal quality.
July 28 – September 22, 2026
- FR
- EN