HighWalk Benchmark

Goal: Assess LLMs' ability to update a technical specification document from code changes.
Required capabilities: Strong code analysis, technical abstraction and writing skills, all within an existing document.
Scope: 46 commits, 95 modified files (+4,587 / -945 lines), Laravel project.

Why HighWalk

Over the past few months, I've worked on automating technical specification updates for projects in their run phase. The challenge is to produce documents for human teams (such as IT departments) with the right level of abstraction: covering architecture and business rules without getting lost in implementation details.

To achieve this, I designed and refined a dedicated Agent Skill. The main challenge lies in assessing the real impact of code changes, then writing and integrating them smoothly into an existing document.

I quickly found that models performing very well in coding benchmarks could struggle with this exercise. HighWalk was created to measure this specific ability to step back: a benchmark that evaluates broad understanding and documentation, beyond writing code or specifications for other agents.

Protocol, criteria, and ranking

Task assessed: Update an existing technical specification document from 46 commits in a Laravel project (95 modified files, +4,587 / -945 lines).

Environment: Runs performed with OpenCode and OpenRouter, with locked providers (the model publisher's provider, and NovitaAI for DeepSeek). Models that support multiple reasoning levels are evaluated at their default level and, when available, at a higher level.

Raw quality: Weighted functional evaluation across 11 criteria. It measures the ability to identify relevant changes, abstract them at the right level, and integrate them consistently into the document. Some criteria do not necessarily require an update: they may involve identifying a change as irrelevant. Omissions and unjustified additions are penalized.

Operational efficiency: Weighted equally between run duration and cost. 10-minute and $1.50 caps reflect practical usage constraints, exceeding both results in a zero score.

Overall score: 80/20 combination of raw quality and operational efficiency. Prioritizes reliable documentation while valuing configurations that are usable day to day.

Repetition: Each configuration (model, reasoning effort, and provider) was run twice. The best run is retained in order to compare the operational potential of the models.

Limitations: This benchmark covers one use case, one project, and one reference document. It is neither a comprehensive statistical measure nor a performance guarantee in other contexts.

Exclusion: Qwen 3.7 Max excluded because it critically altered the document.

Ranking 1: Best overall

Overall ranking. Weighting: 80% raw quality / 20% operational efficiency.

For the first time, two open-weight models lead the overall ranking: GLM 5.3 (high) takes a clear lead over Qwen 3.8 Max (medium). It combines the best raw-quality score with solid operational efficiency and misses no tasks, making it the first model to lead on all three criteria.
Grok 4.5 (high) takes third place, followed closely by Claude Opus 5 (high), whose lower operational efficiency is offset by its excellent raw-quality score.
GPT 5.6 Sol (low), which stands out from Sol's other reasoning-effort levels, and GPT 5.6 Terra (medium) follow.

Ranking 2: Raw quality

This ranking measures an agent's ability to rigorously meet the brief, regardless of time or cost.

GLM 5.3 (high) surpasses Claude Opus 5 (high) on raw quality, widening the gap over the rest of the models in this ranking.
Qwen 3.8 Max (medium) takes second place. It is followed by GPT 5.6 Sol (xhigh), Claude Sonnet 5 (high), and Grok 4.5 (high).
Claude Opus 5's high variant outperforms its xhigh variant. This is also seen across several models: the highest reasoning effort does not necessarily produce the best result.

Ranking 3: Operational efficiency

A normalized ranking based on run time and dollar cost (subject to the $1.50 and 10-minute caps). Considered without quality, it is not meaningful on its own but reveals two observations:

Kimi K3 has a relatively low operational-efficiency score, well below Kimi K2.6, GLM 5.3, and Qwen 3.8 Max (which all outperform it substantially on raw quality).
GPT 5.6 Sol (xhigh) is last on operational efficiency: it is by far the most expensive model and takes the longest. Its efficiency score is zero because it exceeds both the cost and duration caps.

Ranking 4: Completed tasks

This isolates models that did not completely fail any criteria. They are the strongest allies for human review: the agent has at least identified and outlined 100% of the elements.

GLM 5.3 (high), GPT 5.6 Terra (xhigh), Claude Fable 5 (high), and Claude Opus 5 (xhigh) stand out with 100% criteria coverage. That exhaustiveness can come at the expense of relevance, except for GLM 5.3 (high), which excels across the board.

Observations and takeaways

These measurements are based on API cost. With a subscription, some models may become economically more attractive. The final choice also depends on how the service is accessed.

GLM 5.3 (high) is the first model to combine the best overall score, the best raw-quality score, and 100% coverage: it is by far the best overall configuration for this task. It outperforms many models and leaves its open-weight competitors far behind. Its operational efficiency is also solid.
The only other open-weight model that comes close is Qwen 3.8 Max (medium).
GPT 5.6 Sol (xhigh) achieves a good raw-quality score, but its operational efficiency drops to 0 after exceeding the budget and time caps. Its low version is actually the stronger choice.
A higher reasoning effort does not guarantee a better result: Claude Opus 5 performs better in high than in xhigh, a trend also observed with Qwen 3.8 Max, Claude Sonnet 5, and others. Claude Opus 4.8 shows the opposite. Reasoning effort is not an option to maximize by default.
Finally, four configurations have no hard failures: GLM 5.3 (high), GPT 5.6 Terra (xhigh), Claude Fable 5 (high), and Claude Opus 5. They are the most reassuring starting points for human review.

Updated on August 23, 2026