← Back to articles

Local Gemma vs Grok 4.5 and Sonnet 5 on Practical Prompt Tests

5 min read

I already published one full comparison from the model-prompt-tests harness: Grok 4.5 vs Sonnet 5. That post asked whether a hosted frontier model held up on practical prompts. This one asks a different question: when a local candidate joins the same suite, does it look competitive on peer-judged score, and what does the latency table say about using it day to day?

On a 2026-07-21 batch that mixed local and OpenRouter models, local Gemma averaged 4.75 peer score across all 13 prompts. That sat slightly above GPT-5.5 (4.69) and Grok 4.5 (4.62), and ahead of Sonnet 5 (4.19). The catch is not subtle: Gemma’s average latency was about 66 seconds per prompt versus roughly 0.3 seconds for Grok and GPT-5.5 on OpenRouter in the same harness.

This is still one batch, one judge setup, and not a claim that local Gemma “beats” frontier models in production. It is a second franchise entry under the same scoring contract.

Where this sits in the series

Naming for these posts is documented in the repo at docs/benchmark-franchise.md.

Methodology

Harness: model-prompt-tests bench CLI. Prompt suite: the same 13 practical prompts used in the Grok vs Sonnet write-up (debugging, code review, architecture, planning, writing, data analysis, safety, constraints, ambiguity, rubric design, and related tasks).

Role Models in this batch
Local candidates local:gemma (13/13 completed), local:qwen3-coder (4/13 completed)
Hosted candidates openrouter:gpt-5.5, openrouter:grok-4.5, openrouter:sonnet (all 13/13)

Scores below are peer averages from the harness assessment for report 2026-07-21T03-30-20-648Z. Self-scores are tracked separately in the harness and are not the headline. Treat local vs OpenRouter latency as measurements for this run only; they still matter because the prompts and scoring path were shared.

I am not publishing a full GitHub artifact tree for this batch the way the first comparison did. The numbers here come from the local bench report and assessment files generated by bun run bench report / assessment on that day. If you re-run, keep the batch id and summary JSON next to the article draft.

Headline results

Model Completed prompts Avg peer score Avg latency ms Median latency ms Avg output tokens Quality/sec
local:gemma 13 4.75 65860 30380 1182 0.072
openrouter:gpt-5.5 13 4.69 325 289 1586 14.445
openrouter:grok-4.5 13 4.62 318 327 2433 14.521
openrouter:sonnet 13 4.19 2429 1782 1522 1.726
local:qwen3-coder 4 3.75 105109 57450 1368 0.036

Quality/sec is average peer score divided by average latency in seconds. Hosted OpenRouter models dominate that column because they finish quickly in this setup. Local Gemma’s score looks excellent; its quality/sec does not.

local:qwen3-coder only finished four of thirteen prompts in this export, so it is not a fair head-to-head average against the full suite. I leave it in the table as a partial local reference, not as a franchise headline.

Where local Gemma looked strong

On peer winners for individual prompts, Gemma took several coding and architecture-shaped tasks outright in this assessment, including:

  • architecture/database-choice (5.00)
  • coding-build/pomodoro-timer (5.00)
  • debugging/javascript-debounce (5.00)

That matches the story I want from local coding models: strong on structured code tasks when the model stays available offline. It does not mean Gemma was the winner on every prompt. GPT-5.5 still took a long list of planning, writing, safety, and analysis winners in the same assessment file.

Where hosted models still won the usable tradeoff

Grok and GPT-5.5 stayed in the high-4s on average score while returning answers in hundreds of milliseconds on OpenRouter. For interactive agent work, that is the practical difference. A 66-second average wait per prompt is fine for overnight batch evals. It is a poor default for “run this prompt while I wait.”

Sonnet’s average score was lower in this batch (4.19), consistent with the earlier Grok vs Sonnet write-up, but judge agreement on Sonnet rows was high in the assessment. Lower average score is not the same as “broken.”

Failure modes and noise

The assessment flagged large judge spread on some coding-build rows (including a 3-point spread on one Grok response). A few Sonnet judge calls still failed to return valid JSON after retries on hosted candidates. Those rows are the usual LLM-as-judge noise: keep the raw scores, exclude failed judge rows from averages, and do not overfit a narrative to one contested prompt.

Local qwen3-coder’s partial completion is its own failure mode for this franchise post: incomplete matrices should not be sold as full comparisons.

Practical takeaways

  1. Score alone is not the product decision. Local Gemma’s peer average was competitive with frontier OpenRouter models on this suite. Latency was not.
  2. Local models belong in the franchise. Offline candidates are exactly the kind of “honest lab” comparison I want next to hosted ones, as long as completion counts are honest.
  3. Quality/sec is useful inside a batch. It does not replace cost, privacy, or hardware constraints, but it stops a slow model from looking free when it is not free in wall-clock time.
  4. Keep methodology centralized. Details of how bench runs and judges live in the harness intro. Comparison posts should stay on results and caveats.

Limits of this review

  • One multi-model batch from 2026-07-21, not a repeated trial
  • Local model identities are harness ids (local:gemma, local:qwen3-coder), not full weights/version strings in this write-up
  • Peer judges were themselves LLM judges; they can disagree and fail
  • qwen3-coder incomplete (4/13)
  • No claim about current (post-July 2026) model versions or prices
  • Artifacts are harness-local for this batch rather than a published GitHub evidence directory

Next steps

  1. Re-run the full 13-prompt matrix with complete local candidates and record weights/endpoints in the run config.
  2. Publish the next hosted “new release” comparison under the same slug rules when a model drop warrants it.
  3. Keep every franchise post linked from /articles/series/practical-prompt-tests/.