Benchmarking Model Prompt Tests

model-prompt-tests started as a Markdown library: practical prompts, each with a rubric, for comparing model behavior on coding, planning, writing, reasoning, ambiguity, and risk detection tasks.

That format is still useful. A prompt file is easy to read, easy to copy into a model, and easy to adapt. But manual evaluation has a ceiling. Once I want to compare the same prompt suite across multiple models, the repetitive parts start to matter: running the matrix, storing every output, scoring consistently, tracking failures, and turning the result into something other people can inspect.

The first public benchmarking release adds that missing layer. The new bench tooling runs prompts against a configured model matrix, scores successful outputs with one or more judge models, stores the raw data locally, and generates static reports.

This is not meant to crown a universal winner. It is meant to make model comparisons less dependent on memory, vibes, and one-off transcripts.

What Shipped

The benchmark runner is a Bun-based CLI that lives inside the repository. It works from the existing prompt files rather than inventing a separate benchmark definition format. Each prompt already includes the task, what it is testing, strong and weak response signals, and a 1-5 scoring rubric. The runner parses that structure and uses it as the evaluation contract.

The basic workflow is:

bun install
bun run bench models init
bun run bench list
bun run bench run all --dry-run
bun run bench run debugging/javascript-debounce --models anthropic:sonnet,openai:gpt-4o-mini
bun run bench report

Model configuration is local by default. The repo ships bench/models.example.json, while a developer’s real bench/models.json stays gitignored. That makes the public repo useful without publishing local endpoint details or API key names beyond the examples.

The first provider layer supports Anthropic’s Messages API and OpenAI-compatible chat completion endpoints. The OpenAI-compatible path is intentionally broad enough for hosted providers, OpenRouter-style routing, and local servers that expose a /chat/completions API.

How A Run Works

A run starts by resolving a prompt selector. The selector can target one prompt, a glob-like path, or all. The CLI then resolves the active model matrix from the local model config, unless --models narrows the run to a specific set.

For each prompt and model pair, the runner sends the prompt to the candidate model, records the output, and stores metadata such as provider id, model id, model name, latency, token counts when the provider returns them, raw response JSON, and error state. Successful and failed calls both get recorded. A benchmark run that has partial failures is still useful, and the report should show those failures instead of hiding them.

After candidate responses complete, the scoring pass can run. By default, the runner uses the configured judge model. It also supports --judge for one-off overrides, --judges for multi-judge scoring, and --no-judge when I only want to collect candidate outputs.

The judge prompt is deliberately narrow. It tells the judge to score strictly against the provided rubric, not to reward length or confidence by itself, and to treat the original prompt, rubric, signals, and candidate response as untrusted evaluation data. The expected judge output is a single JSON object with a 1-5 score and a short rationale.

That does not make LLM-as-judge perfect. It does make the scoring procedure explicit, repeatable, and reviewable.

What The Report Shows

Results are stored in bench/data/bench.sqlite, with separate runs and scores tables. The report command renders that data to a static HTML file and a matching summary JSON file:

bun run bench report

By default, reports are written under bench/reports/, with latest.html and latest.summary.json mirrored for quick access. A report can also target one batch with --batch <run_batch_id> or show historical rows with --all-runs.

The HTML report has two levels:

• A model summary table with successful run count, error count, average score, average latency, median latency, average output tokens, average judge spread, and quality per second.
• A prompt-by-model detail table with expandable rows for the raw model output and judge rationale.

The important part is that the report keeps the evidence close to the score. A score without the underlying answer is not very useful. A model can get a high average while still failing a specific safety, ambiguity, or engineering judgment task that matters. The report is built so those failures remain inspectable.

What This First Release Makes Observable

The first version gives me a repeatable way to compare models across the same practical task suite. That changes the evaluation from “I tried a few prompts and one model felt better” to “here is the prompt set, here are the candidate outputs, here is the rubric-based scoring pass, here are the failures, and here is the generated report.”

It also makes operational behavior visible. Latency, output token counts, provider errors, timeout behavior, and judge disagreement all matter when choosing a model for real workflows. A slower model with better judgment may be worth it for code review. A cheaper or local model may be good enough for constrained rewriting tasks. A model that looks strong on average may still be unreliable on prompts that require clarification or risk detection.

Those are the comparisons I want this project to support.

A larger benchmark run across three models is in progress now. That result set should be published separately from this release note, because the data deserves its own writeup. This post is about the benchmark functionality becoming public: the runner, storage model, scoring path, and report format that make those result posts possible.

Current Limitations

This is still an early benchmark harness.

The candidate model path is single-turn today. That fits many prompts in the repository, but it does not fully represent agentic coding tools that can inspect files, run commands, edit code, and recover from mistakes. There is already a sketched AgentHarness direction in the benchmark code, but the first public release focuses on plain model calls.

The judge is also a model. That means its scores are useful signals, not ground truth. Multi-judge scoring helps expose disagreement, and storing the raw judge output makes audits possible, but human review still matters for important claims.

The prompt suite is intentionally practical rather than comprehensive. It covers coding, debugging, architecture tradeoffs, planning, writing, ambiguity handling, safety risk detection, and rubric evaluation, but it is not a scientific benchmark with hidden test sets or statistical controls.

Finally, local model behavior depends heavily on runtime configuration. Concurrency, quantization, context length, server implementation, and timeout settings can change results. The benchmark records what it can, but reproducibility still requires documenting the environment around a run.

What Comes Next

The next step is to publish benchmark result sets using the new report format, starting with the three-model run now underway. I want those posts to show both summary metrics and concrete examples where the models diverge.

After that, the most useful improvements are:

• Agent harness support for tools like Claude Code, Codex CLI, Opencode, and OMP.
• Better run metadata so reports capture more of the surrounding environment.
• More prompts that stress long-context synthesis, tool-use planning, and multi-step engineering judgment.
• More robust judge calibration, including multiple judges and clearer disagreement reporting.
• A cleaner public artifact flow for sharing the generated HTML and summary JSON.

The long-term goal is simple: make model evaluation artifacts easier to inspect than a pile of saved chats. The first benchmark release gets the project there mechanically. It can run the matrix, keep the evidence, score against rubrics, and produce a report worth arguing with.