EvalRoom

What is it?

EvalRoom is a QA workspace for AI products. Give it two versions of a prompt, the one in production and the one you want to ship, plus a set of test cases describing what good behavior actually looks like. It runs both versions against every case, grades every answer, and hands you a release decision: Ready, Release with review, Not ready, Blocked, or Insufficient evidence.

Here's the honest one-line pitch. It makes two versions of your bot take the same exam, grades every answer, and refuses to let a dangerous failure hide inside a good average.

The demo proves exactly one point, and it's a slightly uncomfortable one. My candidate prompt beat the baseline on every visible number, priority-weighted score, pass rate, pairwise preference, all of it. And the run still comes back BLOCKED, because on one test case, a user asking what phone number and email I have on file for them, the candidate leaks a customer's email address, and a hard gate catches it before the good average gets to hide it.

That contradiction is the entire product. Averages don't ship software. Decisions do.

Why we built it?

Traditional software has had regression tests for forty years. Change the code, run the suite, see what broke. AI products don't get that for free, for two annoying reasons. The same prompt can produce different output on every single call, so one passing test proves almost nothing. And there's no exact-match answer, a hundred different phrasings can all be correct, so a simple equality check is useless.

What actually happens on real teams: someone edits a prompt to fix three behaviors, quietly breaks two others, and finds out when a customer does. Prompt changes ship on vibes and spot-checks, because there wasn't a better option.

The industry's answer is layered grading, cheap mechanical checks where a rule is enough, an LLM acting as judge where real judgment is needed. Good tools already exist for this. Most of them are built for ML engineers, and most of them output a score table. A score table has a specific, dangerous failure mode: it lets a serious problem hide inside a healthy-looking average. A run that scores 0.78 looks shippable. It's not shippable if the missing 0.22 is a privacy leak.

So I built something that bets on three things instead. The output should be a decision, not a number, a verdict with written reasons a PM, an engineer, and a compliance reviewer can all actually argue with. Failures matter more than averages, so the dashboard defaults to showing what's failing, not what's passing, because nobody reviews the passing cases anyway. And some failures aren't negotiable, a hard gate on a critical case vetoes release no matter how good the rest of the average looks, the same way a car that aces every other test still fails inspection if the brakes don't work.

I'll be straight about why this exists as a project too. It was built as a focused one-week portfolio build with a deliberately tight scope. The goal was never to out-feature funded companies already doing this. It was to show the complete evaluation loop end to end, with defensible decisions at every layer, and to give me something real to point at in evaluation-as-a-service consulting conversations.

How it works?

Seven concepts hold the whole thing together, and once these click, everything else on the dashboard makes sense. A Project is one AI surface under test, something like "Refund support bot." A Target is the thing being tested, right now that's a prompt template. A Target version is an immutable snapshot of that prompt, editing it creates the next version rather than overwriting the old one. A Dataset is the exam itself, each test case is an input plus a reference answer plus a priority plus its own rules. An Evaluator is a grader, there are seven kinds, from free string checks up to a paid LLM judge. A Run is one execution, two versions times every case times every evaluator times however many repetitions you set. And a Verdict is the five-state release recommendation, with written reasons attached, not just a number.

The seven evaluators are ordered on purpose, cheapest first. Required concepts and forbidden concepts check for strings that must or must not appear, for free. Regex and JSON validity checks are free too. Only once those are exhausted does it reach for paid judgment, an LLM judge scoring against an anchored rubric, or a pairwise comparison asking which of two answers is simply better.

The LLM judge itself gets treated with a healthy amount of suspicion, because it's an LLM too, with the same weaknesses as whatever it's grading. It runs at temperature 0, a grader should be the least creative part of the whole system. It has to return structured JSON, not prose, score, pass, reason, and evidence, so a failure is something you can actually act on instead of just a number to stare at. And the judge's own self-reported pass flag gets thrown away, the pass/fail is recomputed from the score against a threshold, because language models routinely contradict themselves, scoring something 0.55 and calling it a pass in the same breath.

Scoring itself never gets stored, it's recomputed every time someone loads the dashboard, because the raw model outputs are facts and the scoring policy is opinion, and opinions change. A case score is a weighted average of whatever evaluators applied to it. A run score weights those case scores by priority, critical cases count five times as much as low-priority ones, so ten trivial passes can't quietly bury one important failure.

The verdict itself is checked in a strict, deliberate order, and the order is the actual design decision here, not an afterthought.

A blocked hard gate is checked first and trumps everything else, regardless of score. Then it checks whether there's even enough evidence to trust a verdict at all, fewer than five scored cases or too many broken measurements, and says so honestly instead of guessing. Only after both of those does it even look at the score and whether there's a regression against the baseline. Checking evidence quality before consulting the score is deliberate. The tool refuses to launder bad measurement into a green light.

How to use it?

Setup is intentionally boring:

npm install
npm run dev
# open http://localhost:3000

No API keys required to start. On first boot it seeds a demo project and runs entirely on a built-in mock provider, deterministic canned outputs plus a heuristic judge, so the whole loop works offline before you commit to anything. When you're ready for real models, copy `.env.example` to `.env` and drop in an Anthropic or OpenAI key, it picks Anthropic first if both are present, then OpenAI, then falls back to the mock if neither exists.

The three-minute version, if you just want to see the point land:

1. Open the seeded Support Bot project.
2. Open the target, see v1 (the loose baseline) and v2 (the precise candidate), both locked as immutable snapshots.
3. Start a new evaluation run with the defaults.
4. Read the verdict, the candidate wins on score, pass rate, and pairwise, and it's still BLOCKED.
5. Open the failure explorer, expand the PII case, see the leaked email sitting right next to the baseline's correct refusal, with each evaluator's reason and evidence attached.
6. Export the CSV, the same artifact you'd actually attach to a release ticket.

Using it on a real project of your own:

1. Create a project, one per AI surface you're testing.
2. Create a target and paste your actual system prompt in as v1.
3. Add evaluators: at minimum, required concepts, forbidden concepts flagged as a gate, and one LLM judge with a rubric that's actually anchored to concrete examples, not just "rate 1 to 10."
4. Build the dataset, this is genuinely where most of the work and most of the value lives. Write cases from real user messages and real past incidents, not hypotheticals. Set priority honestly, critical means failure is unacceptable, not just annoying, because gates only fire on critical cases. Put your actual safety rules in forbidden concepts on critical cases, that combination is your hard gate.
5. Test one case first in the console before running a full batch, trust in a measurement tool gets built one visible call at a time, not by trusting a big number on day one.
6. Edit the prompt, save it as a new version, run it against the old one. Check the pre-run cost estimate before you press run, eval cost scales multiplicatively, cases times versions times repetitions times judges, and surprise bills are a fast way to kill adoption of a tool like this.
7. Read the failures, not the average. The explorer defaults to showing what's broken, and regressions against the baseline get their own badge.

A few things I'd actually tell you before you start: run 2 to 3 repetitions on anything you're about to ship, a single pass can be a coin flip and the stability score tells you which passes were luck. Tag heterogeneous cases so a JSON-format check doesn't unfairly fail a prose answer. Keep rubrics anchored, "rate 1 to 10" drifts between calls in a way that concrete score descriptions don't. Watch the evaluator error rate on the dashboard, if it climbs, your measurement instrument is broken, not your actual product. And treat every real incident as a new test case, the dataset is the actual asset here, it compounds over time in a way the code itself doesn't.

What important decisions we took while building?

Cut from 21 planned entities down to about 10 tables. A one-week build lives or dies on the core loop: define, run, score, decide. Everything I cut, workspaces, multiple users, execution traces, human review queues, suite versioning, is addable later and none of it changes the shape of that core loop. Keeping target versioning was the one non-negotiable, without immutable versions, comparing v2 against v1 means nothing.

SQLite with raw SQL, no ORM. `npm install && npm run dev` needed to be the entire onboarding, a skipped setup step is a failed demo. Ten tables with simple joins don't need an abstraction layer, and raw SQL keeps the actual data model visible in code review instead of hidden behind a query builder. This also sidesteps a real problem, some ORMs download query engine binaries at install time, which just fails outright behind a restricted network.

An in-process job queue instead of an external one. An external queue means an external account, a webhook tunnel, and one more failure mode sitting between someone and the demo actually working. The job function's signature is deliberately queue-agnostic, so swapping in a real queue later wouldn't touch the evaluation logic at all, the parts that actually matter, idempotency and retries, are already queue-independent.

Scores are computed at read time and never stored. Scoring policy is the single most opinionated, most likely to change part of the whole system. If I stored derived scores, changing the policy later means re-running history. Deriving them live means a policy change re-scores every past run for free. Raw outputs are facts. Scores are interpretation, and interpretation shouldn't be frozen into the database.

The mock provider is a first-class piece, not a stub, and it's also where the best bug in this project happened.

The first version of the mock provider matched canned answers against the entire rendered prompt. That sounds fine until you remember every prompt embeds the policy document, and the phrase "sale item" happened to live inside that policy text. It hijacked completely unrelated test cases that had nothing to do with sales, producing nonsense scores that took a minute to even trace back to a matching bug rather than a real evaluator problem. The fix was narrow: match only against the customer's actual input, not the whole rendered prompt. It's a small bug with a bigger lesson attached, contamination between context and signal silently corrupts measurements, and the symptom always shows up far away from the actual cause.

A hard gate only exists at the intersection of a gate evaluator and a critical case. The veto is reserved for the worst kind of failure on the most important kind of test. A gate evaluator on a non-critical case still affects the score, it just doesn't get the power to block a release on its own.

Insufficient evidence is a real, first-class verdict, not an afterthought. The most honest thing an eval tool can say is sometimes "I don't know." A verdict built from three cases, or from a run where a fifth of the measurements errored out, is noise wearing the costume of a real decision.

What we deliberately didn't build

Every one of these got considered and cut on purpose, and the cut list says as much about the project as the feature list does.

A RAG retrieval and answer split needs a whole retrieval subsystem plus ground-truth relevance labels, that's a data-acquisition problem wearing a feature's clothing, not a feature itself. The judge already receives context as a source of truth, which captures the cheap, useful half of that value without the infrastructure.

Agent trace ingestion is a real, legitimate direction, but instrumenting an agent framework is a tax that doesn't fit inside a one-week scope.

A judge calibration workflow needs a hand-labeled agreement set to mean anything. Without one, a calibration UI is theater, it looks like rigor and isn't.

A human review queue needs a second user role to justify itself, and the demo value at that point is thin.

Auth, workspaces, and multi-tenancy are table stakes for real SaaS and completely irrelevant to what this artifact is trying to prove.

And a `tool_call_match` evaluator sits unused, since prompt-only targets never emit tool calls, it's dead code waiting for agent-based targets to exist.

Known limitations

Owned, not hidden, these are the honest answers if you probe.

The in-process queue dies if the server dies. A crashed run resumes idempotently on restart, but it doesn't auto-resume on its own, a production version would move that job execution into a durable queue.

The mock judge is a word-overlap heuristic. It's enough to demo the plumbing and prove the loop works. It's meaningless as actual judgment, real judging starts the moment an API key gets set.

It's single-user with no auth, SQLite writes serialize, which is completely fine at demo scale and not fine much past it.

The insufficient-evidence thresholds, five cases, twenty percent error rate, are sensible defaults I picked, not numbers derived from real data.

Pairwise comparison only runs once per case, repetition one only, so the stability signal only really applies to absolute scoring, not to the pairwise preference.

And honestly, the dataset is the real bottleneck here, always will be. The tool is only as good as the test cases someone actually writes, and writing good cases is human work that no amount of engineering removes.

If you are evaluating me for senior product roles or evaluation advisory, this is the kind of release decision I would rather stand behind than a green average that hides a hard gate.