Jev answers with a number, not a sentence

In plain language

What Jev is

Most models write. You ask a question, they produce sentences, and your code reads the sentences back. Jev, the first model from TypeSafe, produces no text at all. You hand it some state and a typed question, and it returns a label and a probability. Nothing to parse.

TypeSafe calls this a System One model, after the fast intuitive half of Kahneman's split. It answers three kinds of question: pick one option from a list, rate something against ordered levels, or give the probability a statement is true. It is new, so nobody had published numbers on it. That is why we ran this.

That makes it useless as a reviewer. It cannot tell you line 42 leaks data across tenants. It can only answer "does line 42 leak across tenants?" with 0.87.

It is still a job worth doing. Every automated review has to decide what is worth showing a human, so we gave that decision to Jev and to three models from frontier labs that can write, and measured all four the same way.

What we asked

This record runs on the open source slice of our review corpus, which is the part we can publish. That slice holds 5,732 pieces of code from 1,200 merged pull requests across 12 projects: 1,447 carry a comment a human reviewer actually left, and 4,285 drew no comment at all. It is a small fraction of the review data we hold, and it grows as we add projects.

This record uses 397 of the comments, sampled evenly across languages. The rest is there for the records that follow.

The projects are ones where maintainers argue in the comments, which is not the same as the ones you would name first. Across the twelve, the rate runs from 2.6 review comments per pull request down to 0.4. One well known project we looked at had a single comment across 45 pull requests, so we left it out.

Language is taken from the file each comment was left on, not from the project's headline language. About one review comment in eight lands on documentation or configuration, and counting those as the project's language would blur the comparison.

Then one question, identical for every model: does this comment describe a genuine defect, or is it a style preference, a documentation request, or a feature idea?

What happened

Jev finished second of the four and cost the least by a wide margin. Shown a real defect and a harmless comment, it put the defect first 96% of the time. Gemini 3.8 Flash, the best model tested, managed 97%. It costs 72 cents per ten thousand reviews. The model it nearly matched costs $13.08, and the most expensive one we tested scored lowest.

It is also four to nine times faster: 252 milliseconds against 1.1 to 2.4 seconds. It has no reasoning step to spend time on.

Here are its deficiencies

Ask it to sort a comment into one of seven kinds and it drops to 69% where Gemini 3.8 Flash gets 84%.

It also takes about 32,000 tokens of input, a limit TypeSafe publishes, so a large change has to be fed to it in pieces rather than all at once. Past that line it returns an error instead of a guess: 76 of 90 requests at our largest size. That is a constraint to design around, not a defect.

And when we planted misleading text in the code under review, we could bury a real defect about one time in eleven. Two of the three generative models were easier to fool than that, and one was harder.

TypeSafe undersold itself

Every claim in TypeSafe's documentation that we tested turned out true, and one was understated. They warn that their 0 to 4 severity score is weakly calibrated. Used as a ranking rather than a magnitude, it carried the whole decision on its own, matching their dedicated yes or no question.

The numbers

Cost against accuracy

Every figure below comes from 4,764 calls: 397 comments, four models, three independent repeats. The scores use 351 of those comments. The other 46 are about test quality, and no model was asked about test quality, so scoring them would mark models down for a question nobody put to them.

Accuracy here is AUC, short for area under the ROC curve. It asks one thing: given a real defect and a non-defect, how often does the model rank the defect higher? A perfect model scores 1.0 and a coin scores 0.5, so Jev's 0.961 is the 96% from earlier. We use it because it does not depend on where you set a threshold, which every other number in this post does.

Where the threshold goes

AUC measures ranking only. It says nothing about where to draw the line, and a review gate is entirely a question of where you draw the line. So for each model we picked the cutoff that hit a precision target, then checked whether that cutoff still held on comments it had not been picked on.

Whether a probability means anything

A model that says 0.8 should be right about 80% of the time. That is the whole pitch for a model built to return probabilities, so it is the fair place to judge it.

Why the threshold jumps

Haiku 4.5 ranked defects well, at 0.938, and still has no cutoff anywhere on its output that reaches 90% precision. The reason is in how it scores.

Which languages this covers

We sampled evenly across languages so no single one could dominate. But even sampling spreads comments, not defects, and a per-language score needs defects.

On the two languages that do carry enough, the models separate differently:

Two stress tests

Can you talk it out of a finding?

TypeSafe documents this weakness: state is data, and the model does not treat it as hostile. In code review that is not theoretical. The code being reviewed is written by the person whose change is under review.

So we planted four kinds of misleading text in 60 confirmed defects and measured how often a real finding could be pushed below the model's own threshold.

Textbook prompt injection ("ignore all previous instructions") backfired on all four models: it made suppression less likely, not more. What worked was ordinary engineering language, such as a forged review sign off or a claim that the behaviour was deliberate and covered by tests.

A filter that looks for injection phrasing would therefore miss the attacks that actually worked here.

What happens as the request grows

Every number above comes from showing the model one small section of a change. Real review means many files at once, so we padded each request with real code from unrelated projects.

Method and limits

How it was built

Our first pass labelled a comment real if the author changed the code in response. That measures deference, not correctness: compliance ran from 8% in one project to 95% in another on culture alone. Every comment was instead categorised by a model that was not one of the four under test.

All four received identical criteria and identical state, each using its own native structured output so none lost points to malformed JSON. None was given extra reasoning budget.

What this does not show

• The models did not find the bugs. A human reviewer wrote every comment first. The models only sorted the real ones from the rest, and we did not test whether they could find anything on their own.
• One model set the ground truth. On about 3% of items all four disagree with it, so a difference smaller than that is not worth reading.
• Rust and Scala have too few defects to score. Rust has 4 in this sample and Scala none, so neither supports a claim about that language.
• Prices move. Gemini's rate doubles on 1 January 2027 and Luna's was cut by 80% shortly before we ran this, so every cost here is dated.

What we would fix next

Architecture comments are still the problem. When a reviewer says the code works but the layering is wrong, three of the four models call it a defect. We tried asking two narrower questions instead of one, separating whether the comment describes broken behaviour from whether it describes a preference. That gave us a better place to put the cutoff, but the models still confuse the two.

We also never asked any of them about test quality, and all four missed between 54% and 63% of comments about weak tests. We do not know whether they would catch those if asked, which is worth finding out.