djev, the open source Jev, benchmarks close to it

In plain language

What djev is

In our evaluation of TypeSafe's Jev, we tested a model that returns a label and a probability instead of text. Jev is a hosted service: you send your code to TypeSafe and pay per call.

djev is an open source copy of that interface, and TypeSafe has nothing to do with it. Underneath is DiffusionGemma, an open weights model from Google that normally writes text. djev gives it a template with the answer left blank and reads how strongly the model leans toward each option.

We used djev-run, which packages djev to run on a rented cloud GPU. We wanted to know how close it gets to Jev, and whether it runs on a machine you already own.

What we asked

The same question we asked Jev, about the same comments. Each is a review comment from a merged pull request in an open source project, shown with the code it was left on. Does the comment describe a real defect, or a style preference, a documentation request or a feature idea?

djev got the same five questions Jev did. We did not rerun the other four models. Their answers come from the Jev evaluation, matched comment by comment, so every model here is scored on the same 334 comments.

What happened

Given a real defect and a harmless comment, djev ranked the defect higher 91% of the time. Jev managed 96% on the same comments, and the three frontier lab models between 93% and 97%. djev came last of the five.

When djev is confident, it is usually right. It said "over 99%" 28 times and was right 26 times. It also said "under 1%" for 16 of the 71 real defects.

Where it falls short

Its scores do not work as probabilities. Three quarters of its answers are below 0.01, and 5% fall between 0.2 and 0.8, against 35% for Jev. It can still rank comments well, but you cannot read its score as the chance that a comment is a defect.

It also has a hard size limit of 4,096 tokens per request, an eighth of Jev's. Six percent of our comments were too long, and they were disproportionately real bugs: 11 of the 82 defects never got an answer. Jev answered all of them.

Running it yourself

The model needs 14.5 GB of memory, and our Mac mini server has 24 GB. The first attempt crashed the machine. It ran after we closed nine applications and added a watchdog that stops the model before the system runs out of memory. Each answer took 3.4 seconds. Jev's hosted service takes a quarter of a second.

The numbers

Accuracy on the same comments

We measure accuracy with AUC, as in the Jev evaluation: given a real defect and a non-defect, how often the model ranks the defect higher. 1.0 means the defect is always ranked higher. 0.5 is what random guessing scores. AUC does not depend on where you set a threshold.

Jev and djev agree on whether a comment is a defect 90% of the time, and on its category 63% of the time. On the category question djev scores 63% and Jev 68%. djev is weakest on documentation requests: it got 18 of 48 right and labelled 16 of the rest as style.

Where the answers land

A probability is only useful if the model uses the whole range, so we grouped every answer by the probability the model gave.

The Jev evaluation found two frontier lab models that emit only 18 distinct values. djev has a different problem. It emits 208 distinct values here, but packs almost all of them into the last hundredth at each end, so a useful cutoff is closer to 0.001 than 0.5.

Whether a cutoff holds

The ranking underneath is intact, so a working cutoff still exists. We repeated the Jev evaluation's test: choose the cutoff that hits a precision target on half the comments, then check it on the other half.

What the default cutoff does

At a cutoff of 0.5, djev is the most conservative model we have tested. It flags few comments, and most of them are real: 86% precision, against 74% for Jev. It catches 69% of the defects, where Jev catches 89%.

In the Jev evaluation, the biggest source of false alarms was comments about architecture ("this works, but the layering is wrong"). At 0.5, djev flagged 3% of those as defects. Jev flagged 14%.

Running it yourself

What 4,096 tokens leaves out

djev-run caps each request at 4,096 tokens, including the answer template. A request over the cap returns an error instead of a guess. The refused requests are not a random sample.

The fix is the same as for Jev's larger limit: whatever selects the code the model sees decides the quality. With djev, that selection has to be eight times tighter.

What it took on a Mac mini

djev-run is built for an NVIDIA RTX PRO 6000 on Google Cloud Run and depends on NVIDIA's software, so it does not run on a Mac. We rewrote its request handling line for line on Apple's MLX framework, using a 4-bit build of the same Google model.

Memory is the constraint to plan around. The model files are 15.6 GB. We saved 1.1 GB by skipping the image encoder, which this task doesn't use, as djev-run also does. That was still too tight for 24 GB. On the first attempt the machine ran out of memory with no room to swap, and restarted. After that we closed nine applications, and the watchdog still stopped the model eight times, three of them mid-run.

A bug worth knowing about

The usual way to save memory on long requests is to feed them to the model in chunks. In the MLX library we used, that changes the answer. Past 1,024 tokens, chunks of 512, 1,000 and 2,048 tokens give three different answers, and none matches feeding the request whole.

We checked against a second, independent way of running the model, and feeding the request whole matched it exactly. Every number in this post uses that. If you run this model on MLX, test for it.

Method and limits

How it was run

djev got the same state and the same five questions as Jev, in djev-run's request format. We kept djev-run's prompt, answer template and fixed random seed, and replaced only the step that runs the model. Ground truth, the positive class and the scoring are the same as in the Jev evaluation: a comment counts as a defect if the neutral labeller said so, and comments about test quality are excluded because no model was asked about them.

To check the rebuild, we sent the example request from djev-run's documentation. It returned the same answer with the options in the same order, at 0.986 against the documented 0.942.

What this does not show

• We ran djev on a Mac, not in the cloud. We used an MLX 4-bit build of Google's model rather than the NVIDIA 4-bit build djev-run serves. They should agree closely, and the documentation example suggests they do, but we did not measure the difference on this task.
• We did not run the stress tests. We did not repeat the Jev evaluation's injection and context growth tests. The 4,096 token cap would have ended the context test almost at once.
• One model set the ground truth. It is the same labeller as in the Jev evaluation, and not one of the models under test. Differences smaller than about 3% are not worth reading.

What we would fix next

djev asks the model to answer each multiple choice question with a letter. In two of the four requests we checked by hand, the model's top choice was the category name instead of the letter, and djev ignores that. This may be why djev is worse at sorting comments into categories than at spotting defects. We have not measured how often it happens, and that is the next check.

We also have not run the same questions on djev-run's own GPU setup, which would show whether any of this comes from the Mac rather than the model.