Jev Is Overhyped, but I’m Keeping It

I’ve had Jev running in one of my workflows for two days, so far my view is that Jev is overhyped but I am seeing value.

If you’ve read any of my past couple of blog posts then you’d know that I’ve been using Jev to look at job alignment to preferences. Jev gives me a typed integration. I define the possible outputs and it provides me a decision with confidence values, this is simple and nice. It’s cleaner than having an LLM integration although I have not seen evidence that it is more accurate than deterministic rules.

It does not outperform my existing judge

I wanted to give Jev a try because of all the hype I was seeing on socials, and decided to put it into an existing workflow which maybe wasn’t the best use case but it was a scenario it may work in.

A lot of the time when I’ve used LLMs to do a review of something, it is usually answering a set of questions and that is where a model designed specially for these decisions would be useful.

From my testing so far, Jev is cheap to run and results are consistent, although it has yet to beat an LLM as a judge (5.6-sol). The LLM has performed better in ambiguity, handling context and edge cases more reliably. Jev performs okay however that is not good enough when the result could be used to determine what happens next.

The numbers so far

We are still working with a very small dataset. So far across the past 2 days there have been only 17 new jobs. Continuing on the analysis I did in yesterday’s blog post, Jev has matched my assessment on 9 jobs, 52.9%.

From the 5 roles that were new today, only 1 of those were a strong fit in my view, the rest were poor alignments

When I ran these through the LLM judge, it matches my assessment on 82.4%, 14 out of 17.

Once again the dataset is too small to have any real assessment or benchmark, so the view from my side is that there still isn’t enough data and Jev may still have a place in my workflow.

The typed part still helps

The benefit right now is that Jev provided a judgement and the code owns the thresholds, escalation rules or side effects. This makes it useful as a first pass filter where high confidence scenarios can continue, then a second layer could be an LLM or human in the loop.

Right now I would not trust Jev to make important decisions alone. I am comfortable using it as one component in a layered system.

Why I’m leaving it in

If Jev were expensive, I would remove it now.

The results are not good enough to justify paying for it. I can leave it running for another week and collect more examples before I make a decision. I need more time and data to determine whether the problem is the model, the prompt, the question definitions or my thresholds. It leaves too many unanswered possibilities that I would like to explore before I decide to chuck in the towel.

The next test matters more than the launch claims

The launch positioned Jev as a new kind of model and made large comparisons with general purpose LLMs. My initial test has not aligned to those expectations.

For my use case, the question is smaller:

Can Jev become a cheap, reliable first-pass judge for one specific decision?

I do not need it to replace an LLM. I need it to handle the straightforward cases, escalate the uncertain ones and reduce the work sent to the more expensive model.

I’ll keep the current implementation running for another week or so, try fine-tuning and compare the results with the existing LLM judge. Right now Jev is cheap and not costing me anything, so I am seeing this as a little experiment and having fun with it, it just feels underwhelming compared to what I am seeing across socials.