What Is JevBench? A Clear Guide to AI Decision Model Scores

JevBench is a way to test AI systems that make small, structured decisions. It is not a model you can chat with. Imagine an inbox with hundreds of support messages. For each message, you ask, “Should this go to Billing, Delivery, or Other?” A useful system should pick the right answer, show a sensible level of confidence, reply quickly, and be affordable to run. JevBench measures those qualities on a shared set of cases so different systems can be compared.

The leaderboard is helpful, but its first row is not a shopping list. A model can rise because it is fast or cheap even when another model answers more questions correctly. JevBench also has several versions, and the same model may look different when new questions are added. This guide explains how the benchmark works, what its current results say about Jev and other decision models, and how to test a question that matters to you in the Jev AI homepage Playground.

What does JevBench actually test?

The JevBench project focuses on typed decisions. A task gives the system some state, usually text or a structured record, then a question with a bounded set of possible answers. The system returns a Choice, a Score, or a yes-or-no answer called Noul, along with probabilities when it supports them. That is much narrower than asking a general chatbot to write an essay. It is close to what software needs when it must route a ticket, apply a rule, or decide whether a statement is supported by a document.

For example, the state might say that a customer received a damaged chair and wants to replace it. The question could be “What is the main issue?” with Damage, Refund, and Other as choices. The expected answer is recorded before the model runs. JevBench checks whether the returned choice matches it. If the model reports 95% confidence but regularly misses similar cases, that is another problem the benchmark can reveal. A quick answer that cannot be trusted may be worse than a slightly slower one that flags uncertainty honestly.

The public task set covers four levels of difficulty. Easy items include clear intent or an explicit yes-or-no fact. Standard items ask about policies, routing, extraction, and ordered scores. Judge items test whether an answer meets a stated requirement. Hard items include more ambiguous, multi-step, or adversarial decisions. These groups stop a model from looking impressive simply because it handles straightforward labels. The published method lists 72 easy, 96 standard, 146 judge, and 220 hard public decisions in the current version.

How does the JevBench score work?

JevBench reports four axes. Intelligence asks whether the model selected the right answers, adjusted for how often a random choice could be right. Calibration asks whether the probabilities fit the model's actual reliability. Speed uses response times. Cost estimates or measures the expense of processing decisions. Each axis is shown on a 0–100 scale. The official JevBench Score combines them with an equal-weight harmonic mean, so one weak axis can pull the overall result down.

There is one detail worth remembering: the JevBench Score is not an accuracy percentage. A score of 64 does not mean 64% of questions were answered correctly. It is a combined rating built from four differently scaled measures and additional rules for weak results. If you mainly care about correct policy judgments, read the accuracy and intelligence columns before the overall rank. If you run millions of low-risk routing decisions, latency and cost may matter more. Neither choice is wrong; the rank just cannot make it for you.

Calibration is easy to overlook. Suppose a model says “Delivery” with 80% confidence on 100 comparable cases. If roughly 80 are truly delivery issues, that confidence is useful. If only 50 are, your app may act too boldly. JevBench measures this behavior on its test questions, including cases with known answer distributions. It does not promise that the same probabilities will be calibrated on your company's messages. Category names, language, and writing style can all change the picture.

Why are there public and sealed questions?

The September 24, 2026 JevBench v1.4.2 release reports results from 534 public decisions and 308 sealed decisions. Public examples let people inspect the task design and reproduce part of the evaluation. Sealed examples have hidden text and answers; the board publishes only aggregate results for them. This makes it harder to build a model that simply memorizes or tunes itself to the visible questions.

The gap between public and sealed results deserves attention. A model that handles known question styles well may struggle when it sees new cases. JevBench uses part of the sealed result in its intelligence measure and reduces the score when the public-to-sealed gap becomes large. The sealed set is deliberately difficult: on this release, even leading decision models have much lower sealed accuracy than public accuracy. That is a warning to test new workflows carefully, not evidence that every model is useless.

Version numbers matter here. Older JevBench articles may describe a three-part score, a different weighting, or a different top model. Those can be accurate for their release and still disagree with the current board. Check the version, date, task count, exact model checkpoint, and scoring method before comparing two screenshots. The project's public code and task definitions also make JevBench more useful than a ranking with no explanation, though private examples cannot be fully inspected by readers.

What do the current JevBench rankings say about Jev?

On the v1.4.2 official composite snapshot, Decider-4B v2 ranks first at 64.1, Jev 1.13.0 second at 63.3, and JevK5 v0.2.0 third at 62.0. That is a close result, not a large separation. The live JevBench board also offers different views, including answer quality. Always note which view you are reading: its headline capability view and its official four-axis composite answer different questions.

The figures illustrate why the JevBench ranking needs context. Jev scored 53.1 on intelligence and 76.3 on calibration; Decider-4B v2 scored 49.4 and 75.0. Decider-4B led on speed and cost in the benchmark's setup. On the 534 public items, Jev answered 86.6% correctly against Decider-4B's 83.5%; on the 308 sealed items, the figures were 36.7% and 34.7%. If answer quality is the main concern, those columns say more than the overall order. Our Jev vs Decider-4B guide looks at that pair in greater detail.

Prices deserve the same care. Some JevBench costs come from a published service rate; others are labeled estimates based on a comparable hosted model's input-token price. The latter is not the actual electricity, rental, or staffing cost of a local GPU. Speed can also compare a model running close to the evaluator with another reached over a hosted API. The board discloses these differences. When a claim sounds too neat, look for the endpoint and pricing notes beside the number.

How is JevBench different from a general AI benchmark?

Many AI benchmarks ask a model to solve exam questions, write code, or explain an image. JevBench asks a narrower question: can a system take state and a fixed rubric, then return a useful bounded judgment? It includes open-weight models, hosted decision models, and adapted general systems under the same decision task format. A model that writes beautiful prose may be a poor fit if your application needs one option and a reliable probability. A model that scores well on JevBench is not automatically good at writing, search, or image generation.

There are also other projects using similar benchmark names for classification experiments. When you see a JevBench result quoted online, check whether it refers to Benchmark Heaven's typed-decision JevBench and which release it uses. An accuracy figure from a separate text-classification dataset cannot be placed beside the official four-axis score as if they measured the same thing. Different labels, input lengths, and evaluation rules can change the result.

Turn a JevBench insight into your own test

The best use of JevBench is to help you ask better questions about your own work. Pick 30 to 50 cases that people on your team have already resolved. Include easy examples, unclear ones, and examples that should go to Other or manual review. Write one decision question and list allowed answers without overlap. Record the expected answer before running a model. Then compare correct choices, high-confidence mistakes, response time, and the cases that need a person.

Try one case in the Jev AI homepage Playground. Open Your own case, paste a short, non-sensitive message, and ask a Choice, Score, or Yes / No question. Select Run Jev when the service is available. Change one important detail, such as “refund not received” to “replacement not received,” while keeping the question and options the same. If the answer changes sensibly, you have learned something about your task. If it does not, save both cases and refine the question. The What Is Jev? article explains the decision types.

The Playground runs Jev, not the JevBench test suite or the other ranked models. It is a practical place to explore how structured decisions behave before you build a larger evaluation. Once you have your own labeled cases, you can judge whether the model's behavior fits your workflow. A public leaderboard helps you choose what to test; your own cases tell you what to trust.

JevBench FAQ

Is JevBench a Jev model?

No. JevBench is an independent benchmark for decision systems. Jev is one of the models it evaluates.

Does first place mean the most accurate model?

No. The official JevBench Score blends answer quality, calibration, speed, and cost. Check the accuracy, intelligence, and task-family results when correctness is your priority.

Are all JevBench questions public?

No. The current release combines 534 public decisions with 308 sealed decisions. The sealed set is reported in aggregate to test how models handle less familiar cases.

Can I run JevBench on this website?

This site's homepage Playground runs Jev on examples and your own text when the service is available. It does not run the full JevBench suite or host every model on the leaderboard.