How Does Jev Make Fast Calibrated Decisions?

TL;DR

Jev returns calibrated probability distributions instead of generated text, allowing application code to automate clear decisions and escalate uncertain cases. It supports choice, ordered score, and yes-or-no questions in one pass, while confidence-based routing separates cases that can be acted on immediately from those requiring confirmation, a larger language model, or human review.

Transcript

Everyone's talking about Jeff, but nobody is talking about the why and the how behind it. In this video, we'll first walk through how an LLM reasons and how Jeff does it differently. We'll see what makes Jeff so much faster and cheaper at the same time, and finally we'll build a game demo inspired by the popular game Papers, Please to show where Je... Read More

Key Insights

• Jev is a System One AI model designed primarily for application code, rather than chatbot conversations with people. It evaluates a supplied state, answers structured questions, and returns numerical probabilities that software can compare against thresholds before taking an action.
• A normal language model's generated confidence score is not necessarily calibrated because the described pre-training and RLHF process rewards next-word prediction and human-preferred answers. That process does not directly verify whether a stated confidence number matches the frequency of correct outcomes.
• RLCD is TypeSafe's reinforcement learning approach for calibrated decisions. It uses generated training data with known correct answers, allowing the system to score whether predicted probabilities match reality without relying on a person's subjective rating of each response.
• Calibration means that answers assigned a probability of 0.8 should be correct about 80 percent of the time across comparable decisions. This relationship makes the probabilities more useful for automated workflows than an unsupported confidence number produced as ordinary text.
• Jev supports three question types: choice selects among declared options, score places an item on an ordered scale, and null evaluates a yes-or-no statement with a number between zero and one. Multiple questions can be evaluated together in one run.
• A choice answer is a probability distribution whose values add up to one. If a ticket receives 0.62 for technical and 0.38 for billing, the result exposes meaningful ambiguity that a single department label would conceal.
• Confidence indicates how evenly probability is divided among available options. A dominant option produces high confidence, while a near-even split produces low confidence, giving application code a basis for deciding whether to act, confirm, or escalate.
• Jev's described routing policy acts automatically above about 0.9 confidence, confirms decisions between 0.5 and 0.9, and escalates decisions below 0.5. Escalation can send the case to a person or a conventional language model capable of slower reasoning.

Questions & Answers

Q: What is Jev and how is it different from a regular LLM?

Jev is a System One AI model from TypeSafe that is designed to answer application code with calibrated probabilities rather than generated prose. A regular language model may return a sentence or a single structured label, but Jev distributes probability across declared options and supplies confidence information. This lets software distinguish clear decisions from ambiguous ones before acting.

Q: Why should application code use probabilities instead of text answers?

Application code needs outputs that can drive predictable branches and actions. A sentence may mention several departments, making keyword matching unreliable, while a single label conceals whether the decision was clear or borderline. A probability distribution shows the relative support for each declared option, allowing code to act on strong results and route uncertain cases for confirmation or review.

Q: Why is a confidence score generated by a regular LLM unreliable?

The training process described for regular language models focuses first on predicting the next word and then on producing answers that people rate highly through reinforcement learning from human feedback. It does not necessarily compare the model's stated confidence against known correct outcomes. As a result, a generated number such as 0.9 may sound precise without being tied to actual correctness frequency.

Q: What does calibration mean for Jev's probabilities?

Calibration means predicted probabilities are intended to correspond to observed correctness across many answers. If Jev assigns a probability of 0.8 to a group of decisions, about 80 percent of those decisions should turn out to be correct. This makes the number operationally meaningful because an application can establish thresholds based on expected reliability rather than treating confidence as unsupported prose.

Q: What question types can Jev answer?

Jev accepts choice, score, and null questions. A choice question selects among declared alternatives such as billing, technical, and sales. A score question places an item on an ordered scale such as calm, annoyed, or furious. A null question evaluates a yes-or-no statement and returns a number between zero and one. Several questions can be asked in one run.

Q: How does confidence-gated routing work with Jev?

Confidence-gated routing connects the returned confidence value to different actions. The example policy acts automatically when confidence is about 0.9 or higher, requests confirmation when confidence falls between 0.5 and 0.9, and sends results below 0.5 to a person or a conventional language model that can reason more deeply. Thresholds therefore control automation and escalation.

Q: How can Jev classify ambiguous support tickets?

The ticket text is supplied as the state, and the routing question lists billing, technical, and sales as declared choices. Instead of returning only one department, Jev can assign values such as 0.62 to technical and 0.38 to billing. The distribution exposes ambiguity, while the accompanying confidence helps determine whether to route immediately, confirm the result, or request review.

Q: Why can Jev cost less than text-generating language models?

Text-generating models charge for input and output tokens, and the transcript says output can cost more because every generated token requires another processing cycle. Jev returns numbers and probability sets instead of producing substantial text. Its stated price is $42 per billion input tokens, about 4 cents per million, with output tokens free, and multiple questions can be answered in one pass.

Summary & Key Takeaways

• Conventional language models can classify support tickets, but a single label hides whether the decision was obvious or borderline. Asking such a model to generate its own confidence score does not make that score reliable because its described training process did not directly compare predicted confidence with known outcomes.

• TypeSafe developed Jev as a System One model for applications whose primary consumer is code. It receives a state and one or more structured questions, then returns probabilities for declared choices, ordered score levels, or yes-or-no statements. Its RLCD training approach is intended to keep probabilities calibrated against correctness.

• Jev enables confidence-gated workflows. High-confidence decisions can trigger immediate action, medium-confidence decisions can be confirmed, and low-confidence cases can go to a person or a larger language model. Because Jev returns numbers rather than generated prose and answers multiple questions in one pass, its pricing excludes output-token charges.