Score

Rate a passage.

Retrieval returns the ten nearest passages whether or not any of them answer the question. Passing all ten to an LLM costs input tokens on passages that cannot help it answer. A Score question rates each passage against ordered levels, so your code can drop the weak ones before paying to read them.

The rubric sent to Jev

0Unrelated. The passage is about a different subject and contributes nothing to the question.
1Related topic. The passage concerns the same subject but does not address what was asked.
2Partial. The passage supplies part of what the question asks for, or implies the answer without stating it.
3Direct. The passage states the answer to the question.

Score

Rate a passage

Run with

Shows recorded and illustrative answers. This site's own live testing is unavailable.

Output

Level: 2.83 of 3

Direct. The passage states the answer to the question.

Level 01.0%
Level 12.0%
Level 210.0%
Level 387.0%

This score is Jev's read of how well the passage answers the question, against the rubric above. It does not check whether the passage is true, and it rates this passage alone, not the rest of your results. Confidence: 0.83.

Illustrative example

Not a live or recorded response. Shown to illustrate the shape of an answer.

Build this

passage-relevance.ts

import { score, TypeSafeClient } from "@typesafe-ai/sdk";
 
const client = new TypeSafeClient(); // reads TYPESAFE_API_KEY, defaults to jev-latest
 
const question = "What is the maximum file size for uploads?";
 
const { answers } = await client.systemOne({
  state: question,
  model: "jev-latest",
  questions: {
    relevance: score(
      "How well does this passage answer the question it was retrieved for? Judge only how useful the passage is for answering that question, not whether it is well written, recent, or true.",
      [
        "Unrelated. The passage is about a different subject and contributes nothing to the question.",
        "Related topic. The passage concerns the same subject but does not address what was asked.",
        "Partial. The passage supplies part of what the question asks for, or implies the answer without stating it.",
        "Direct. The passage states the answer to the question.",
      ],
    ),
  },
});
 
const { score: level, legend, probabilities, confidence } = answers.relevance;
 
// `level` can land between two whole levels; round it before indexing `legend`.
// Object.values() reads levels in order (0, 1, 2, ...) without depending on the exact key type.
console.log("Level " + level.toFixed(2) + ": " + Object.values(legend)[Math.round(level)]);
console.log(probabilities, confidence);

Type-checked against @typesafe-ai/sdk 0.6.0 as part of this site's build. Not executed against the live API by this guide.

npm install @typesafe-ai/sdk. Requires Node.js 20+. Set TYPESAFE_API_KEY in your environment.

Keep TYPESAFE_API_KEY on the server. Never embed it in client code or ship it in a browser bundle.

Reading the three samples

Answers it directly. The question asks for a file size limit and the passage states one. This should land near the top level, and it is the case your retrieval step is supposed to produce.

Same topic, no answer. Both the question and the passage are about uploads, so a similarity search will rank this passage highly. It never mentions size. This is the gap the recipe is for: embedding similarity measures whether two texts are about the same thing, which is not the same as whether one answers the other.

Answer implied, not stated. The passage says SSO is on Enterprise and that every other plan uses passwords or Google. A reader can deduce that Team does not include SSO, but the passage never says so. Whether that counts as an answer depends on what you intend to do with it, which is why this sample is marked ambiguous and why the level alone is not enough to act on.

The level is one axis, confidence is another

The answer carries a confidence alongside the score, and confidence is derived from the shape of the probability distribution: concentrated on one level means a certain answer, spread across several means an uncertain one.

That matters because two very different passages can produce nearly the same score. Probability massed on “partial” is Jev saying this passage partly answers the question, and confidence will be high. Probability split between “related topic” and “direct” averages out to roughly the same number while meaning something else entirely: Jev could not place the passage, and confidence will be low. Thresholding on the score alone treats those two identically. Reading both does not:

const { score: level, confidence } = answers.relevance;

if (confidence < 0.6) {
  review.push(passage);        // Jev did not commit. Do not silently drop it.
} else if (level >= 2) {
  keep.push(passage);          // Answers the question, at least in part.
}                              // Everything else falls away.

This is confidence-gated routing. Note that it is your code overriding a usable answer because the model was not certain enough. Adding an “uncertain” level to the rubric would not achieve the same thing, because the model can select such a level confidently.

Where this fails

It rates one passage against one question, in isolation. It cannot tell you that the corpus contains no answer at all, that two passages are duplicates, or that the best passage is still wrong. Ranking a shortlist is a different job from filtering one.

The levels are a rubric you wrote, and the answer is only as well defined as they are. If “partial” and “direct” overlap in your domain, Jev will place passages between them and confidence will fall, which is the model reporting your rubric back to you.

Thresholds belong to your application. Dropping everything below level 2 is right for a cost-sensitive pipeline and wrong where a missed passage matters more than a wasted token.

How the TypeSafe cookbook differs

TypeSafe's classifying RAG passages cookbook sends one request per passage, each carrying four Noul questions: whether the passage addresses the subject of the query, whether it states information usable in a direct answer, whether it contradicts the premise of the query, and whether it contains a prompt injection. Code then combines those four numbers. This page folds the first two into one ordered scale, where level 1 is on the subject but no answer and level 3 states the answer, so it says nothing about contradictions or injected instructions. Read the cookbook before building the real pipeline.

For the cost side of dropping passages before they reach a model, see the pricing guide, and for whether this call belongs with Jev at all, Jev vs LLMs.

Other recipes

All recipes