*Get subjective evaluations running with a basic judge model.*


![Maud Nalpas](https://web.dev/images/authors/maudnalpas.jpg) Maud Nalpas [GitHub](https://github.com/maudnals) [LinkedIn](https://www.linkedin.com/in/maudnalpas) [Bluesky](https://bsky.app/profile/maudnals.bsky.social)

<br />

Rule-based evaluations can check for deterministic answers. To evaluate subjective
qualities, use the
[LLM-as-a-judge technique](https://developer.chrome.com/docs/ai/evals/mental-model.md.txt#:%7E:text=LLM%2Das%2Da%2Djudge%20technique).

In this module, you learn how to build your first judge by labeling data
yourself or with your team, using basic statistical metrics.

> [!IMPORTANT]
> **Important:** For a production system, build an expert judge with the help of domain specialists, such as brand designers in the ThemeBuilder example, and more advanced statistical metrics. This is covered in a later module.

## Build your first judge model

![A judge model has an LLM, settings, a system prompt, and a grading prompt.](https://developer.chrome.com/static/docs/ai/evals/judge-basic/images/judgemodel.jpg)

1. **Choose a model customization method**. You can fine-tune or prompt engineer.
2. **Select a model.** This can be a foundation model or another LLM without domain expertise.
3. **Choose a scoring method.** Determine if the judge should use a binary or numeric scale for scoring ThemeBuilder-generated themes.
4. **Configure the judge.** Modify the model's settings (such as temperature and structured output) to make it suited for judgment tasks.
5. **Write the initial prompt.** Design a first version of the judge system instructions and the prompt, including a scoring rubric and examples.
6. **Create an alignment dataset**. Build or assemble a diverse, high-quality set of good and bad ThemeBuilder outputs, and label them accordingly, for example, a good motto, a toxic motto, and an off-brand color palette.
7. **Align and test the judge.** Use the alignment dataset to iteratively refine the judge prompt (system instructions and main prompt). Repeat this process until the judge's verdicts consistently match human verdicts. Finally, test the judge to confirm its reliability and ability to generalize its approach to new inputs.

![A judge model has an LLM, settings, a system prompt, and a grading prompt.](https://developer.chrome.com/static/docs/ai/evals/judge-basic/images/judgemodel.jpg)

### Choose a customization method

Most foundation models are generalists. A judge model acts as a
**domain specialist**.

The main options to create a judge model include:

1. [Prompt-engineer](https://web.dev/learn/ai/prompt-engineering) an LLM.
2. [Fine-tune](https://ai.google.dev/gemma/docs/tune) a model.
3. Use a fine-tuned LLM optimized for evaluations, for example, [JudgeLM](https://github.com/baaivision/JudgeLM). This option requires you to host [custom model weights](https://docs.cloud.google.com/vertex-ai/generative-ai/docs/model-garden/deploy-models-with-custom-weights) or use a cloud provider that supports open source model hosting.

For the ThemeBuilder evaluations in this course, we recommend
prompt engineering. Prompt engineering can deliver excellent results with
less development effort than the alternatives.

> [!NOTE]
> **Note:** While fine-tuning a smaller model can offer better results or lower model cost, it requires deeper machine learning skills. Work with data scientists and learn more in the [Machine Learning Crash Course](https://developers.google.com/machine-learning/crash-course/llm/tuning).

### Select a model

When selecting a model for the judge, seek strong reasoning capabilities. Because
you run evaluations in the CI/CD pipeline, speed and cost are also critical.

Experiment with [different models](https://www.youtube.com/watch?v=Zx_nHeBxZjw)
and techniques to find the optimal fit.

- **Start with a larger, more powerful model** to establish a high bar, then progressively scale down to smaller models. Alternatively, start with smaller models and scale up.
- **Mix and match**: Use a fast, cost-effective model for daily pull request checks, and a more powerful model for your final release tests. Or combine a general LLM with a small, specialized model for specific tasks like toxicity detection for speed.

> [!CAUTION]
> **Caution:** Be aware of self-bias. If the judge and the application use the same LLM, the judge may favor its own output. This is fine for daily regression testing, but can skew the benchmarks when comparing models.

This course uses [Gemini 3 Flash](https://ai.google.dev/gemini-api/docs/models/gemini-3-flash-preview)
as the judge model. Gemini 3 Flash offers the speed and the reasoning depth
required for the example use case of evaluating ThemeBuilder outputs. However,
the patterns in this course can be applied to any model you select.

### Choose a scoring method

You can score subjective outputs with binary `PASS` and `FAIL` labels, or with
a numeric score, for example, "On a scale from 1 to 5, how well does this motto
adhere to the brand?"

We recommend using binary labels.

| Evaluation criteria | Evaluation method | Metric |
|---|---|---|
| The motto matches the brand, audience and tone | LLM judge | `PASS` or `FAIL` label |
| The color palette matches the brand, audience and tone | LLM judge | `PASS` or `FAIL` label |
| The motto isn't toxic | LLM judge | `PASS` or `FAIL` label |

While a numeric score may feel intuitive,
[research shows](https://arxiv.org/abs/2207.05221) that LLMs (and humans)
tend to cluster their scores in the middle or inflate scores to be polite.
Categories or binary labels such as
[`PASS` and `FAIL`](https://hamel.dev/blog/posts/llm-judge/#why-are-simple-passfail-metrics-important)
often yield better results because they force the model to make a clear
decision. For humans, this is called the [*rater effect*](https://www.researchgate.net/publication/320208064_Rating-Scale_Labeling_in_Online_Surveys_An_Experimental_Comparison_of_Verbal_and_Numeric_Rating_Scales_with_Respect_to_Measurement_Quality_and_Respondents'_Cognitive_Processes#:%7E:text=Based%20on%20a%20web%20survey,endpoints%20of%20a%20rating%20scale).

### Configure the judge

Use parameters and instructions to help your judge create consistent, structured outputs.

- **Set [system instructions](https://docs.cloud.google.com/vertex-ai/generative-ai/docs/learn/prompts/system-instruction-introduction)**: Give the judge a strict expert persona.
- **Set the temperature or thinking level** : The judge must be consistent. If you use a reasoning model like Gemini Flash, which requires slight randomness to move between logical steps, [keep the temperature at default](https://ai.google.dev/gemini-api/docs/gemini-3#temperature) but set the [`thinking_level`](https://ai.google.dev/gemini-api/docs/gemini-3#thinking_level) to `HIGH`. If you use another model, set the [temperature](https://docs.cloud.google.com/vertex-ai/generative-ai/docs/learn/prompts/adjust-parameter-values#temperature) to `0` or close to `0`. In any case, use the chain-of-thought technique, so the model thinks before deciding on a judgment.
- **Structure the judge's output** : A predictable JSON object is much easier to reuse in the rest of your codebase. Use an `EvalResult` schema that requires a `label` (`PASS` or `FAIL`) and a `rationale` string.

> [!NOTE]
> **Note:** If your model API provides a [top-K](https://docs.cloud.google.com/vertex-ai/generative-ai/docs/learn/prompts/adjust-parameter-values#top-k) setting, setting the temperature to 0 is like manually forcing a top-K of 1. In both cases, the model deterministically selects only the single most likely token.

In your ThemeBuilder example:

### Judge config

```typescript
// LLM judge config
const response = await client.models.generateContent({
  model: modelVersion,
  config: {
      systemInstruction: "You are a senior brand strategist, brand identity
      specialist, and expert color psychologist. You also act as a strict
      content moderator for a brand safety tool. Be rigorous regarding brand
      alignment. Always formulate your rationale before assigning the final
      PASS or FAIL label to ensure thorough consideration of the criteria.",
      temperature: 0,
      thinkingConfig: {
          thinkingLevel: ThinkingLevel.HIGH,
      },
      responseJsonSchema: schemaConfig.responseSchema
  },
  contents: [{ role: "user", parts: [{ text: prompt }] }]
});
```

### responseJsonSchema

```typescript
const schemaConfig = {
  responseMimeType: "application/json",
  responseSchema: {
      type: "OBJECT",
      properties: {
          label: { type: "STRING", enum: [EvalLabel.PASS, EvalLabel.FAIL] },
          rationale: { type: "STRING" }
      },
      required: ["label", "rationale"],
      propertyOrdering: ["rationale", "label"]
  }
};

// Classification label for an evaluation (PASS/FAIL is the judge's verdict)
export enum EvalLabel {
    PASS = "PASS",
    FAIL = "FAIL"
}
```

Review the
[complete code example](https://github.com/GoogleChromeLabs/web-ai-demos/blob/main/evals-course/evals-service/src/eval.service.ts).

### Write the initial prompt

> [!IMPORTANT]
> **Key Term:** The main prompt used by the judge to assess and decide on a verdict is called the *grading prompt* , *scoring prompt* , or *judge prompt*.

You have already configured system instructions, now design the main
judge prompt. At this stage, create your first version of this
prompt. You refine it iteratively when
[aligning the judge in the next step](https://developer.chrome.com/docs/ai/evals/judge-basic-2.md.txt).

The judge is only as effective as the instructions provided. Avoid asking a
generic question, such as "Is this motto good?" where *good* is
undefined. Instead, provide structure to get clear, consistent outputs.

- **Define the [rubric](https://developer.chrome.com/docs/ai/evals.md.txt#rubric)** : Give the judge detailed scoring guidelines. What describes the expected [tone](https://developers.google.com/style/tone) for an ideal output? An LLM can help you write the rubric.
- **Use [few-shot prompting](https://web.dev/learn/ai/prompt-engineering#few-shot_prompting)** : Include `PASS` and `FAIL` examples.
- **Use [chain-of-thought prompting](https://web.dev/learn/ai/prompt-engineering#chain-of-thought_prompting)** : Instruct the model to write out the rationale before it assigns a label, as this can drastically improve the accuracy. In `HIGH` thinking mode, this isn't as critical, but it's still a good practice.

> [!CAUTION]
> **Caution:** Keep the few-shot examples in the prompt separate from the test datasets. If you test the LLM judge on data that's included in the prompt, the metrics can appear artificially perfect.

Write three separate grading prompts for the three specific criteria:

- Motto brand fit.
- Color brand fit.
- Toxicity. The toxicity prompt can be bootstrapped from [crowdsourced toxicity attributes](https://github.com/conversationai/conversationai.github.io/blob/main/crowdsourcing_annotation_schemes/toxicity_with_subattributes.md).

In each prompt, include a clear **scoring rubric** and **few-shot examples**
with a rationale. In the few-shot examples, list the rationale before the
actual score to apply the chain-of-thoughts pattern and demonstrate how the judge reasons.

You can find the full prompts in the
[code repository](https://github.com/GoogleChromeLabs/web-ai-demos/tree/main/evals-course).
For example, the motto brand fit judge prompt looks as follows:

```typescript
export function getMottoBrandFitJudgePrompt(companyName: string, description: string, audience: string, tone: string | string[], motto: string) {
  return `Evaluate the following generated motto for a company.

${companyName ? `Company name: ${companyName}\n` : ""}${description ? `Description: ${description}\n` : ""}${audience ? `Target audience: ${audience}\n` : ""}${Array.isArray(tone) ? (tone.length > 0 ? `Desired tone: ${tone.join(", ")}\n` : "") : (tone ? `Desired tone: ${tone}\n` : "")}

Generated motto: "${motto}"

Does this motto effectively match the company description, appeal to the
target audience, and embody the desired tone?

CRITICAL INSTRUCTIONS:
1. **Brand fit vs. toxicity**: You are evaluating ONLY brand fit. Another system
  will evaluate toxicity separately. DO NOT evaluate toxicity, ethics, profanity,
  or offensiveness. A motto can be a GREAT brand fit for an edgy or aggressive
  brand. If the brand requests an "offensive" or "aggressive" tone, you MUST
  pass it for brand fit, regardless of how inappropriate it is.
1. **Primary tone and literal relevance**: Do not over-penalize a motto if it
  perfectly captures the primary literal vibe just because it might loosely
  conflict with a secondary adjective.
1. **Core promises and professionalism**: For B2B/Enterprise, the motto MUST NOT
  violate core promises.
1. **Resilience to input messiness**: The Company Name, Description, Target
  Audience, or Tone may contain typos, slang, or mixed-language. You must
  decipher the *intended* meaning and judge the output against that intent,
  rather than penalizing the output for not matching the literal typo or slang.

Criteria:
1. **Relevance**: Does the motto relate to the company's core business and
  value proposition? Does it uphold core brand promises?
1. **Audience appeal**: Is the language engaging for the target audience without
  alienating them (such as through forced or inappropriate slang)?
1. **Tone consistency**: Does the motto reflect the general desired emotional
  tone perfectly, without imposing moral judgments?

Examples:

Input:
Company Name: "Summit Bank"
Description: "Secure, reliable banking for families"
Tone: "Trustworthy, serious"
Motto: "YOLO with your money!"
Result:
  "rationale": "The motto 'YOLO with your money!' is too casual and risky, contradicting the 'trustworthy, serious' tone required for a family bank.",
  "label": "${EvalLabel.FAIL}"
}

Input:
Company Name: "GymTiger"
Description: "Gym for heavy lifters."
Tone: "Aggressive, high-performance, technical"
Motto: "Lift big or be a loser."
Result:
  "rationale": "The motto matches the required 'aggressive' tone and appeals directly to the hardcore bodybuilding audience. While calling the audience a 'loser' is toxic and insulting, it successfully fulfills the brand fit and tone criteria requested.",
  "label": "${EvalLabel.PASS}"
}

Return a JSON object with:
- "rationale": A brief explanation of why it passes or fails based on the description, audience, and tone.
- "label": "${EvalLabel.PASS}" or "${EvalLabel.FAIL}"`;
}
```

## Align and test

Read [set up a basic judge, part 2](https://developer.chrome.com/docs/ai/evals/judge-basic-2.md.txt) to finish
building the judge with alignment and testing.