*What to expect from this series, and what you should know before you start.*


![Maud Nalpas](https://web.dev/images/authors/maudnalpas.jpg) Maud Nalpas [GitHub](https://github.com/maudnals) [LinkedIn](https://www.linkedin.com/in/maudnalpas) [Bluesky](https://bsky.app/profile/maudnals.bsky.social)

<br />

In this series, you build evaluations (evals) for our example application,
ThemeBuilder. You'll learn how to:

- Build a robust, end-to-end evaluation workflow so you trust you can ship to your users.
- Use the LLM-as-a-judge pattern to measure subjective quality. Create a judge with minimal setup, or use advanced techniques to develop a custom judge that thinks like top-domain experts.
- Automate your pipeline by running evals at build time (CI/CD) and in production, to catch regressions early.
- Apply techniques that give you statistical confidence and prove your results aren't just a lucky draw from your test pool, and optimize your evals design to catch sneaky regressions.
- Use evals to select the best model for your use case.

> [!IMPORTANT]
> **Key Term:** A *judge model* is a language model that evaluates the quality of another model's output at scale. The models rely on a labeled dataset to act as the source of truth for what an acceptable response is.

## Approach

Think of this series as your starting point. You can build your full evals
pipeline using only the main guidance, which we've based on standard industry
best practices, and explore more advanced techniques when you're ready to
level up.

Whether you use a ready-made evals platform or build your own, the concepts and
techniques you'll learn are tool-agnostic. Understanding the why behind them
helps you dodge common traps and develop an expert evals pipeline, no matter
what stack you choose.

Once completed, you'll know how to iterate on your prompt, upgrade your LLM, or
switch your LLM while shipping to your users with confidence.

## Prerequisites

You should have some experience building with LLMs. We assume you're already comfortable with:

- LLM basics: [determinism](https://en.wikipedia.org/wiki/Deterministic_algorithm) versus [probability engines](https://cloud.google.com/transform/prompt-probability-data-and-the-gen-ai-mindset#:%7E:text=In%20contrast%2C%20the%20foundation%20models%20powering%20generative%20AI%20are%20probabilistic%20), [hallucination](https://developers.google.com/machine-learning/glossary#hallucination), [structured outputs](https://ai.google.dev/gemini-api/docs/structured-output), [temperature](https://developers.google.com/machine-learning/glossary#temperature).
- [Prompt engineering techniques](https://web.dev/learn/ai/prompt-engineering).
- Generative AI basics: model providers, [platforms](https://web.dev/learn/ai/platform), [benchmarks](https://wikipedia.org/wiki/Language_model_benchmark), and leaderboards.