*Non-exhaustive list of sources used in this course and evals tools that can help you.*


![Maud Nalpas](https://web.dev/images/authors/maudnalpas.jpg) Maud Nalpas [GitHub](https://github.com/maudnals) [LinkedIn](https://www.linkedin.com/in/maudnalpas) [Bluesky](https://bsky.app/profile/maudnals.bsky.social)

<br />

For more resources on testing and AI, we recommend the following resources.

- [Learn Testing](https://web.dev/learn/testing): Refresh your approach to testing.
- [Learn AI](https://web.dev/learn/ai): Design AI systems for your websites and web applications.
- [Google DeepMind Evals](https://deepmind.google/research/evals/): Multiple standardized benchmarking tools for different types on models
- [Gemini Evaluations Playbook](https://googlecloudplatform.github.io/applied-ai-engineering-samples/genai-on-vertex-ai/gemini/evals_playbook/): Recipes for experimenting and evaluating generative AI models with Vertex AI.
- [Responsible AI toolkit](https://ai.google.dev/responsible/docs/evaluation): Evaluate models and systems for safety.
- [Evaluating your evals](https://eugeneyan.com/writing/llm-evaluators/): A meta lesson on how to understand what evals to use, and what works effectively.
- [Building better AI benchmarks: How many raters are enough?](https://research.google/blog/building-better-ai-benchmarks-how-many-raters-are-enough/) Understand an evaluation framework for ML models that optimizes the trade-off between the number of items and raters per item, to build reproducible AI benchmarks.

## Course sources

We relied on several sources to write this series, including:

- AI Engineering: Building Applications with Foundation Models, [Chip Huyen](https://scholar.google.com/citations?user=pwg2DGwAAAAJ)
- [De-risking QA for LLM-powered applications](https://hablich.dev/articles/deriskingQA4AI.html) by Michael Hablich, Chrome DevTools
- [Using LLM-as-a-Judge For Evaluation: A Complete Guide](https://hamel.dev/blog/posts/llm-judge/) by Hamel Husain

## Eval tools

Examples of evals solutions and tools include:

- [AlignEval](https://eugeneyan.com/writing/aligneval/)
- [Arize AX](https://arize.com/ax) and [Phoenix](https://arize.com/phoenix/)
- [Braintrust](https://www.braintrust.dev/docs/evaluate)
- [Datadog](https://docs.datadoghq.com/llm_observability/evaluations/)
- [DeepEval](https://deepeval.com/docs/getting-started)
- [Gen AI evaluation service](https://docs.cloud.google.com/vertex-ai/generative-ai/docs/models/evaluation-overview) and [API](https://docs.cloud.google.com/vertex-ai/generative-ai/docs/model-reference/evaluation)
- [Inspect Evals](https://inspect.aisi.org.uk/evals/)
- [JudgeLM](https://github.com/baaivision/judgelm)
- [LangSmith](https://docs.langchain.com/langsmith/evaluation)
- [Evaluation harness](https://github.com/EleutherAI/lm-evaluation-harness)
- [OpenEvals](https://github.com/langchain-ai/openevals?tab=readme-ov-file)

There are many more eval tools available. If you are using other tools,
[share them with us](https://developer.chrome.com/docs/ai/team).