Offline evals involve running your app against a dataset before production to measure performance and track changes over time.

In-the-loop evals occur during the agent's operation, enabling real-time corrections and improvements to response quality before the agent responds.

The evaluation process is continuous; it's not enough to just build a dataset and run it once. Ongoing evaluations are necessary to maintain quality in production systems.

There are three types of evaluations: offline evals, online evals, and in-the-loop evals, each serving different purposes in the evaluation lifecycle.

Writing effective evals is becoming a critical skill for product managers. Evals are like quizzes for models to gauge how well they know a certain set of subject material or how good they are at responding to specific questions.

We encourage rapid production to gather real user feedback, which is essential for improving our products and understanding customer needs.

Evals are considered the true crown jewel data asset for companies, as they provide context for why prompts were designed a certain way.

As an employee, you often have access to the information needed to assess whether the company is doing well, such as product analytics and user engagement.

Many developers overlook the importance of creating simple evaluations, which can help identify regressions and improve system performance incrementally.