Online evals assess the app's performance in real-time using actual production data, allowing for immediate tracking of performance.

The evaluation process is continuous; it's not enough to just build a dataset and run it once. Ongoing evaluations are necessary to maintain quality in production systems.

There are three types of evaluations: offline evals, online evals, and in-the-loop evals, each serving different purposes in the evaluation lifecycle.

Using in-the-loop evals is beneficial when the tolerance for mistakes is low, although it may increase time and costs.

Many developers overlook the importance of creating simple evaluations, which can help identify regressions and improve system performance incrementally.

Writing effective evals is becoming a critical skill for product managers. Evals are like quizzes for models to gauge how well they know a certain set of subject material or how good they are at responding to specific questions.

Building effective control evaluations is difficult, as they need to closely resemble actual deployment settings to yield valid insights.

It's crucial for application developers to build their own datasets and evaluators, as academic datasets often do not represent real user interactions. To build effective evaluators, it's essential to create datasets tailored to specific use cases, which can be easily done in LangSmith by adding input-output pairs to a dataset.

In-the-loop evals occur during the agent's operation, enabling real-time corrections and improvements to response quality before the agent responds.