Quality is the number one blocker for getting AI agents into production, according to a survey of agent builders.

To bridge the gap from prototype to production and improve quality, using evaluations throughout different stages of development is essential.

There are three types of evaluations: offline evals, online evals, and in-the-loop evals, each serving different purposes in the evaluation lifecycle.

Offline evals involve running your app against a dataset before production to measure performance and track changes over time.

Online evals assess the app's performance in real-time using actual production data, allowing for immediate tracking of performance.

In-the-loop evals occur during the agent's operation, enabling real-time corrections and improvements to response quality before the agent responds.

Using in-the-loop evals is beneficial when the tolerance for mistakes is low, although it may increase time and costs.

Evaluations consist of two parts: data and evaluators, which can vary depending on whether you're using ground truth or reference-free methods.

It's crucial for application developers to build their own datasets and evaluators, as academic datasets often do not represent real user interactions. To build effective evaluators, it's essential to create datasets tailored to specific use cases, which can be easily done in LangSmith by adding input-output pairs to a dataset.

Great evaluations start with great observability, and in LangSmith, we ensure that tracing captures all inputs, outputs, and steps to facilitate thorough evaluations.

There are three types of evaluators: deterministic code-based evaluations, LLM-as-judge techniques for more complex assessments, and human annotation for real-time feedback.

Using LLMs as judges for evaluating outputs is promising, but it requires careful prompt engineering to ensure accurate grading of responses.

We're launching open-source evaluators that simplify the process of evaluating code, retrieval-augmented generation, extraction, and tool calling, making it easier for developers to get started.

The evaluation process is continuous; it's not enough to just build a dataset and run it once. Ongoing evaluations are necessary to maintain quality in production systems.