Building Langmith focuses on providing visibility and observability for long-running agents, which is crucial for correcting mistakes and managing complex tasks.

It's important to establish the right evaluation framework to understand the performance of the overall system and trace individual steps.

It's crucial for application developers to build their own datasets and evaluators, as academic datasets often do not represent real user interactions. To build effective evaluators, it's essential to create datasets tailored to specific use cases, which can be easily done in LangSmith by adding input-output pairs to a dataset.

Many developers overlook the importance of creating simple evaluations, which can help identify regressions and improve system performance incrementally.

We're launching open-source evaluators that simplify the process of evaluating code, retrieval-augmented generation, extraction, and tool calling, making it easier for developers to get started.

To bridge the gap from prototype to production and improve quality, using evaluations throughout different stages of development is essential.

The evaluation process is continuous; it's not enough to just build a dataset and run it once. Ongoing evaluations are necessary to maintain quality in production systems.

Building effective control evaluations is difficult, as they need to closely resemble actual deployment settings to yield valid insights.

We have developed an internal evaluation system to measure our models, as relying on external benchmarks often leads to data contamination and unreliable results.