LLM-powered systems continue to move steadily into production, but this process is presenting

teams with challenges that traditional software practices don't commonly encounter.

Models and agents are non-deterministic systems, which makes it difficult to test changes,

reason about failures, and confidently ship updates.

This has created the need for new evaluation tooling

designed specifically around the properties of LLMs. Comet is a platform with Roots and MLOps

that has evolved to support teams building modern LLM-powered applications. The company recently

launched OPEC, which is an open source platform focused on

evaluation, optimization, and observability for LLM agents. Together, the tools aim to bring the rigor

of traditional engineering and ML workflows to the rapidly evolving world of agent-based systems

by treating prompts, tools, and workflows

as optimizable components that can be evaluated and improved over time.

Gideon Mendels is the co-founder and CEO of Comet.

He previously worked at Google on hate speech and deception detection, and he founded

GroupWise, which trained and

deployed NLP models processing billions of chats. In this episode, Gideon joins Kevin Ball

to discuss how agent development sits between software engineering and ML, why e-vals are the

missing foundation for most AI teams, prompt optimization as a search problem,

and the future for continuously improving agents in production.

Kevin Ball, or K. Ball, is the vice president of engineering at Mento and an independent

coach for engineers and engineering leaders. He co-founded and served as CTO for two companies,

founded the San Diego JavaScript meetup,

...