Evaluations are critical for assessing the quality, performance, and effectiveness of software during development.

Common evaluation methods include code reviews and automated testing, and can help identify bugs, ensure compliance with requirements, and measure software reliability.

However, evaluating LLMs presents unique challenges due to their complexity,

versatility, and potential for unpredictable behavior. Onker Goil is the CEO and founder of

Brain Trust Data, which provides an end-to-m platform for AI application development and has a focus on

making LLM development robust and iterative.

Anker previously founded Empira, which was acquired by Figma, and he later ran the AI team at Figma.

Anker joins the show to talk about brain trust and the unique challenges of developing evaluations

in a non-deterministic context.

This episode is hosted by Sean Falconer. Check the show notes for more information on Sean's work

and where to find him. Anker, welcome the show.

Thank you so much for having me.

Yeah, absolutely.

Thanks for being here.

So let's talk about bearing trust.

How did you guys get started?

What was the original inspiration?

Yeah, so prior to brain trust, I used to lead the

AI team at Figma. And before that, I started a company called Impura, which Figma acquired. And at

impura, we were in like the stone ages of AI pre-chat GPT and built a product that helped you

extract data from documents. And basically, every time we would change something, like whether it was changing our models

or when we started using language models, like the prompts or even the code that fed stuff

...