Welcome to Quantum Magazine's podcast.

Each episode, we bring you stories about developments in science and mathematics.

I'm Susan Vallett.

In October of 2018, Google released a language model algorithm called Burt. It was Google's first language

model that was self-taught on a massive volume of online data, but it still struggled

with one thing. That's next. Quantum Magazine is an editorially independent online publication supported by the Simons Foundation

to enhance public understanding of science.

When Google released Burt, Nora Kastner, a language model researcher, quickly loaded it on her

laptop. Like her peers, Kastner was impressed that Burt could complete user's sentences and answer

simple questions. It seemed as if the large language model, or LLM, could read text at least as well as a human. But Kastner, who at the time

was a graduate student at Ludwig Maximilian University of Munich, remained skeptical. She felt

LLMs should understand what their answers mean and what they don't mean. It's one thing to know that a bird can fly.

Kastner says a model should automatically also know that the negated statement, a bird cannot fly,

is false. But when she and her advisor, Henrik Chutzh, tested Bert and two other LLMs in 2019, they found that the models behaved as

if words like not were invisible.

Since then, LLMs have skyrocketed in size and ability.

Ding Xiao leads the safe artificial intelligence lab at Carnegie Mellon University.

The algorithm itself is still similar to what we have before.

However, the scale and the performance is really astonishing.

But while chatbots have improved their human-like performances,

they still have trouble with negation.

They know what it means if a bird can't fly, but they

...