Welcome to Quantum Magazine's podcast.

Each episode, we bring you stories about developments in science and mathematics.

I'm Susan Vallage.

To better understand how neural networks learn to simulate writing,

researchers are training simpler versions, using something right out of bedtime. Synthetic children's stories. That's next.

It's season three of the joy of why, and I still have a lot of questions. Like, what is this thing we call time? Why does altruism exist? And where is Janelle Levin?

I'm here. Astrophysicist and co-host. Ready for anything. That's right. I'm bringing in the A-team.

So brace yourselves. Get ready to learn. I'm Janelle Levin. I'm Steve Stroggatz.

And this is... Quantum Magazine's podcast, The Joy of Why. New episodes drop every other Thursday.

As countless students know, learning English is no easy task. But when the student is a computer,

one approach works surprisingly well.

Simply feed mountains of text from the Internet into a giant mathematical model called a neural network.

That's the operating principle behind generative language models, like OpenAI's chatPT, whose ability to converse coherently,

if not always truthfully, on a wide range of topics, has surprised researchers and the public

over the past year or so.

But the approach has its drawbacks.

For one thing, the training procedure required to transform vast text archives into state-of-the-art

language models is costly and time-intensive. For another, even the people who train large

language models find it hard to understand their inner workings. That, in turn, makes it hard

to predict the many ways they can fail.

Faced with these difficulties, some researchers have opted to train smaller models on smaller

data sets and then study their behavior.

...