How bad could AI go in the worst case scenario? Let's look beyond the near-term risks and explore what could really happen if the wheels come completely off.

That's coming up right after this. Welcome to Big Technology Podcast, a show for cool-headed and nuanced conversation of the tech world and beyond.

Well, on the show, we've explored a lot of the

downsides of AI, a lot of the near-term risks, the business implications of what happens if things

don't continue to accelerate a pace. We haven't had a dedicated episode looking at what could

happen if things really go wrong. And so we're going to do that today. We're joined today by, I think,

the perfect guest for this conversation. Anthony Aguirre is here. He's the executive director

at the Future of Life Institute and also a professor of physics at UC Santa Cruz. Anthony, great to

see you. Welcome to the show. Thanks so much for having me on. Great to see you.

Great to see you. Nice to have a conversation again this time in public.

Suffice to say, you're not excited about all the progress that the AI industry is making.

Well, that's not quite true. So there's lots of progress in AI that I just love.

You know, I use AI models all the time. I love lots of AI applications in science and technology, lots of things where AI are tools that are letting us do things

that we couldn't do before. The thing that I'm concerned about is the direction that we're headed

in, which is toward increasingly autonomous and general and

intelligence systems, things that we've been calling AGI for a long time. And this, I think,

is different at some level from what we've been doing. And I think is where most of the danger

lies, especially on the large scale and in the longer term.

And there have been a number of studies in the training scenarios within the foundational model companies or foundational research houses, which are frontier research labs,

actually I think is probably the best way to refer to them. Where

AI has had this, it seems like a value or an instinct to try to preserve itself. In testing

scenarios, it's tried to copy its code out of the scenario when it thinks its values are being

...