So having covered living in space from the practical to the ridiculous, you did touch earlier on AI governance.

You seem to have gone a little bit negative on the whole thing.

What was going on there? I think that the problem with the way that we examine AI is we tend

to over and under anthropovise it in that we tend to assume that it can't become dangerous

until it's like us, and that's not true. And we also tend to assume that it can't become dangerous until it's like us, and that's not true.

And we also tend to assume that it only becomes dangerous in the ways that human beings are dangerous, and that's also not true.

So I understand in principle, but what do you mean by that?

So the classic example of the AI going wrong was the paperclip machine, right?

Like a paperclip machine is dangerous because, and it's a toy example, it's comical people laugh at it.

You tell a machine, you give it a reward function, you make it optimizer, is the AI research term,

where its reward function is just

the more paperclips there are, the higher the reward, right? And assuming that it's a sufficiently

powerful AI, what it immediately does is kill everyone on earth and turn them into paper clips.

You know, you extract the iron from blood, you, you convert everything, because there's no

bounding to it, right. But that's obviously ridiculous.

The idea of the AI, given that autonomy and not stopped at any point, is silly.

Okay, well, that might be silly, but I can well envisage a future in which the AI is told,

the most important thing that you can do is stop climate change.

By the way, humans cause climate change, right off you go and run our society.

Yeah.

And now there are ways to build AI that aren't optimizers.

You can build satisfacers, which essentially say if you pass a certain threshold, you don't care. The problem is that satisfacer's tend to build optimizers, you can build satisfices, which essentially say if you pass a certain threshold

...