0:00

I believe you can make the entire case for

0:02

being extremely concerned about AI, assuming

0:04

that AI will never be smarter than a human. Instead,

0:07

it will be as capable as the most

0:09

capable humans. And there will be a ton

0:12

of them because unlike humans, you

0:14

can just copy them. You can use your copies

0:16

to come up with ways to make it more efficient,

0:18

just like humans do. Then you can make more copies.

0:21

And when we talk about whether AI could defeat

0:23

humanity, and I've written one blog post on whether AI

0:25

could kind of like take over the world, they don't

0:28

have to be more capable than humans. They

0:30

could be equally capable and there could be more of them

0:32

that, that could really do it. That could really be enough

0:35

that then we wouldn't be, humans wouldn't be

0:37

in control of the world anymore. So I'm

0:39

basically generally happy

0:41

to just have all discussions about AI and what

0:43

the risks are just in this world where like,

0:46

there's nothing more capable than a human, but it's pretty scary

0:48

to have a lot of those that have different values

0:50

from humans that are kind of a second advanced species.

0:52

Um, that's not to rule out that, that some

0:54

of these super intelligence concerns could be real. It's just

0:56

like, they're not always necessary and they can sideline

0:59

people.

1:01

People always love an interview with Holden,

1:04

founder of Open Philanthropy. We last

1:06

spoke with Holden in 2021 about

1:08

his theory that we're plausibly living in the

1:10

most important century of all of those that are

1:12

yet to come. And today we discuss other

1:15

things that have been preoccupying him lately, including

1:17

what the real challenges are that are raised by

1:19

rapid advances in AI. Why not

1:21

just gradually solve those problems as they come up? What

1:24

multiple different groups are able to do about it, including

1:27

listeners to this show, governments, computer

1:29

security experts, journalists, and on and on.

1:32

What various different groups are getting wrong about

1:34

AI in Holden's opinion, how we might

1:36

just succeed with artificial intelligence

1:38

by sheer luck. Holden's four

1:41

different categories of useful work

1:43

to help with AI, plus a few random

1:45

audience questions as well. At the end, we

1:48

also talk through why Holden rejects

1:50

impartiality as a core principle

1:52

of morality and his

1:54

non-realist conception of why it is that he

1:56

bothers to try to help others at all. After

1:59

the interview.

1:59

I also respond to some reactions

2:02

we got to the previous interview with Ezra

2:04

Klein. Without further ado, I bring you Holden

2:06

Karnovsky.

2:18

Today I'm again speaking with Holden Karnovsky.

2:21

In 2007, Holden co-founded the charity evaluator

2:23

GiveWell, and then in 2014, he co-founded

2:26

the foundation Open Philanthropy, which works

2:28

to find the highest impact grant opportunities, and has

2:30

so far recommended around $2 billion in grants. But

2:33

in March 2023, Holden started

2:35

a leave of absence from Open Philanthropy to instead

2:37

explore working directly on AI safety and

2:39

ways of improving outcomes from recent advances

2:42

in AI. He also blogs at cold-takes.com

2:45

about futurism, quantitative macro history,

2:47

and epistemology, though recently he's had, again,

2:49

a particular focus on AI, writing posts like

2:51

How We Could Stumble Into AI Catastrophe, Racing

2:54

Through a Minefield, The AI Deployment Problem, and

2:56

Jobs That Can Help With the Most Important Century. I

2:58

should note that Open Philanthropy is 80,000 Hours' biggest

3:00

supporter, and that Holden's wife is Daniela

3:03

Amadei, the president of the AI Lab Anthropic.

3:05

Thanks for returning to the podcast, Holden.

3:07

Thanks for having me. I hope to talk about

3:09

what you'd most like to see people do to positively

3:11

shape the development of AI, as well as your

3:13

reservations about utilitarianism. But first,

3:16

what are you working on at the moment, and why do you think it's important?

3:19

Sure. Currently on a leave of

3:21

absence from Open Philanthropy, just taking a little

3:23

time to explore different ways I might be able

3:25

to do direct work to reduce

3:27

potential risks from advanced AI. One

3:30

of the main things I've been trying to do recently, although

3:32

there's a couple things I'm looking into, is understanding

3:34

what it might look like to have AI safety

3:36

standards, by which I mean documenting

3:40

expectations that AI companies

3:42

and other labs won't build and deploy

3:44

AIs that pose too much risk to the world, as

3:47

evaluated by some sort of systematic evaluation

3:50

regime. These

3:52

expectations could be done via

3:54

self-regulation, via regulation-regulation.

3:57

There's a lot of potential interrelated pieces. So

4:00

to make this work, I think you would need ways of

4:02

evaluating when an AI system is dangerous.

4:04

That's sometimes called eval's. Then you would

4:06

need potential standards that would

4:09

basically talk about how you connect

4:11

what you're seeing from the eval's with what kind of measures

4:13

you have to take to ensure safety. Then also

4:16

to make this whole system work, you would need some

4:18

way of making the case to the general public that standards

4:20

are important, so companies are more likely

4:22

to adhere to them. And so things

4:25

I've been working on include trying to understand

4:27

what standards look like in more mature industries,

4:30

doing one case study and kind of trying to fund

4:32

some other ones there, trying to learn lessons

4:35

from what's already worked in the past elsewhere. I've

4:37

been advising groups like ArcEval's

4:40

thinking about what eval's and standards could look like.

4:43

And I've been trying to understand what pieces of the

4:45

case for standards are and are not resonating with people

4:47

so I can think about how to kind of increase

4:50

public support for this kind of thing. So

4:52

it's pretty exploratory right now, but that's been one of the

4:54

main things I've been thinking about. Yeah,

4:56

I guess there's a huge surface area on

4:58

ways one might attack this problem. Why the focus

5:01

on standards and eval's in particular?

5:03

Yeah, sure. I mean, we can get to this, but

5:05

I've kind of been thinking a lot about what

5:08

the major risks are from advanced AI and what

5:10

the major ways of reducing them are. And,

5:13

you know, I think there's kind of a few different

5:15

components that seem most promising

5:17

to me as part of,

5:19

I don't know, most stories I can tell in my head for

5:21

how we get the risks down very far.

5:24

And this is the piece of the puzzle that

5:26

to me feels

5:27

like it has a lot of potential, but there isn't much

5:29

action on it right now and that someone

5:32

with my skills can potentially play a big role in

5:34

helping to get things off the ground, helping to spur

5:36

a bit more action, getting people

5:39

to, you know,

5:40

to just like move a little faster. I'm

5:42

not sure this is like what I want to be doing for

5:44

the long run, but I think it's in this kind of nascent

5:47

phase where my kind of background

5:49

with just starting up things in very vaguely

5:51

defined areas and getting to the point where they're a little bit more

5:53

mature is maybe helpful.

5:56

Yeah. How has it been no longer leading

5:58

an organization with tons of... employees

6:00

and being able to self-direct a little bit more. You

6:02

wouldn't have been in that situation for quite some time,

6:04

I guess.

6:05

Yeah, I mean, it was somewhat gradual.

6:07

So we've been talking for for

6:10

several years now, I think, at least since 2018 or

6:12

so about succession at Open Philanthropy,

6:15

because I've always been a person who likes

6:17

to start things and likes to be in there in that nascent

6:19

phase and prefers always

6:21

to find someone else who can run an organization

6:24

for the long run. And we, you know, a couple

6:26

years ago, Alexander became co-CEO

6:29

with me and has taken on more and more duties.

6:31

So it wasn't an overnight thing. And

6:33

I'm still not completely uninvolved.

6:35

I mean, I'm

6:35

still on the board of Open Philanthropy. I'm

6:37

still meeting with people. I'm still advising, you

6:40

know, kind of similar to GiveWell. I had a very, very

6:43

gradual transition away from GiveWell and

6:45

still talk to them frequently. So

6:47

it's, you know, it's been a gradual thing. But for me, it

6:49

is an improvement. I think it's not my happy

6:52

place to be at the top of that org chart.

6:54

Yeah.

6:55

Okay, so today, we're

6:57

not going to rehash the basic arguments for worrying

6:59

about ways that AI advances

7:02

could go wrong, or ways that maybe this century

7:04

could turn out to be really unusually important.

7:06

I think, you know, AI risk, people have heard of it now.

7:09

We've done lots of episodes on it. I

7:12

guess people who wanted to hear your broader worldview

7:14

on this could go back to previous interviews, including

7:16

with you, such as episode 109, Holden Knofsky on

7:18

the most important century. In

7:20

my mind, the risks are both

7:22

pretty near term, and I think increasingly kind of apparent.

7:25

So to me, it feels like the point in this

7:27

whole story where we need to get down a bit more to brass tacks

7:30

and start debating what is to be done and

7:32

figuring out what things might might really help that we could

7:35

get moving on. That said, we should

7:37

take a minute to think about, you know, which aspect of

7:39

this broader problem are we talking about today? And which one

7:41

are you thinking about? Of course, there's risks

7:43

from misalignment. So AI models

7:46

completely flipping out and trying to take over would

7:48

be an extreme case of that. Then there's misuse, where

7:51

the models are doing what people are telling them to do,

7:53

but we wish that they weren't,

7:55

perhaps. And then I guess that there's other

7:57

risks like just speeding up history, causing a

8:00

whole lot of stuff to happen incredibly quickly, and perhaps that

8:02

leading us into disaster. Yeah, which

8:04

of the aspects of this broader problem do you

8:06

think of yourself as trying to contribute to Solve right now?

8:09

Yeah, I mean, first off to your point, I mean, I

8:11

am happy to focus on solutions, but I do think

8:13

it can be hard to have a really good conversation on solutions

8:16

without having some kind of shared understanding of the problem.

8:18

And I think while a lot of people are getting

8:20

vaguely scared about AI, I

8:23

think there's still a lot of room to have, you

8:26

know, a lot of room to disagree on what exactly

8:28

the most important aspects of the problem are, what

8:30

exactly the biggest risks are. For

8:32

me, the two you named, misalignment and misuse,

8:35

are definitely big. I would throw some others

8:37

in there too that I think are also big.

8:39

I think, you know, we may

8:41

be on the cusp of having a lot

8:43

of things work really differently about the world, and

8:46

in particular having kind of what you might

8:48

think of as new life forms, whether that's AIs

8:50

or, you know, I've written in the past on cold

8:52

takes about digital people, that if we had

8:54

the right technology, which we might be able to develop with AIs

8:57

help, we might have kind of, you know, simulations

8:59

of humans that we ought to think of as kind of humans

9:02

like us. And that could lead to a lot

9:04

of challenges, you know, just the

9:06

fact, for example, that you could have human

9:08

rights abuses happening inside a computer. It

9:11

seems like a very strange situation that

9:13

society has not really dealt with before.

9:15

And I think there's a bunch of other things like that. What

9:18

kind of world do we have when someone can just make

9:21

copies of

9:22

people or of minds and

9:24

ensure that those copies believe certain things and

9:27

defend certain ideas that I think

9:29

could challenge the way a lot of our existing institutions

9:31

work. So there's a nice piece,

9:33

Propositions About Digital Minds, that I think is a flavor

9:35

for this. So I think there's a whole bunch of things I

9:38

would point to as important. I think out

9:41

of, you know, in this category, I think if I had

9:43

to name one risk that I'm most focused

9:45

on, it's probably the misaligned AI risk. It's probably

9:47

the one about, you know, you

9:50

kind of build these very capable, very

9:52

powerful

9:52

AI systems. They're these systems that

9:55

if, for whatever reason, they were pointed at bringing

9:57

down all of human civilization, they could. And

9:59

then something about your training is kind of

10:01

sloppy or leads to unintended

10:04

consequences so that you actually do have AIs

10:06

trying to bring down civilization. I think that

10:08

is probably the biggest one, but I think there's also

10:11

a meta threat that to

10:13

me is really the unifying catastrophic risk

10:15

of AI. And so for me that I

10:18

would abbreviate as just saying like explosively

10:20

fast progress. So the central

10:23

idea of the most important century series that I wrote

10:26

is that if you get an AI with

10:28

certain properties, there's a bunch of

10:29

reasons from economic theory, from someone

10:32

from economic history. I think we're also putting

10:34

together some reasons now that you can take more from

10:36

the specifics of how AI works and

10:38

how algorithms development works to expect

10:41

that you could get a dramatic acceleration

10:43

in the rate of change and particularly in the

10:45

rate of scientific and technological advancement,

10:48

particularly in the rate of AI advancement itself so

10:50

that things move on a much faster timescale

10:53

than anyone is used to. And one of the central

10:55

things I say in the most important century series is that if

10:57

you imagine a wacky sci-fi

10:59

future, the kind of thing you would

11:02

imagine thousands of years from now for

11:04

humanity with all these wacky technologies, that

11:06

might actually be years or

11:09

months from the time when you

11:11

get in range of these super powerful AIs

11:13

that have certain properties. That to me is

11:15

the central problem. And I think all these other risks that

11:17

we're talking about, they wouldn't

11:20

have the same edge to them if it weren't for that.

11:22

So misaligned AI, if AI

11:25

systems got very gradually more powerful

11:27

and we spent a decade with systems that

11:29

were

11:29

kind of close to as capable

11:32

as humans, but not really, and then a decade with systems

11:34

that were about as capable as humans with some strengths and

11:36

weaknesses, then a decade of systems a little more

11:38

capable, I wouldn't really be that worried.

11:40

I feel like this is something we could kind of adapt to as we

11:42

went and figure out as we went along. Similarly,

11:45

with misuse, AI systems might

11:47

end up able to help develop

11:49

powerful technologies that are very scary, but that wouldn't

11:51

be as big a deal. It would be kind of a continuation

11:53

of history if this just

11:56

went in a gradual way. And my big concern

11:58

is that it's not gradual.

11:59

Maybe we're digging on that a little bit more, is exactly

12:02

how fast do I mean and why, even though I have covered

12:04

it somewhat in the past, because that to me is really

12:06

the central issue. And one of the reasons I'm so

12:08

interested in AI safety standards is because

12:11

it is kind of, no matter what risk

12:13

you're worried about, I think you hopefully

12:15

should be able to get on board with the idea that you should measure

12:18

the risk and not unwittingly

12:21

deploy AI systems that are carrying a ton of the

12:23

risk before you've at least made a deliberate, informed

12:25

decision to do so. And I think if

12:28

we do that, we can anticipate a

12:29

lot of different risks and stop them from

12:32

coming at us too fast. Too fast is the central

12:34

theme for me.

12:35

Yeah. Yeah, it's very interesting framing

12:37

to put the speed of advancement like front

12:39

and center as this is kind of the key way that this

12:41

could go off the rails and in all sorts of different directions.

12:44

So, Eliezer Jutkowski has this kind of classic story

12:46

about how you get an AI taking over the

12:48

world like remarkably quickly. And

12:50

a key part of the story as he tells it is this

12:52

sudden self-improvement loop where the AI

12:55

gets better at doing AI research and that improves itself

12:57

and then it's better at doing that again. And so you get

12:59

this recursive loop where suddenly you go from

13:01

somewhat human level intelligence to something that's very,

13:04

very, very superhuman. And

13:05

I think many people

13:06

reject that primarily because they reject the

13:09

speed idea that they think, yes, if you got

13:11

that level of advancement over a period of days, sure,

13:13

that might happen. But actually, I just don't expect

13:15

that recursive loop to be quite so

13:17

quick. And likewise, if we might

13:19

worry that AI might be used by people to

13:22

make bioweapons, but if that's something that gradually came

13:24

online over a period of decades, we probably have all kinds of responses

13:26

that we could use to try to prevent that. But if it goes

13:28

from one week to the next, then we're in

13:30

a tricky spot. Do you want to expand on that? Is there maybe

13:33

insights that come out of this speed-focused

13:35

framing of the problem that people aren't taking

13:38

quite seriously enough?

13:39

Yeah, I should first say I don't know that I'm on the

13:41

same page as Eliezer. I can't totally always tell,

13:43

but I think he is picturing probably a more

13:46

extreme and faster thing that I'm picturing and

13:48

probably for somewhat different reasons. I think

13:50

a common story in some corners of this

13:53

discourse is this idea of an AI that

13:56

it's this simple computer program and it rewrites

13:58

its own source code. And it's like the,

14:00

you know, that's where all the action is. I don't think

14:03

that's exactly the picture I have in mind, although there's

14:05

some similarities. And so that, you know, the

14:07

kind of thing I'm picturing is maybe more like a months

14:09

or years time period from getting sort

14:12

of near human level AI

14:14

systems and what that means is definitely debatable

14:16

and gets messy, but near human level

14:19

AI systems to just like very, very powerful

14:21

ones that are advancing science and technology

14:24

really fast. And then science and technology,

14:26

like at least on certain fronts that are not, that

14:29

are the less bottlenecked

14:29

fronts, and we could talk about bottlenecks in a minute,

14:32

you get like a huge jump. So I

14:34

think my view is at least somewhat more moderate

14:36

than Eliezer's and at least has somewhat different dynamics.

14:39

But I think there is, you know, both

14:42

points of view are talking about this rapid change. I

14:44

think without the rapid change,

14:46

A, things are a lot less scary generally. B,

14:48

I think it is harder to justify a lot

14:51

of the stuff that AI concerned people do to try and

14:53

get out ahead of the problem and think about things in advance,

14:55

because I think a lot of people sort of complain

14:57

with this discourse that it's really hard to know the

14:59

future and all this stuff we're talking about what future

15:02

AI systems are going to do, what we have to do about

15:04

it today. It's very hard to get that right. It's

15:06

very hard to anticipate what things will be like in

15:08

an unfamiliar future. And I think when people

15:10

complain about that stuff, I'm just like very sympathetic.

15:12

I think that's like,

15:13

right. And if I thought that

15:15

we had the option to adapt to everything

15:18

as it happens, I think I would in many

15:20

ways be tempted to just work on other problems

15:22

and then kind of in fact adapt to things as they happen

15:24

and see what's happening and see what's most needed. And

15:27

so I think a lot of the case for planning

15:29

things out in advance, trying to tell stories of

15:32

what might happen, trying to figure out

15:34

what kind of regime we're going to want and put the pieces

15:36

in place today, trying to figure out what kind of research

15:38

challenge is going to be hard and doing today. I think a

15:40

lot of the case for that stuff being so important does

15:43

rely on this theory that things could move

15:45

a lot faster than anyone is expecting. I

15:48

am in fact very sympathetic to people who would rather

15:50

just adapt to things as they go. They're

15:52

the right way to do things. And I think

15:55

many attempts to anticipate future problems

15:57

or things I'm just not that interested in because

15:59

of this issue.

15:59

But I think AI is a place where we have to

16:02

take the explosive progress things seriously

16:04

enough that we should be doing our best to prepare for it. Yeah,

16:07

I guess if you have this explosive growth, then

16:10

the very strange things that we might be

16:12

trying to prepare for might be happening in 2027 or incredibly soon. Something

16:16

like that, yeah. Yeah, it's imaginable, right?

16:18

And it's all extremely uncertain

16:20

because we don't know. In my

16:22

head, a lot of it is like there's a set

16:24

of properties that an AI system could have, roughly

16:27

being able to do roughly everything humans are

16:29

able to do to advance science and technology or

16:31

at least able to advance AI research. We

16:33

don't know when we'll have that. And so it's like, you

16:35

know, one possibility

16:36

is we're like 30 years away from that.

16:38

But once we get near that, things will

16:41

move incredibly fast. And that's a world

16:43

we could be in. We could also be in a world where we're only a few years from

16:45

that, and then everything's going to get much crazier than anyone

16:47

thinks, much faster than anyone thinks. Yeah,

16:49

I guess one narrative is that

16:51

it's going to be exceedingly difficult to

16:54

align any artificial intelligence because you

16:56

know, you have to solve these 10 technical problems

16:58

that we've almost gotten no traction on so far.

17:01

So just from, you know, it gets decades

17:03

or centuries in order to fix them.

17:05

On this speed focused narrative,

17:07

it actually seems a little bit more positive because you

17:09

might be saying,

17:10

it might turn out that from a technical standpoint,

17:12

this isn't that challenging. The problem will be

17:15

that things are going to run so quickly that we might only

17:17

have a few months to figure out how, like

17:20

what solution we're choosing and actually try

17:22

to apply it in practice. But of course, in

17:24

as much as we just need to slow down, that

17:26

is something that in theory, at least people could agree

17:28

and actually just and try to coordinate

17:31

in order to do. Do you think that that is going to be a

17:33

part of the package that we ideally just want to

17:35

coordinate people as much as possible to make this

17:37

as gradual as feasible? Well,

17:39

these are separate points. So I think

17:41

you could believe in the speed and also

17:43

believe the alignment problems really hard. Believing

17:46

in speed doesn't make the alignment problem any easier.

17:48

And I think that the speed point is really just

17:50

bad news. I think it's just, you know, I

17:52

hope things don't move that fast. If

17:55

things move that fast, I think most human

17:57

institutions ways of reacting to things just we

17:59

can't count on that. them to work the way they normally do and

18:01

so we're gonna have to do our best to get out ahead of things

18:03

and and plan ahead and make things better in

18:05

advance as much as we can and it's mostly just

18:08

bad news. There's a separate thing which is that yeah I

18:10

do I am less of a I

18:12

do I do feel less convinced

18:14

than some other people that the alignment

18:16

problem is this like incredibly hard technical problem

18:19

and more feel like yeah if we did have

18:21

a relatively gradual set of developments I think

18:23

we'd have good I think even with a very

18:25

fast developments I think there's a good chance we just get lucky

18:27

and we're fine. So I think they're two different

18:30

axes. I know you talked with Tom Davidson about

18:32

this a bunch so I don't want to make it like the main theme of the

18:34

episode but I do think like in case someone

18:36

hasn't listened every ADK podcast ever just

18:38

just getting a little more into the why of

18:41

why you get such an explosive growth and why not

18:43

I think this is a really key premise and

18:45

I think right most of the rest of what I'm saying doesn't

18:47

make much sense without it and I want to own that

18:50

yeah I don't want to lose out on the fact that I

18:52

am sympathetic to a lot of reservations about

18:54

working on AI risk so yeah maybe

18:56

it would be good to cover that a bit.

18:58

Yeah let's let's do that so one

19:00

obvious mechanism by which things could speed up is that you have

19:02

this positive feedback loop where the AI's get better

19:04

at improving themselves is there is there much more to the

19:06

to the story than that?

19:09

Yeah I mean I think it's worth recheving

19:11

that briefly I mean I think one observation

19:13

I think is interesting and this is a you know report by

19:15

David Rudman for Open Philanthropy that goes through this

19:18

one thing that I've wondered is just like if you take

19:20

the path of world economic growth throughout

19:22

history and you just kind of extrapolated

19:24

forward in the simplest way you can what do you get

19:27

and it's like well it depends what time period you're looking

19:29

at if you look at economic history since 1900 or 1950 we've had

19:31

a few percent of your growth over that entire

19:34

time and if you extrapolate it forward you get a few

19:36

percent of your growth and you

19:38

just get the world everyone is kind of already expecting and

19:40

the world that's in the UN projections all that stuff the

19:43

interesting thing is if you zoom out and

19:45

look at all of economic history you

19:47

see that economic progress for most of

19:49

history not not recently has been accelerating

19:53

and if you try to model that acceleration in a

19:55

simple way and project that out in a simple way you

19:57

get basically the economy going to infinite

19:59

size sometimes this century, which is like a wild

20:03

thing to get from a simple extrapolation.

20:06

I think the question is like, why is that and why might

20:08

it not be? I think the

20:10

basic framework I have in mind is like, there is

20:13

a feedback loop you can have in the economy where

20:15

people have new ideas, new

20:18

innovations that makes them more productive. Once

20:20

they're more productive, they have more resources. Once

20:22

they have more resources, they have more kids.

20:24

There's more population or there's fewer deaths and there's more

20:27

people. So it goes more people,

20:29

more ideas, more resources, more

20:31

people, more ideas, more resources. When

20:34

you have that kind of feedback loop in place, any economic

20:36

theory will tell you that you get what's called super

20:38

exponential growth, which is growth that's accelerating.

20:41

It's accelerating on an exponential basis and

20:43

that kind of growth is very explosive,

20:46

is very hard to track and can go to infinity in finite

20:48

time. The thing that changed a couple hundred

20:50

years ago is that one piece of that feedback loop

20:53

stopped for humanity. People

20:55

basically stopped turning more resources into more

20:57

people. So right now, when people get richer,

20:59

they don't have more kids, they just get

21:01

richer. Buy another car. Yeah, exactly. And

21:04

so that feedback loop kind of broke. That's

21:07

not like a bad thing that it broke, I don't think, but it

21:09

kind of broke. And so we've had

21:11

just like what's called normal exponential growth, which

21:13

is still fast, but which is not the same thing,

21:15

doesn't have the same explosiveness to it. And

21:18

the thing that I think is interesting and different about

21:20

AI is that if you get

21:23

AI to the point where it's doing the same thing

21:25

humans do to have new ideas, to

21:27

improve productivity, so this is like the science

21:29

and invention part, then you can

21:31

turn resources into AIs

21:34

in this very simple linear way that

21:36

you can't do with humans. And so you

21:38

could get an AI feedback loop and

21:41

just to be a little more specific about what it might look like,

21:43

right now AI systems

21:46

are getting a lot more efficient. You can do a lot more with

21:48

the same amount of compute than you could 10 years ago, actually

21:51

a dramatic amount more. I think something,

21:53

various measurements of this or something like you can get the same

21:55

performance for something like 18X or 15X

21:58

less compute compared to like a few years

21:59

ago, maybe a decade ago.

22:01

Why is that? And it's because there's a bunch of human

22:03

beings who have worked on making AI algorithms

22:06

more efficient. So to me, the big scary

22:08

thing is when you have an AI that does

22:11

whatever those human beings were doing. And there's

22:13

no particular reason you couldn't have that because what those human

22:15

beings were doing, as far as I know, was mostly kind

22:17

of like sitting at computers, thinking of stuff,

22:19

trying stuff. There's no particular reason

22:21

you couldn't automate that. Once you automate that,

22:23

here's the scary thing. You have

22:25

a bunch of AIs. You use those AIs

22:28

to come up with new ideas to make your AIs more efficient.

22:30

Then let's say that you make your AIs

22:33

twice as efficient. Well, now you have twice as many AIs.

22:36

And so if having twice as many AIs can make your

22:38

AIs twice as efficient, again, there's really no

22:40

telling where that ends. And Tom Davidson

22:42

did a bunch of analysis of this, and I'm still

22:44

kind of poking at and thinking about it, but I think there's at least a decent

22:46

chance that that is the kind of thing that leads to

22:49

explosive progress where AI

22:51

could really take off and get very capable, very

22:53

fast. And you can extend that somewhat to other

22:56

areas of science. And it's like, some

22:59

of this will be bottlenecked. Some of this will be like, you

23:01

can only move so fast because you have to do a bunch of experiments

23:04

in the real world. You have to build a bunch of stuff. And

23:06

I think some of it will only be a little bottlenecked or will only

23:08

be somewhat bottlenecked. And I think there are some feedback loops

23:11

just kind of going from you get more

23:13

money, you're able to kind of quickly

23:16

with automated factories build more stuff

23:18

like solar panels, you get more energy,

23:20

and then you get more money, and then you're able

23:22

to do that again. And it's like in that loop, you

23:24

have this part where you're making everything more efficient all the time.

23:26

And I'm not going into all the details

23:29

here. It's been gone into more detail in my blog

23:31

post, The Most Important Century and Tom Davidson in

23:33

his podcast and continues to think about it. But

23:37

that's the basic model is that you

23:39

have this feedback loop that we have observed in history that

23:41

doesn't work for humans right now, but could work for AIs,

23:44

where you have AIs, have ideas

23:47

in some sense, make things more efficient. When

23:49

things get more efficient, you have more AIs. That creates

23:51

a feedback loop. That's where you get your super exponential growth.

23:54

Yeah, so one way of describing

23:56

this is talking about you get the

23:58

artificial intelligence becomes more

24:00

intelligent and that makes it more capable of improving

24:03

its intelligence and so it becomes super, super smart.

24:05

But I guess the way that you're telling it emphasizes

24:08

a different aspect, which is not so much that it's becoming

24:10

super smart, but that it is becoming super numerous

24:12

or that you can get effectively a population explosion.

24:15

And I think some people are skeptical of this super

24:17

intelligent story because they think you get really declining

24:19

returns to being smarter and that there's like some ways

24:21

in which it just doesn't matter how smart you

24:23

are, the world's too unpredictable, say, for you

24:26

to come up with a great plan. But

24:28

this is a different way by which you can get the same outcome,

24:30

which is just that you

24:32

have this enormous increase in the number of thoughts

24:35

that are occurring on computer chips,

24:37

more or less. And at some point, you know, 99% of

24:39

the thoughts that are happening on Earth could basically be

24:42

happening, be occurring inside artificial intelligences.

24:45

And then as they get better and they're able to make more chips

24:47

more quickly, again, you basically just

24:49

get the population explosion.

24:51

Yeah, that's exactly right. And I think this is

24:53

a place where I think some people get a little bit rabbit holed

24:55

on the AI debates because I think there's

24:58

a lot of room to debate how big

25:00

a deal it is to have something that's quote unquote extremely

25:03

smart or super intelligent or much smarter than human. And

25:06

it's like, OK, maybe if you had like something

25:08

that was kind of like a giant brain or something and

25:10

way smarter, whatever that means, than us,

25:13

maybe what that would mean is that it would like instantly see

25:15

how to make all these super weapons and conquer

25:17

the world and how to convince us of anything. And there's

25:19

all this stuff that that could mean and people debate

25:21

whether it could mean that. But it's uncertain.

25:24

It's a lot less uncertain if you're finding yourself skeptical

25:27

of what this smart idea means

25:29

and where it's going to go and what you can do with it. If you

25:31

find yourself skeptical of that, then just forget about it. And

25:33

just I believe you can make the entire

25:36

case for being extremely concerned about AI,

25:38

assuming that AI will never be smarter

25:40

than a human. Instead it will be as

25:43

capable as the most capable humans. And

25:45

there will be a ton of them because unlike

25:47

humans, you can just copy them. You

25:50

can copy them. You can use your copies to

25:52

come up with ways to make it more efficient,

25:54

just like humans do. Then you can

25:56

make more copies. And when we talk about whether

25:59

AI could defeat humanity. and I've written one blog post

26:01

on whether AI can kind of like take over the world, they

26:03

don't have to be more capable than humans.

26:06

They could be equally capable and there could be more of

26:08

them. That could really do it. That could really

26:10

be enough that then we wouldn't be,

26:12

humans wouldn't be in control of the world anymore. So

26:15

I'm basically generally happy

26:17

to just have all discussions about AI and

26:19

what the risks are just in this world where

26:21

like there's nothing more capable than a human, but

26:23

it's pretty scary to have a lot of those that have

26:25

different values from humans that are kind of a second advanced

26:28

species. That's not to rule out that

26:30

some of these super intelligence concerns could be real. It's

26:32

just like they're not always necessary and they can

26:34

sideline people.

26:36

Yeah, you can just get beaten by force

26:38

of numbers more or less. I think it's a little bit of a shame

26:40

that this sheer numbers argument

26:42

hasn't really been made very much. It

26:44

feels like the super intelligence story has been

26:46

very dominant in the narrative and media

26:49

and yet many people get off the boat because they're

26:51

skeptical of this intelligence thing. I think

26:54

it kind of is the fault of me and maybe people who've

26:56

been trying to raise the alarm about this because the focus really

26:58

has been on the super intelligence aspect rather than the super

27:01

numerousness that you could

27:03

get. Yeah, I don't know. I mean, I think there's valid concerns

27:05

like from

27:06

that angle for sure. And I'm not trying to

27:08

dismiss it, but I do. I

27:10

think there's a lot of uncertainty about what

27:12

super intelligence means and where it could go. And

27:14

I think you can raise a lot of these concerns without needing to have

27:16

a subtle view there. So

27:19

yeah, with the Jaya Kocher and Rowan Shah,

27:21

I found it really instructive to hear from them

27:24

about what are some kind of common opinions that

27:26

they don't share or maybe even just regard

27:28

as misunderstandings. Yeah, so maybe

27:30

let's go through a couple of those to help maybe situate you in the space

27:33

of ideas here. What's a common opinion

27:35

among kind of the community

27:36

of people working to address

27:38

AI risk that you personally don't share?

27:40

Yeah, I mean, I don't know. I think one kind of vibe I

27:42

pick up and I don't always have the exact quote

27:45

of whoever said what, but a vibe I pick up is this

27:48

kind of framework that kind of says, you know, if we

27:50

don't align our AIs, we're all going

27:52

to die. And if we can align our AIs,

27:54

that's great. And we've solved the problem. And that's

27:56

the problem we should be thinking about. And there's nothing

27:58

else really worth worrying about. You

28:00

know, it's kind of like alignment is the whole game would be

28:02

the hypothesis and I Disagree

28:04

with with both ends of that, but especially the

28:06

latter so to take the first end would be

28:09

like, you know If we if we don't align AI

28:11

we're all dead I mean first off I just think

28:13

it's like really Unclear even in the

28:15

even in the worst case where you get an AI that

28:17

has like its own values And there's

28:19

a huge number of them and they kind of team up and take

28:21

over the world Even then it's like really unclear

28:24

if that means we all die I think there's like I

28:26

know there's debates about this I have I have tried

28:28

to understand that I know that the the muri

28:31

folks I think feel really strongly clearly we all die

28:33

I've tried to understand where they're coming from and I have not

28:36

I think a key point is it just you know Could

28:38

be very very cheap as a percentage

28:40

of resources for example to

28:43

let humans have a nice life on earth and

28:45

not expand further and and be cut off

28:47

in certain ways from Threatening, you know AI's

28:49

ability to do what it wants that could be very

28:51

cheap compared to wiping us all out And there could

28:53

be a bunch of reasons one might want to do

28:55

that Some of them kind of wacky some

28:57

of them kind of you know, well, maybe You

29:00

know maybe in another part of the universe There's kind

29:03

of someone like the AI that

29:05

was trying to design its own AI

29:07

and that thing ended up with values like the humans

29:09

And you know, maybe there's some kind of trade that could be

29:11

made using like a causal trade And we don't need

29:14

to get into what all this means But it's like you don't

29:16

need much the thing is you don't need or like

29:18

maybe the AI is actually being simulated by humans Or

29:20

something or by some smarter version of humans or some more

29:22

powerful version of humans and being tested

29:24

to see if it'll wipe Out the humans would be nice to them It's like you

29:27

don't need a lot of reasons, you know to

29:29

kind of like leave one planet out if you're kind

29:31

of expanding throughout the galaxy So that would

29:33

be one thing is it just like I don't know It's like kind of uncertain

29:35

what happens even in the worst case and then

29:37

there's like I do think there's a bunch of in-between

29:40

cases where we kind of have a eyes

29:42

that are like there they're

29:44

sort of aligned with humans like if you if you

29:46

think about a analogy that often comes

29:48

up is like humans and natural selection where Humans

29:51

kind of were put under pressure by natural selection

29:54

to have lots of kids or to you know Do inclusive

29:56

reproductive fitness and we've

29:58

kind of okay. We invented birth control and a

30:00

lot of times we don't have as many kids as

30:03

we could and stuff like that. But also humans still

30:05

have kids and love having kids. A lot of humans

30:07

have 20 different reasons to have kids and

30:10

after a lot of the original ones have been knocked out

30:12

by weird technologies, they still find some other reason to have kids.

30:15

I don't know, I found myself one day wanting

30:17

kids and had no idea why and invented

30:20

all these weird reasons. I don't know,

30:23

it's not that odd to think that you could

30:25

have AI systems that just kind of like, yeah,

30:27

they're pretty off kilter from what we were trying to make

30:30

them do, but it's not like they're doing something completely

30:32

unrelated either. It's not like they have no drives

30:34

to do a bunch of stuff related to the stuff we wanted them

30:36

to do. Then you could also just have situations

30:39

where, especially in

30:41

the early stages of all this, where you might have

30:43

kind of near human level AIs and

30:45

so they might have goals of their own but they might not

30:48

be able to coordinate very well or they might not be able

30:50

to reliably overcome humans so

30:52

they might end up cooperating with humans a lot. We might

30:54

be able to leverage that into kind of

30:57

having AI allies that help us build

30:59

other AI allies that are more powerful so we might

31:01

be able to stay in the game for a long way. I don't know,

31:03

I just think things could be very complicated.

31:05

It doesn't feel to be like if you

31:07

screw up a little bit with the alignment problem then we all die.

31:11

The other part, if we do align

31:13

the AI, we're fine. I disagree with much more

31:15

strongly. I just think- All right.

31:18

More strongly than that. Okay, yeah, yeah, yeah. Go

31:20

for it. The first one, I mean, look, I think it would

31:22

be really bad to have misaligned AI and

31:24

I think despite the feeling that I feel it is fairly

31:27

overrated in some circles, I still think it's like the

31:30

number one thing for me. Just

31:32

like the single biggest issue in AI is

31:34

just like we're building these potentially

31:37

very powerful, very replicable, very numerous

31:39

systems and we're building them in ways we don't

31:41

have much insight into whether

31:43

they have goals, what the goals would be. We're

31:46

kind of introducing the second advanced species onto

31:48

the planet that we don't understand and if that

31:50

advanced species becomes more numerous and or more

31:52

capable than us, we don't have a great

31:54

argument to think that's going to be good for us. I'm

31:57

on board with alignment risk.

31:59

I don't know.

31:59

thing, not the only thing, the number one thing. But

32:02

I would say, if you just assume that

32:04

you have a world of very capable

32:07

AIs that are doing exactly what humans

32:09

want them to do, yeah, that's very

32:11

scary. And I think if that was the world we knew we were going

32:13

to be in, I would still be totally full-time on AI

32:16

and still feel that we had so much work to do and we

32:18

were so not ready for what was coming.

32:20

Certainly, there's the fact that because

32:23

of the speed at which things move, you could

32:25

end up with whoever kind of leads the way

32:27

on AI or is least cautious having

32:30

a lot of power. And that could be someone really bad.

32:33

And I don't think we should assume that just because

32:35

that if you had some head of state

32:37

that has really bad values, I don't think we

32:39

should assume that that person is going to end up being nice

32:42

after they become wealthy or powerful

32:44

or transhuman or mind uploaded

32:46

or whatever. I don't think there's really any reason

32:48

to think we should assume that. And then I think

32:50

there's just a bunch of other things that if things are moving fast,

32:53

we could end up in a really bad state. Like, are

32:55

we going to come up with decent frameworks

32:58

for making sure that digital

33:01

minds are not mistreated? Are we going to come up with

33:03

decent frameworks for kind of like how

33:06

to ensure that as we get the ability to

33:08

create whatever minds we want, we're using that

33:10

to create minds and help us seek the truth

33:12

instead of create minds that have whatever beliefs

33:14

we want them to have stick to those beliefs

33:17

and try to shape the world around those beliefs. I

33:19

think Carl Schulman put it as are

33:21

we going to have AI that makes us wiser

33:23

or more powerfully insane? So

33:25

I think there's just a lot. I

33:27

think we're kind of on the cusp of something that

33:30

is just potentially really big, really

33:32

world changing, really transformative and going to

33:34

move way too fast. And I think even

33:36

if we threw out the misalignment problem, we'd have a lot of work

33:38

to do. And I think a lot of these issues are actually not getting

33:40

enough attention.

33:41

Yeah. I think some of that might be going on there

33:44

as a bit of equivocation in the

33:46

word alignment. So you can imagine some

33:48

people might mean by creating an aligned

33:50

AI, it's like an AI that kind of goes and does what you tell

33:52

it to like a good employee or something. Whereas

33:54

other people mean it is following

33:56

the correct ideal values and behaviors and

33:58

is going to work to generate. generate the best outcome.

34:01

And these are really quite separate

34:03

things, although very far apart. Yeah.

34:06

Well, the second one, I don't

34:08

even know if that's a thing. I don't even

34:10

really know what it's supposed to be. I mean, there's something a little

34:12

bit in between, which is like,

34:14

you can have an AI that you ask

34:16

it to do something, and it does what you would

34:18

have told it to do if you had been more informed

34:20

and if you knew everything it knows. That's the

34:23

central idea of alignment that I tend to think of, but

34:25

I think that still has all the problems I'm talking about. Some

34:28

humans seriously do intend

34:31

to do things that are really nasty and seriously

34:33

do not intend in any way, even if they knew more,

34:35

to make the world as nice as we would like it to be. And

34:38

some humans really do intend and

34:40

really do mean and really will want to

34:42

say, right now I have these values. Let's

34:44

say this is the religion I follow. This is what I

34:46

believe in. This is what I care about. And I am creating

34:49

an AI to help me promote that religion, not to help me question

34:51

it or revise it or make it better. So yeah, I think it's

34:53

that middle one, I think it does not make it safe. There

34:55

might be some extreme version that's like,

34:57

an AI that just figures out what's objectively best

34:59

for the world and does that or something. And I'm just

35:01

like, I don't know why I would, I don't know why you would think that would even be a

35:03

thing to aim for. That's not the alignment problem that I'm interested

35:06

in having solved, yeah. Yeah. Okay,

35:08

what's something that some kind of safety

35:11

focused folks that you potentially collaborate

35:13

with or at least talk to, but they think

35:15

that they know, which you think in fact, we

35:18

just, nobody knows.

35:20

Yeah, I mean, I think in general, there's

35:23

this kind of question in deep

35:25

learning of, you train an agent

35:27

on one distribution of data or

35:30

reward signals or whatever. And now you're

35:32

wondering when it goes out of distribution,

35:34

when it hits a new kind of environment

35:36

or a new set of data, how it's gonna react to that. So

35:38

this would be like, how does an AI generalize

35:41

from training to out of distribution?

35:44

And I think in general, people have

35:47

a lot of trouble understanding this and

35:49

have a lot of trouble predicting this. And I think that's not

35:51

controversial. I think that's known, but I think

35:53

it kind of comes down to, or it

35:55

relates to some things where people do seem overly

35:57

confident. A lot of what people are doing right

35:59

now. with these AI models is they're doing what's called reinforcement

36:02

learning on human feedback or from human

36:04

feedback, where the basic idea is you

36:06

have an AI that tries something and then

36:08

a human says, that was great or that wasn't so

36:10

good, or maybe the AI tries two

36:13

things, the human says which one was better. And

36:16

if you do that, and that's a major way

36:18

you're training your AI system, there's

36:20

this question of what do you get as a result

36:23

of that? Do you get an AI system

36:25

that is actually doing what humans want

36:28

it to do? Do you get an AI system that's doing what humans

36:30

would want it to do if they kind of knew all the facts?

36:33

Do you get an AI system that is like tricking

36:35

humans into thinking it did what they wanted it

36:37

to do? Do you get an AI system that's sort of trying

36:39

to maximize paperclips? And one way to do that is to

36:41

do what humans wanted to do. So as far as you can tell, it's

36:43

doing what you want it to do, but it's actually trying to maximize paperclips.

36:46

Like, which of those do you get? And I think just like

36:48

people don't, like we don't know, and

36:50

I see overconfidence on both sides

36:52

of this. I think I see people saying, we're

36:54

going to basically train this thing to

36:56

do nice things and it'll keep doing nice things as

36:59

it operates in an increasingly changing world. And

37:01

then I see people saying, we're going to train AI to do nice

37:03

things. And it will basically pick up on

37:05

some weird correlate of our training

37:08

and try to maximize that and will

37:10

not actually be nice. And I'm just like, geez,

37:13

we don't know. We don't know. And

37:15

there's arguments that say, oh, wouldn't it be weird if it

37:17

came out doing exactly what we wanted it to do? Because

37:19

there's this wide space of other things

37:21

that could generalize to it. I just think those arguments

37:23

are just kind of weak and they're not very well fleshed out. There's

37:26

genuinely just a ton of vagueness

37:28

and not good understanding of what's

37:31

going on in a neural net and how

37:33

it generalizes from this kind of training. So

37:35

the upshot of that is I think people are

37:38

often just overconfident that AI

37:40

alignment is going to be easier hard. I think there's people

37:42

who think, we basically got the right framework, we

37:44

ought to debug it. And there's people who

37:46

think this framework is doomed, it's not going to work, we need something better. And

37:49

I just don't think either is right. I think if

37:51

we just go on the default course and we

37:53

just kind of train AIs based

37:55

on what looks nice to us, that could totally

37:57

go fine and it could totally go disastrously.

37:59

No, weirdly few people who

38:02

believe both those things, a lot of people

38:04

seem to be overconfidently in one camp or

38:06

the other on that. Yeah, I'm completely with you

38:08

on this one. I think it's one of the things that I've started to

38:10

believe more over the last six months. No

38:13

one really has super strong arguments for what kind

38:15

of motivational architecture these ML

38:18

systems are going to develop. Yeah. Well,

38:20

I suppose that's maybe an improvement relative to where it was before, because

38:22

I was a little bit more on the Duma side before.

38:26

I feel like this one, there should be some empirical way

38:28

of investigating this. People do have

38:30

good points that a really super intelligent

38:33

and incredibly sneaky model will behave exactly

38:35

the same regardless of its underlying

38:37

motive. But you could try to investigate this on less

38:39

intelligent, less crafty models, and surely you would

38:42

be able to get some insight into the way it's thinking

38:44

and how its motives actually cash

38:46

out. I

38:47

think it's really hard. It's

38:50

just like really, I mean, it's really

38:52

hard to make generalized statements about how

38:55

an AI generalizes to weird new

38:57

situations. And

38:59

yeah, there is work going on trying

39:01

to understand this, but it's going to be, it's just

39:04

been hard to get anything that feels satisfyingly analogous

39:06

to the problem we care about right now with AI

39:08

systems and their current capabilities. And even once

39:10

we do, I think there'll be plenty of arguments that are just like,

39:12

well, once the AI systems are more capable than that, everything's

39:15

going to change. And AI will generalize

39:17

differently when it understands who we

39:19

are and what our training is and how that works

39:21

and how the world works. And it understands

39:23

that it could take over the world if it wanted to. That

39:26

actually could cause an AI to generalize differently.

39:28

So as an example, this is something

39:30

I've written about on ColdTakes. I call it the King

39:33

Lear problem. So King Lear is a Shakespearean

39:35

character who kind of has three daughters

39:37

and he asks them each to describe their love for him. And

39:40

then he kind of like hands the kingdom over to the ones

39:42

that he feels good about after hearing their speeches and

39:44

he just picks wrong. And then that's too bad

39:47

for him. And the issue is

39:50

it's like it flips on a dime. It's like the two

39:52

daughters who are like the more evil ones were

39:54

doing a much better job pretending they loved him

39:56

a ton because they knew that they

39:58

didn't have power yet. power later. So

40:00

it actually like their behavior depended

40:03

on their calculation of what was going to happen. And

40:05

so the analogy to AIs, it's kind of like you

40:07

might have an AI system that's like kind of maybe

40:10

what its motivational system is, is it's

40:12

trying to maximize

40:14

the number of humans that are saying, hey, good job,

40:17

this obviously bit of a simplification or dramatization.

40:20

And it kind of is understanding at all

40:22

points, that if it could

40:25

take over the world, enslave all the humans,

40:27

make a bunch of clones of them and like run

40:29

them all in loops saying good job. If it could,

40:32

then it would and it should. But

40:34

if it can't, then maybe it should just cooperate

40:36

with us and be nice. You can have an AI system

40:38

that's like running that whole calc and humans often run

40:40

that whole calc like right as a kid

40:43

in school, I might often be thinking, well, you

40:45

know, if I can get away with this, then this is

40:47

what I want to do. If I can't get away with this, maybe

40:49

I'll just do what the teacher wants me to do. So you could

40:51

have the eyes with that whole motivational system. And then it's

40:53

like, cool. So now it's like you put them in a test

40:55

environment and you test if they're going to be nice

40:58

or try and take over the world. But in the test

41:00

environment, they can't take over the world, so they're going to be nice.

41:03

Now you're like, great, this thing is safe. You put it out in the world.

41:05

Now there's a zillion of it running around. Well, now it can take over the

41:07

world. So now it's going to behave differently. So you can have

41:10

just like one consistent motivational system

41:12

that is fiendishly hard to do a test

41:15

of how that system generalizes when it has power because

41:17

you can't generalize. You can't test what happens

41:20

when it's no longer a test.

41:22

What's the view that's common among ML researchers,

41:24

which you would you disagree with?

41:26

You know, it depends a little bit with ML researchers

41:28

for sure. I would definitely

41:31

say that I've been a big, bitter lesson

41:33

person since at least 2017. And,

41:36

you know, I got a lot of this from just like

41:38

Dario Amade, my wife's brother

41:40

who is CEO of Anthropic and I think

41:43

has been like very insightful. A lot

41:45

of what's gone on in the AI over the last few years is just

41:47

like bigger models,

41:49

more data, more training. And there's

41:53

an essay called The Bitter Lesson by an ML researcher,

41:55

Rich Sutton, that just kind of says, you know, ML

41:57

researchers keep coming up with clever and clever ways

42:00

to design AI systems and then those

42:03

cleverness is keep getting obsolete it by

42:05

just making the things bigger and just

42:07

like training them more and putting in more

42:09

data and so you know I've had a lot

42:11

of arguments over the last few years and

42:13

you know in general I have heard people arguing with

42:16

each other that are just kind of like on one

42:18

side it's like well today's AI systems

42:20

can do some cool things but they'll never be able to do this

42:22

and to do this like maybe that's reasoning

42:25

creativity you know something like that we're

42:27

gonna need a whole new approach to AI and

42:29

then the other side will say no I think we just need to make them

42:31

bigger and then they'll be able to do this I tend

42:35

to almost entirely toward that toward

42:37

that just make a bigger view I think just at

42:39

least in the limit if you if you took

42:42

an AI system and made it really big you

42:44

might need to make some tweaks but the tweaks wouldn't

42:46

necessarily be like really hard or

42:48

require giant conceptual breakthroughs I

42:50

do tend to think that that whatever it is

42:52

humans can do we could probably eventually

42:54

get an AI to do it and eventually it's not gonna

42:56

be a very fancy AI it could be just like a

42:59

very simple AI with some easy

43:01

to articulate stuff and a lot of the challenge come

43:03

from making it really big putting in a lot of data I think

43:06

this view has become like more popular over

43:08

the years than it used to be but it's still like pretty

43:11

debated I think a lot of people are still

43:13

looking at today's models and saying hey there's

43:15

fundamental limitations we're gonna need a whole new approach

43:17

to AI before they can do extra why

43:19

I'm just kind of out on that I

43:22

I think it's possible I'm not confident this is just

43:24

like where my instinct tends to lie

43:26

that's a disagreement I think another disagreement I have

43:29

with with some ML researchers I think not

43:31

all at all but but there's sometimes just

43:33

I feel like a background sense that

43:35

just like

43:36

sharing openly information

43:39

publishing

43:40

open sourcing etc it's just

43:42

like good that it's kind of it's kind of bad

43:44

to do research and keep it secret and it's good

43:46

to do research like publish it and

43:49

I you know I don't I don't feel this way

43:51

I think the things we're building could be very

43:54

dangerous at some point and I think that point

43:56

can come a lot more quickly than anyone is expecting

43:58

I think when that point some

44:00

of the open source stuff we have could

44:02

be used by bad actors in

44:05

conjunction with later insights to create

44:07

very powerful AI systems that

44:10

we aren't thinking of in ways we aren't thinking of right now,

44:12

but we won't be able to take back later. And

44:14

in general, I do tend to think that academia

44:17

has kind of this idea that sharing information

44:19

is good built into its fundamental

44:21

ethos, and that might often be true. But I think

44:23

there's times when it's kind of clearly false and academia

44:26

still kind of pushes it. You know, gain of function

44:28

research being like kind

44:29

of an example for me, where just like people are

44:32

very, very into the idea of like

44:34

making a virus more deadly and publishing

44:36

how to do it. And I think this is just an example

44:38

of where just culturally there's some

44:41

background assumptions about information

44:43

sharing that I just think the world is more complicated

44:45

than that.

44:46

Yeah, I definitely encountered people from time

44:48

to time who are, they

44:49

have this very strong prior this very

44:52

strong assumption that everything should be open and people

44:54

should have access to everything and then I'm like, what if someone

44:56

was designing a hydrogen bomb that you could make with,

44:58

you know, equipment that you could get from your house? I'm just

45:01

like, I don't think that it should be open. I think we should

45:03

probably stop them from doing that. Yeah, yeah,

45:05

yeah. And certainly if they figure

45:07

it out, we shouldn't publish it. Yeah, yeah.

45:09

I suppose it's just that that's a sufficiently rare case

45:11

that

45:12

it's very natural to develop the intuition in

45:14

favor of openness from the from the 99 out of 100 cases

45:16

where that's not too unreasonable. Yeah,

45:18

I think it's usually reasonable. But I think I think bioweapons

45:20

is just like a great counter example, or just like, it's

45:23

not really balanced. It's not really like,

45:25

well, for everyone who, you

45:27

know, who tries to design or release some horrible

45:29

pandemic, we can have someone else using open

45:31

source information to design a countermeasure. Like, that's

45:34

not actually how that works. And so, yeah, I

45:36

think this attitude at least needs to be complicated

45:38

a little bit more than it is.

45:40

Yeah. What's something that listeners might

45:42

expect you to believe, but which you actually don't? Yeah,

45:45

I don't really know what people think, I think, but some

45:48

things that sometimes I kind of pick up. I

45:50

mean, I write a lot about the future. I do a lot

45:52

about, you know, a lot of stuff about, well, as

45:55

coming, we should like prepare and do this and don't do that. I

45:57

think a lot of people think that I think I have

45:59

like this great ability to predict the future

46:02

and that I can spell it out in detail and count on it.

46:04

And I think a lot of people think I'm like underestimating

46:07

the difficulty of predicting anything. And

46:10

you know, I think I may in fact be underestimating

46:12

it, but I think I do feel a lot

46:15

of gosh, it is so hard

46:17

to, you know, be even a decade ahead or five

46:19

years ahead of what's going to happen. It

46:22

is so hard to get that right and enough detail to be helpful.

46:24

A lot of times you can get the broad outlines of something, but

46:27

to really be helpful seems really hard. Even on COVID,

46:29

it's like I feel

46:29

like a lot of the people who saw it coming in advance

46:32

weren't necessarily able to do much to make things

46:34

better. And that someone includes open philanthropy.

46:36

And we had a biosecurity program for years before

46:39

COVID. And I think there was some helpfulness

46:41

that came of it, but not as much as there could have been.

46:43

And so, you know, I think in general, I just

46:45

like, I don't know, a lot of how I am is I'm just like,

46:48

pretty the future is really hard. Getting out of the

46:50

head of the future is really hard. I'd really like rather never

46:52

do it. I think in many ways I would

46:54

just like rather work on stuff like GiveWell

46:57

and global health charities and animal welfare and just

46:59

like adapt to things as they happen

47:01

and not try and get out ahead of things. And there's

47:03

just like a small handful of issues that

47:05

I think are important enough that and may

47:07

move quickly enough that we just we just have

47:09

to do our best. I think we can I don't

47:12

think we should feel this is totally hopeless. I think I think

47:14

we can in fact do some good by

47:17

getting out ahead of things and planning in advance. And

47:20

a lot of my feelings we got to do the best we can more

47:22

than hey, I know what's coming.

47:24

Yeah. Okay. And

47:26

the other one is what's something you expect quite a lot of listeners might

47:28

believe, which which would you think you'd be

47:30

happy to disabuse them of?

47:32

Yeah, there's a line that you've

47:34

probably heard before that is

47:37

something like this. It's something like most

47:39

of the people we can help are in future generations.

47:41

And there are so many people in future generations

47:44

that that kind of just ends the conversation

47:46

about how to do the most good that it's clearly

47:48

an astronomically the case that focusing

47:51

on future generations dominates all

47:53

ethical considerations or at least dominates all

47:55

considerations of like how to do the most good

47:57

with your philanthropy or your career. I kind

47:59

of think of this is like philosophical long-termism

48:01

or philosophy first long-termism that's very you

48:04

know kind of feels like you've ended the argument after you've

48:06

pointed to the number of people in future generations

48:09

and we can get to this a bit later in the interview I don't

48:11

I don't think it's a garbage view I give some credence

48:13

to it I think it's somewhat seriously and I think it's like

48:16

underrated by the world as a whole but

48:18

I would I would say that I give a minority

48:20

of my moral parliament to thinking this way I would

48:23

say like more of me than not thinks

48:26

that's not really the right way to think about doing good

48:28

that's not really the right way to think about

48:29

ethics and I don't think we can trust these

48:32

numbers enough to feel that it's such a blowout

48:35

and the reason that I'm currently focused

48:37

on what's classically considered long-term is causes

48:39

especially AI is that I believe

48:41

the risks are imminent and real enough

48:43

that you know even with much less aggressive

48:46

valuations of the future they are you know

48:48

competitive or perhaps the best thing to work on

48:50

another random thing I think is that if you if you

48:53

really want to play the game of just like being all about

48:55

the big numbers and only thinking about the populations

48:58

that are the biggest that you can help future generations

49:00

are like extremely tiny compared

49:02

to you know persons you might

49:04

be able to help through a causal interactions with

49:06

other parts of the multiverse outside our light cone I

49:09

don't know if you want to get to that or just refer people back to

49:11

Joe's episode on that but that's that's more of a

49:13

nitpick on that take yeah people can go

49:15

back to listen to the episode with Joe Car Smith if I'd

49:17

like to understand what we just said there let's

49:20

come back to AI

49:20

now I think I want to spend quite a lot of time

49:23

basically understanding what you think different actors

49:25

should be doing to you know governments AI labs you know our

49:28

listeners so what different ways that they might be able to contribute

49:30

to improving our odds here but

49:32

maybe before we do that it might could be worth talking about like

49:34

trying to envisage scenarios in which things

49:36

go relatively relatively well you've

49:39

argued that you're very unsure how things are gonna play out

49:41

but it's possible that we might model

49:44

through and get a reasonably good outcome even

49:46

if we basically carry on doing the fairly

49:48

reckless things things that we're doing right now

49:51

not because you're recommending that we take that path right but

49:53

rather just because it's it's relevant to know whether

49:55

we're just far off like completely far off

49:57

the possibility of any good outcome given Now,

50:00

what do you see as the value of laying out positive

50:02

stories or ways that things might go well?

50:04

Yeah, so I've written a few pieces that

50:06

are kind of laying out, here's an excessively

50:09

specific story about how the

50:11

future might go that ends happily with

50:14

respect to AI, that ends with kind of, you know,

50:17

we have AIs that didn't develop or didn't

50:19

act on goals of their own enough to disempower

50:22

humanity. And then we kind of ended up with this

50:24

world where we're all, the world is getting more capable

50:26

and getting better over time and none of the various disasters

50:28

we're sketching out happened. I've written

50:30

like three different stories like that. And then one story,

50:33

the opposite, how we could stumble

50:35

into AI catastrophe where things just go

50:37

really badly. Why have I

50:39

written these stories in general? You know,

50:41

I think it's not that I believe these stories,

50:43

it's not that I think this is what's going to happen. But I

50:45

think a lot of times when you're

50:48

thinking about general principles

50:50

of like what we should be doing today to reduce risks from

50:52

AI, it is often helpful,

50:54

just like my brain works better imagining specifics.

50:57

And I think it's often helpful to kind of imagine

50:59

some specifics and then extract back from

51:02

the specifics to general points and

51:04

see if you still believe them. So for example,

51:06

I've actually done, I mean, these are the ones I've published,

51:08

but I've done a lot of thinking of what are

51:10

different ways the future can go well. And it's

51:12

like there are some themes in them. It's like there's almost

51:15

there's almost no story of the future

51:17

going well, that doesn't have like

51:19

a part that's like, and no evil

51:21

person steals the AI weights and

51:24

goes and does evil stuff. And so, you

51:26

know, it has highlighted, I think, I think the importance

51:28

of security, the importance of just like information

51:30

security, just like you're training a powerful AI

51:33

system, you should make it hard for someone to steal

51:35

has like popped out to me as a thing that just

51:37

like keeps coming up in these stories, keeps

51:39

being present. It's hard to tell a story where it's not

51:42

a factor. It's easy to tell a story where it is a factor.

51:44

You know, another factor that has come up for me is just like,

51:46

there needs to be

51:47

some kind of way

51:49

of stopping or disempowering

51:52

dangerous AI systems. You can't just

51:54

build safe ones. Or like if you build the safe

51:57

ones, you have to somehow use them to help you stop

51:59

the dangerous because eventually people will build

52:01

dangerous ones. And I think the most promising

52:03

general framework that I've heard for

52:06

doing this is this idea of a kind

52:08

of evals-based regime where you test

52:11

to see if AIs are dangerous. And based

52:13

on the tests, you kind of have the world

52:15

coming together to stop them or you don't. And

52:17

I think even in a world where you have very powerful, safe

52:20

AI systems, you still need some kind of, probably,

52:22

you still need some kind of regulatory framework for

52:25

how to use those to use force to stop

52:27

other systems. And so these are

52:29

general factors that I think it's a little bit like

52:31

I think it's how some people might do math by imagining

52:34

a concrete example of a mathematical object,

52:36

seeing what they notice about it, and then abstracting

52:39

back from there to the principles. That's what I'm doing with

52:41

a lot of these stories. I'm just like, can I tell

52:43

a story specific enough that it's not obviously crazy?

52:45

And then can I see what themes there

52:48

are in these stories and which things I robustly

52:50

believe after coming back to reality? That's

52:52

a general reason for writing stories like that. The

52:54

specific story you're referring to, I wrote a post

52:57

on last round called Success Without Dignity,

52:59

which is

52:59

kind of a response to Eliezer Kowalski writing

53:02

a piece called, I think it was called Death With Dignity.

53:04

Yeah, we should possibly explain that idea. I think some

53:07

people have become so pessimistic about our prospects

53:10

of actually

53:11

avoiding going extinct, basically, because they think

53:13

this problem is just so difficult. But they've said, well,

53:15

really, the best we can do is to not make

53:17

fools of ourselves in the process of going extinct,

53:19

that we should at least cause our own extinction

53:21

in some way that's like barely respectable, if I

53:23

guess aliens were to read the story or to uncover

53:26

what we did. And they call this kind of dignity

53:29

or death with dignity. Death with dignity. Sorry, anyway. Yeah.

53:31

And to be clear, the idea there is not

53:33

like we're literally just trying to have

53:35

dignity. It's like the idea is like, that's a proximate

53:38

thing you can optimize for that actually increases

53:40

your odds of success the most

53:41

or something. Yeah, and my response- For

53:43

many people, that's a little bit tongue in cheek as well. Yeah, yeah,

53:45

yeah, for sure. And my response though

53:47

is a piece called, Success Without Dignity, that's just

53:49

kind of like, well, I don't know, it's just actually

53:51

pretty easy to picture a world where we just like,

53:54

we just like do everything wrong. And like,

53:56

there's no real positive surprises from here.

53:59

At least not-

53:59

in terms of like people who are deliberately

54:02

trying in advance to reduce AI x-risk, like

54:04

there's no big breakthroughs on AI alignment, there's

54:06

no like real happy news, just a lot of stuff just happens

54:08

normally and happens on the path it's on, and then

54:11

we're fine, and why are we fine? Well, we basically

54:13

got lucky, and I'm like, can you tell a story like

54:15

that? And I'm like, yeah, I think I can tell a story like that,

54:17

and why does that matter? I think it matters because

54:19

it's, um, I think a number of people have

54:21

this feeling with AI that they're just like, we're

54:23

screwed by default, we're gonna have to get like 10

54:25

really different hard things all right,

54:28

or we're totally screwed, and so therefore we

54:30

should be trying like really crazy

54:32

swing for the fences stuff and forgetting about

54:34

interventions that like help us a little bit. And

54:37

yeah, and I have the opposite take, I just think like, look,

54:39

if nothing further happens, there's some

54:41

chance that we're just fine, basically by luck, so

54:44

we shouldn't be doing like overly crazy things to increase

54:46

our variance if they're not, you know, if they're

54:48

not like kind of highly positive and expected value,

54:51

and then I also think like, yeah, things that just

54:53

like help a little bit, like, yeah, those are good.

54:55

They're good at face value, they're good in the way you'd

54:57

expect them to be good, they're not like

54:59

worthless because they're not enough, and

55:02

so I think, you know, things like just working harder

55:04

with AI systems

55:06

to get the reinforcement AI systems

55:08

are getting to be accurate, so just that, you know, this idea

55:10

of accurate reinforcement where you're not

55:12

rewarding AI systems specifically

55:15

for doing bad stuff, you know,

55:17

that's a thing you can kind of like get more right or get

55:19

more wrong, and doing more attempts to do that is kind

55:22

of basic, and it's not, it doesn't involve

55:24

like clever re-thinkings of what cognition means

55:26

and what alignment means and how we get a perfectly aligned

55:28

AI, but I think it's a thing that

55:30

could matter a lot, and that putting more effort into

55:32

could matter a lot, and I feel that way too, about improving

55:35

information security, like you don't have to make

55:37

your AI impossible to steal, make it hard to steal is worth

55:39

a lot, you know, so there's a lot of just

55:41

generally a lot of things I think people can do to reduce AI

55:43

risks that don't rely on a complicated

55:45

picture. It's just like this thing helps, so just do it because

55:47

it helps. Yeah,

55:49

we might go over those interventions in just a

55:51

second. Yeah, maybe it's possible to like flesh out the story

55:53

a little bit, like yeah, how could we get a

55:55

good outcome? Mostly through luck.

55:58

So I broke these... success

56:00

without dignity idea into

56:02

a couple phases. So there's the initial

56:05

alignment problem, which is the, you know, the thing

56:07

most people I think in the time of Doom or Headspace

56:09

tend to think about, which is how do we,

56:11

how do we build a very powerful AI system that

56:14

is not trying to take over the world or disempower

56:16

our humanity or kill all humanity or whatever. And

56:18

so there, I think if you are training

56:21

systems that are

56:22

human, I call them human level ish. So

56:25

an AI system that's like got kind

56:27

of similar range capabilities to human,

56:29

it's going to have some strengths and some weaknesses relative

56:31

to a human. If you're training that kind of system,

56:34

I think that

56:35

you may just get systems that are pretty

56:37

safe, at least for the moment, without

56:39

a ton of breakthroughs or special

56:42

work. You might get it by pure

56:44

luck ish. So it's basically like this thing I

56:46

said before about how, you know, you have an AI system

56:48

and you train it by basically saying

56:50

good job or bad job when it does something. It's

56:52

like human, human feedback for a human

56:55

level ish system that could easily result

56:57

in a system that like

56:58

either it really did generalize to doing

57:00

what you meant it to do or it generalized

57:03

to this like thing where it's like

57:05

trying to take over the world. But that

57:07

means cooperating with you because whenever it's

57:09

too weak to take over the world. And in fact, these human level

57:11

ish systems are going to be like too weak to take over the world. So they're

57:13

just going to cooperate with you. It could mean that.

57:16

So you could get like either two of those generalizations.

57:18

And then like,

57:20

it does matter if you're like I just

57:22

said, if your reinforcement is accurate. So

57:24

you could kind of like have an AI system where you say,

57:26

hey, go make me a bunch of money. And then

57:28

unbeknownst to you, it goes and like breaks

57:31

a bunch of laws and hacks into a bunch of stuff

57:33

and brings you back some money or even like fakes

57:35

that you have a bunch of money and then you say good job. Now

57:38

you've actually rewarded it for doing bad stuff. But

57:40

if you can take that out, if you can basically avoid

57:42

doing that and have your have your

57:45

kind of like good job when it actually did

57:47

a good job, that I think increases the

57:49

chances that that it's going to generalize

57:51

to basically just doing doing a good

57:53

job or at least doing what we roughly intended and

57:56

not kind of pursuing goals of its

57:58

own, if only because that wouldn't work. And so I

58:00

think you could, you know, a lot of this is to say,

58:02

you could solve the initial alignment problem by almost

58:05

pure luck, by this kind of reinforcement learning

58:07

from human feedback, generalizing well. You could

58:09

add a little effort on top of that and make it more

58:11

likely, like getting your reinforcement more accurate.

58:13

There's some other stuff you could do in addition

58:15

to kind of catch some of the failure modes and

58:18

straighten them out, like red teaming

58:20

and simple checks and balances. I won't go into the details

58:22

of that. And if you get some combination

58:24

of luck and skill here, you end up with

58:27

AI systems that are roughly human

58:29

level, not immediately

58:31

dangerous anyway. Sometimes they call them

58:33

jinkily aligned. It's like they are

58:35

not trying to kill you at this moment. That doesn't mean you

58:37

solve the alignment problem. But

58:40

at this moment, they are approximately trying to help you.

58:42

Maybe if they can all coordinate and kill you, they

58:45

would, but they can't. Remember, they're kind of human-like.

58:47

So that's the initial alignment problem. And

58:49

then once you get past that,

58:51

then I think we should all just

58:53

forget about the idea that we have any idea what's gonna

58:55

happen next. Because now you

58:59

have a huge, potentially huge number

59:01

of human-level-ish AIs. And

59:03

that is just incredibly world-changing. And

59:06

there's this idea of, that

59:08

I think sometimes some people call it, getting

59:10

the AIs to do our alignment homework for us. So

59:13

it's this idea that once you have human-level-ish

59:15

AI systems, you have them kind of working

59:17

on the alignment problem in huge numbers.

59:20

And it's like, in some ways I hate this idea because

59:22

it's just very lazy and it just is like, oh

59:24

yeah, we're not gonna solve this problem until later when the

59:26

world is totally crazy and everything's moving really

59:28

fast and we have no idea what's gonna happen. So I hate

59:30

the idea in that sense. We'll just ask the

59:32

agents that we don't trust to make themselves trustworthy.

59:35

Yeah, exactly. So there's a lot to hate about this

59:37

idea, but heck, it could work.

59:40

It really could. Because you could have a situation

59:43

where just in a few months, you're able

59:45

to do the equivalent of thousands of years of

59:48

humans doing alignment research. And if these

59:50

systems are just not at the point where they can

59:52

or want to

59:53

screw you up, that really

59:55

could do it. I mean, we just don't know that like

59:57

thousands of years of human levelish alignment research isn't

59:59

enough.

59:59

to just like get us a real solution.

1:00:02

And so that's kind of how you get through

1:00:04

a lot of it. And then you still have another problem in

1:00:07

a sense, which is that you do

1:00:09

need a way to stop dangerous systems. It's

1:00:11

not enough to have safe AI systems. But

1:00:13

again, you have help from this giant automated

1:00:15

workforce. And so in addition to

1:00:18

coming up with ways to make your system safe, you can come up with ways

1:00:20

of showing that they're dangerous and when they're

1:00:22

dangerous and being persuasive about the importance

1:00:24

of the danger. And that again, feels like something

1:00:26

that like, I don't know, I feel like if we had 100

1:00:29

years before AGI right now, there'd be a good chance that

1:00:31

normal flesh and blood humans could pull this off. So

1:00:34

in that world, there's a good chance that an automated workforce

1:00:36

can cause it to happen pretty quickly. And you

1:00:38

could pretty quickly get, get

1:00:40

understanding of the risks agreement that we need to stop

1:00:43

them. And you have more safe

1:00:45

AI's than dangerous AI's and you're trying to stop the dangerous

1:00:47

AI's and you're measuring the dangerous AI's or

1:00:50

you're stopping any AI that refuses to be measured or who's

1:00:52

developer refuses to measure it. And so then

1:00:54

you have a world that's kind of like this one, where like, yeah,

1:00:56

there's a lot of evil people out there, but there's, but they

1:00:59

are generally just kept

1:00:59

in check by being outnumbered by people who

1:01:02

are at least law abiding, if not incredibly angelic.

1:01:04

So you get a world that looks like this one, but it just

1:01:06

has a lot of like AI's running around in it. And

1:01:09

so we have like a lot of progress in science and technology

1:01:11

and that's, that's a fine ending potentially.

1:01:14

Okay, so that's one flavor

1:01:16

of story. Is there any other broad themes

1:01:18

in the positive stories that it could be worth bringing

1:01:20

out before we move on?

1:01:21

I think I mostly covered it. I mean, the other

1:01:23

two stories involve less luck

1:01:26

and more like you have one or two actors

1:01:28

that just do a great job. Like, you know, you have one

1:01:30

AI lab that it's just ahead of everyone else and

1:01:33

it's just like doing everything right. And that improves

1:01:35

your odds a ton. You know, for a lot

1:01:37

of this reason that like being a few months ahead could

1:01:39

mean you have like, you know, a lot

1:01:42

of subjective time of having your automated workforce

1:01:44

do stuff to be helpful. And so there's one

1:01:46

of those with like a really fast takeoff and one of them with a

1:01:48

more gradual takeoff.

1:01:50

But I think, you know, I think that does, that does kind of highlight

1:01:52

again that like one really good

1:01:54

actor who's like really successful could

1:01:56

move the needle a lot even when you get less

1:01:58

luck. So I think there's...

1:01:59

There's a lot of ways things could go well. There's a lot

1:02:02

of ways things could go poorly. I feel like

1:02:04

I'm saying just like

1:02:05

really silly, obvious stuff now that just should

1:02:07

be everyone's starting point, but I do

1:02:09

think it's not where most people are at right now. I think these risks

1:02:11

are extremely serious. They're kind of my

1:02:14

top priority to work on. I think

1:02:16

anyone's saying we're definitely gonna be fine. I don't know where the heck

1:02:18

they're coming from, but anyone's saying we're definitely doomed. I

1:02:20

don't know, same issue.

1:02:21

Okay,

1:02:23

so the key components of the story here was, so

1:02:25

one, you didn't get bad people stealing the models

1:02:27

and misusing them really early on. Or

1:02:30

there was some limit to that and they were outnumbered or

1:02:32

something like that, yeah. Okay, and

1:02:34

initially we ended up training models that are

1:02:36

more like human level intelligence and it turns out to

1:02:38

not be so challenging to have

1:02:41

moderately aligned models like that. And

1:02:43

then we also managed to turn those AIs, those

1:02:47

folks I guess, I'm gonna say it, towards

1:02:50

the effort of figuring out how to align additional

1:02:52

models that would be more capable still.

1:02:54

And or how to slow things down and put in

1:02:56

a regulatory regime that stops things if we don't know

1:02:58

how to make safe, yeah. Right, okay, or they

1:03:00

help with, yeah, they help with a bunch of other governance

1:03:02

issues, for example, and then also by

1:03:04

the time these models have proliferated and

1:03:07

they might be getting used by irresponsibly

1:03:10

or by bad actors, those folks are just

1:03:12

massively outnumbered as they are today by people

1:03:15

who are largely sensible. Okay, so

1:03:17

I'm really undecided on

1:03:19

how plausible these stories are. I guess

1:03:22

I

1:03:22

play some weight or some credence

1:03:25

on the possibility that pessimists like Eliezer,

1:03:27

Yudkowsky are right and that this kind of thing

1:03:29

couldn't happen for one reason or another. And

1:03:32

we really would. I think that's possible, yeah. Yeah, I

1:03:34

guess, what do you make of the argument

1:03:36

that, let's say we're 50-50, short

1:03:38

between kind of the Eliezer worldview and

1:03:40

the Holden worldview just outlined, that in

1:03:43

the case of Eliezer's right, we're kind of just screwed, right?

1:03:45

And things that we do on the margin, a bit of extra

1:03:47

work here and there, just isn't gonna change the story.

1:03:50

Basically, we're just gonna go extinct with very

1:03:52

high probability.

1:03:52

Whereas if you're right, then

1:03:55

things that we do might actually move the needle and we

1:03:57

have a decent shot. So it makes more sense to act.

1:03:59

as if we have a chance, as if some

1:04:02

of this stuff might work, because our decisions

1:04:04

and our actions just aren't super relevant

1:04:07

in the pessimist case. Does that sound like a sensible

1:04:09

reasoning? I mean, it seems a little bit suspicious

1:04:12

somehow. I think it's a little bit suspicious. I

1:04:14

mean, I think it's fine if it's 50-50. I think

1:04:16

Eliezer has complained about this. He's kind of said, you

1:04:18

know, look, you can't condition

1:04:21

on a world that's fake, and you should live

1:04:23

in the world you think you're in. I think that's right. So

1:04:26

I should say, I think I do want to say a couple meta

1:04:28

things about the success of that dignity story.

1:04:29

One is, I do want people to know, this is

1:04:32

not like a thing I cooked up. This is, you

1:04:34

know, I think of my job, I'm not an AI

1:04:36

expert, I think of my job being especially

1:04:38

a person who's generally been a funder, having

1:04:40

access to a lot of people, having to make a lot of people judgments.

1:04:43

My job is really to figure out who to listen

1:04:45

to about what and how much weight to give whom about what.

1:04:48

So I'm getting the actual substance

1:04:50

here from people like Paul Chris Giano,

1:04:52

Carl Schulman, and others, and this

1:04:54

is not holding reasoning things out and being like, this

1:04:56

is how it's going to be. This is me synthesizing,

1:04:59

hearing from a lot of different people, a lot of them

1:05:01

highly technical, a lot of them experts, and

1:05:03

just trying to say who's making the most sense, also

1:05:06

considering things like track records, like who should be getting

1:05:08

weight, things like expertise. So

1:05:10

that is an important place to know where I'm coming from.

1:05:12

But I do, having done that, I

1:05:15

do actually feel that this success

1:05:17

of that dignity is just like a serious possibility, and

1:05:19

I'm way more than 50-50 that

1:05:21

this is possible. That

1:05:24

according to the best information we have now, this

1:05:26

is a reasonable world to be living in, is a world where this could

1:05:28

happen and we don't know, and way less

1:05:29

than 50-50, the kind of the LAS model

1:05:32

of, yeah, we're doomed for sure with like 98 or 99% probability,

1:05:36

is right. I don't put zero credence on it, but it's just

1:05:38

not my majority view. But the other thing,

1:05:40

which relates to what you said, is

1:05:42

I don't want to be interpreted as saying this

1:05:44

is a reason we should chill out. So it

1:05:46

should be obvious, but my argument should stress

1:05:48

people out. My picture should stress people out

1:05:50

way more than any other possible picture. I use

1:05:52

the most stressful possible picture because it's

1:05:55

like

1:05:55

anything could happen, every little bit helps. Like,

1:05:58

if you help a little more, a little less, like that would be a good idea.

1:05:59

actually matters in this potentially huge

1:06:02

kind of fated humanity kind of way. And that's

1:06:04

a crazy thing to think in its own way. And it's certainly not a relaxing

1:06:07

thing to think, but it's yeah, it's what I think as far as I can tell.

1:06:10

Yeah,

1:06:10

I'm I'm stressed. Don't worry. Okay,

1:06:12

good. Okay. Okay. So,

1:06:16

so this kind of worldview leads

1:06:18

into something that you wrote, which is this kind of four

1:06:20

intervention playbook for possible success

1:06:23

with AI. And you kind of describe

1:06:25

four different categories of interventions that we might

1:06:27

engage in in order to to try to improve

1:06:29

our odds of success. I think there was alignment research,

1:06:32

standards and monitoring, creating a successful

1:06:34

and careful AI lab. And finally, information

1:06:36

security. I think we've touched on all these a

1:06:39

little bit, but maybe we could go over them again. Is there

1:06:41

anything you want to say about alignment research as an intervention

1:06:43

category that you haven't said already? Well,

1:06:45

I mean, I've kind of pointed at it. But

1:06:48

I think in my head, I see value

1:06:50

in I think there's like versions of alignment

1:06:53

research that are very like blue sky

1:06:55

and very like we have to have a fundamental

1:06:58

way of being like really sure that any

1:07:01

arbitrarily capable AI is like

1:07:03

totally aligned with what we're trying to get it to do. And

1:07:05

I think that I think that's very hard

1:07:07

work to do. I think a lot of the actual work being done on it

1:07:10

is not valuable. But I think if you can move the

1:07:12

needle on it, I think it's super valuable. And

1:07:14

then there's work that's like a little more prosaic and

1:07:16

it's a little more like, well, can we, you

1:07:18

know, use our train our eyes

1:07:20

with human feedback and find some way that screws

1:07:22

up and kind of patch the way it screws up and go to

1:07:24

the next step. A lot of this work is

1:07:26

like pretty empirical as being done at AI labs.

1:07:29

And I think that works. It's just like super valuable as well.

1:07:31

And so that is a take I have in alignment research. I

1:07:34

do think almost all alignment research

1:07:37

is believed by many people to be totally useless

1:07:39

and or harmful. And

1:07:41

I tend not to super feel that way. I think if anything,

1:07:44

the line I would draw is there is some

1:07:46

alignment research that seems like

1:07:48

it's necessary eventually to commercialize.

1:07:50

And so I'm a little less excited about that because I do think it will

1:07:52

get done regardless on the

1:07:54

way to whatever we're worried about. And so I do

1:07:56

tend to draw lines about like, how likely is

1:07:59

this research to get done?

1:08:00

you know, not by normal commercial motives, but

1:08:02

I do think there's a wide variety of alignment

1:08:05

research that can be helpful, although

1:08:07

I think a lot of alignment research also is not helpful,

1:08:09

but that's more because it's like not aimed

1:08:11

at the right problem, and less because it isn't like exactly

1:08:14

the right thing. And so that, yeah, that's a take on

1:08:16

alignment research. Then another another take is I

1:08:18

have kind of highlighted what I call threat assessment

1:08:21

research as a thing that you could consider

1:08:23

part of alignment research or not, but

1:08:25

it's probably the single category that feels to me

1:08:27

like

1:08:28

the most in need of more work right now,

1:08:30

given where everyone is at, and that

1:08:32

would be, you know, basically work trying

1:08:35

to kind of create the problems

1:08:37

you're worried about in a controlled environment where

1:08:40

you can age a show that they

1:08:42

could exist and understand the conditions

1:08:44

under which they do exist. So, you know, problems

1:08:46

like a misaligned AI that is pretending to be aligned,

1:08:49

and see you can actually like study alignment techniques

1:08:51

and see if they work on many versions of the problem. So

1:08:53

like, you know, you could think of it as like model organisms

1:08:56

for AI where, you know, in order to cure

1:08:58

cancer, it really helps to be able to give cancer to mice.

1:09:00

In order to deal with AI misalignment, it really

1:09:02

helps to be able to create, if we could ever create a deceptively

1:09:05

aligned agent that is like, you know, secretly

1:09:08

trying to kill us, but it's too weak to actually kill

1:09:10

us, that would be way better than having the first

1:09:12

agent that's secretly trying to kill us be something that actually can kill

1:09:15

us. So I'm really into kind of creating,

1:09:17

creating the problems we're worried about in controlled environments.

1:09:20

Yeah, yeah. Okay, so the second category

1:09:22

was standards and monitoring, which you've already

1:09:24

touched on. Is there anything high level you want to

1:09:27

say about that one?

1:09:28

Yeah, this is kind of, to me, the most

1:09:30

nascent, or like the one that just, it's, there's

1:09:33

not much happening right now, and I think there could be a lot

1:09:35

more happening in the future, but the basic idea

1:09:37

of standards of monitoring is this idea of you, you

1:09:39

have tests for whether AI systems are dangerous, and

1:09:41

you have a regulatory or self-regulatory

1:09:44

or a normative, you know, informal

1:09:46

framework that says dangerous AI should

1:09:48

not be trained at all or deployed. So,

1:09:51

and not be trained, I mean like, you

1:09:53

found initial signs of danger in one AI model, so

1:09:55

you're not going to make a bigger one. Not just you're not going to deploy,

1:09:57

you're not going to train it.

1:09:58

I'm excited about standards.

1:09:59

of monitoring in a bunch of ways. I think

1:10:03

it feels like it has to be eventually part

1:10:05

of any success story. There has to be some framework

1:10:07

for saying, hey, we're going to stop

1:10:09

dangerous AI systems. But it also,

1:10:12

in the short run, I think it's got more advantages than

1:10:14

sometimes people realize, where I think

1:10:16

it's not just about slowing things

1:10:18

down. It's not just about stopping directly

1:10:20

dangerous things. I think a good standards

1:10:23

and monitoring regime would create massive

1:10:25

commercial incentives to actually

1:10:27

pass the tests. And so if the tests are

1:10:29

good,

1:10:29

if the tests are well-designed to actually

1:10:32

catch danger where the danger is, you could

1:10:34

have massive commercial incentives to actually

1:10:36

make your AI system safe and show that

1:10:38

they're safe. And I think we'll get much different

1:10:41

results out of that world

1:10:43

than out of a world where everyone is trying to make

1:10:45

and show AI system safety is doing it out of the goodness

1:10:47

of their heart.

1:10:48

Or just for a salary.

1:10:50

It seems like standards and monitoring is kind of a new

1:10:53

thing on the public discussion, but it seems like people are

1:10:55

talking around this issue or governments

1:10:57

are considering this, that the labs are now publishing

1:11:00

papers in this vein. To

1:11:02

what extent do you think you'd need complete coverage

1:11:05

for some standards system in order for it

1:11:07

to be effective? I'm just imagining, I guess it seems

1:11:09

like OpenAI, DeepMind, Anthropic,

1:11:12

they all are currently saying pretty similar

1:11:14

things about they're quite concerned about extinction

1:11:16

risk, or they're quite concerned about ways that AI could go wrong. But

1:11:18

it seems like the folks at Meta, led

1:11:20

by Yann LeCun, kind of have a different attitude. And it

1:11:23

seems like it might be quite a heavy lift to get them to voluntarily

1:11:25

agree to join the same sorts of standards

1:11:27

and monitoring that some of those other labs might

1:11:29

be enthusiastic about. And I wonder, how much does it,

1:11:32

is there a path to getting everyone on board?

1:11:34

And if not, would you just end up

1:11:37

with the most rebellious, the least anxious,

1:11:39

the least worried lab basically running ahead?

1:11:41

Well, I think that latter thing is definitely risk, but I think

1:11:43

it could probably be dealt with. I mean, one

1:11:46

way to deal with it is just to build it into the standards regime

1:11:48

and say, hey, you know, you can do

1:11:50

these dangerous things, train an AI system, or

1:11:53

deploy an AI system, you can do them. A, if you can show

1:11:55

safety, or B, if you can show that someone

1:11:57

else is going to do it, you know, you

1:11:59

could even say, hey,

1:11:59

When someone else comes even close

1:12:02

even within an order of magnitude of your dangerous

1:12:04

system now you can deploy your dangerous system Doesn't

1:12:07

that kind of

1:12:08

but but that it seems like the craziest people can then kind

1:12:10

of just force everyone else or like lead everyone else

1:12:12

Down the the garden path.

1:12:14

Well, it's just I what's the alternative, right? It's just like

1:12:16

you can you can design the system every one I think it's

1:12:18

like you can either you can say okay You have

1:12:21

to wait for them to actually catch you in which case it's hard.

1:12:23

It's hard to see how this standard system does harm It's

1:12:25

still kind of a scary world Yeah Or you can say you can wait

1:12:27

for them to get anywhere close which now you've got Potentially

1:12:29

a little bit of acceleration thrown in there and

1:12:32

and maybe you did or didn't decide that was better than

1:12:34

actually just like slowing Down all the cautious players. I think

1:12:36

is a real concern I will say

1:12:38

that I don't feel like you have to get universal

1:12:41

consensus from the jump for a few reasons One

1:12:44

is it just one step at a time? So I think

1:12:46

one is it just like if you can start

1:12:48

with some of the leading labs being into this

1:12:51

There's a lot of ways that other folks could come on

1:12:53

board later. Some of it is just like, you know peer pressure

1:12:56

Yeah, we've seen with the corporate campaigns

1:12:58

for farm animal welfare that you've probably covered Just

1:13:01

like once a few dominoes fall it gets

1:13:03

very hard for others to hold out because they kind of

1:13:05

look like way more You know way more callous

1:13:07

or in the AI system case way more reckless Of

1:13:10

course, you know, there's also the possibility for regulation

1:13:12

down the line and I think regulation Could

1:13:15

be more effective if it's based on something

1:13:17

that's already been like implemented in the real world That's

1:13:20

actually working and that's actually detecting dangerous

1:13:22

systems So I don't know a lot of me is just like

1:13:24

one step at a time a lot of me is just like you see

1:13:26

if You can get a system working for anyone

1:13:29

that catches dangerous AI systems and stops

1:13:32

until it can show they're safe And then you think

1:13:34

about how to expand that system And then

1:13:36

a final point is the incentives point we're just like this

1:13:38

is not the world I want to be in and this is not a world I'm that

1:13:40

excited about but in a world where the leading labs

1:13:43

are

1:13:43

Using kind of a standards and E-Vals

1:13:46

framework for a few years and then no one

1:13:48

else ever does it and then eventually we just have To drop

1:13:50

it. Well, that's still a few years in which I think

1:13:52

you are gonna have meaningfully different incentives for those for

1:13:54

those leading labs About how they're gonna prioritize You

1:13:57

know tests of safety and actual safety

1:13:59

measures

1:14:00

Yeah. Do you think there's room for a big

1:14:03

business here, basically? Because I would think with

1:14:05

so many commercial applications of ML

1:14:07

models, people are going to want to have them certified

1:14:10

that they work properly and that they don't flip out and

1:14:12

do crazy stuff. And in as much as this is going to become

1:14:14

a boom industry, you'd think that

1:14:16

the group that has the greatest expertise in like

1:14:18

independently vetting and evaluating like

1:14:21

how models behave when they're put in a different environment might

1:14:23

just be able to sell this service for

1:14:25

a lot of money. Well, there's the independent expertise,

1:14:28

but I think in some ways I'm more interested in the financial

1:14:30

incentives for the companies themselves. So if you look

1:14:32

at like big drug companies,

1:14:35

a lot of what they are good at is

1:14:37

the FDA process. A lot of what they're good at is running

1:14:39

clinical trials, doing safety studies, proving

1:14:42

safety, documenting safety, arguing safety and

1:14:44

efficacy. You could argue about

1:14:46

whether there's like too much caution at the FDA. I think

1:14:48

in the case of COVID, there may have been some of that, but

1:14:51

certainly it's a regime where there's big

1:14:53

companies that a major priority,

1:14:55

maybe at this point a higher priority for them than innovation

1:14:58

is actually measuring and demonstrating safety

1:15:01

and efficacy. And so you could imagine landing

1:15:03

in that kind of world with AI. And I think that would just, yeah,

1:15:05

that would be a very different world from the one we're going to go into

1:15:07

by default. I do think that that's not

1:15:09

just about, you know, the FDA is not the one making money

1:15:12

here, but it's changing the way that

1:15:14

the big companies think about making money, certainly

1:15:16

redirecting

1:15:16

a lot of their efforts into demonstrating

1:15:19

safety and efficacy as opposed to coming up with new kinds of drugs,

1:15:21

both of which have some value. But I think

1:15:23

we're a bit out of balance on the AI side right now. Yeah.

1:15:27

Yeah, it is funny. For

1:15:29

so many years, I've been just infuriated by the FDA.

1:15:31

And I feel like these people, they only consider

1:15:33

downside, they only consider risk, they don't think

1:15:36

about upside nearly enough. And now I'm like, can

1:15:38

we get some of that insanity over here, please? Yeah,

1:15:41

yeah, yeah. No, I know. I know. There

1:15:43

was a very funny Scott Alexander piece

1:15:45

kind of making fun of this idea. But

1:15:46

I mean, I think it's legit. It's just honestly, it's just

1:15:49

kind of a boring opinion to have. But I think that

1:15:52

I think that innovation is good. And I think safety

1:15:54

is good. And I think we have a lot, we

1:15:56

have a lot of parts of the economy that are just way

1:15:58

overdoing the safety. Just like you can't, you

1:16:01

can't give a haircut without a license and you

1:16:03

can't like build an in-law unit in your

1:16:05

house without like a three year process

1:16:08

of forms. And you know, Open Philanthropy works on a lot

1:16:10

of this stuff. We are the first institutional funder of

1:16:12

the IMBI movement, which is this movement

1:16:14

to make it easier to build houses. I think

1:16:16

we overdo that stuff all the time. And I think the

1:16:18

FDA sometimes overdoes that stuff in a horrible

1:16:21

way. And I think during COVID, you know, I do believe that

1:16:23

things moved way too slow. And then I think with

1:16:25

AI, we're just not doing anything. There's just no framework

1:16:27

like this in place at all. So I don't know how about

1:16:29

a middle ground?

1:16:29

Yeah. If only we could get the same level

1:16:32

of review for these potentially incredibly dangerous

1:16:34

self-replicating AI models that we have for building a

1:16:36

block of apartments. Yeah, right. Exactly.

1:16:39

In some ways, I feel like this incredible paranoia

1:16:41

and this incredible focus on safety, if there's one place

1:16:43

it would make sense, that would be AI. I

1:16:46

honestly, weirdly, I'm not saying that

1:16:48

we need to get AI all the way to

1:16:50

being as cautious as like regulating housing

1:16:53

or drugs. Maybe it should be less cautious

1:16:55

than that. Maybe. But right now, it's

1:16:57

just nowhere. And so I think, you know, I think

1:16:59

you could think

1:16:59

of it as like there's FDA and zoning energy

1:17:02

and then there's like AI energy and like, yeah,

1:17:04

maybe housing should be more like AI. Maybe AI should

1:17:07

be more like housing. But I definitely feel like we

1:17:09

need more caution in AI. That's what I think. More caution

1:17:11

than we have. And that's not me saying that we need

1:17:13

to forever be in a regime where

1:17:15

safety is the only thing that people care about. Yeah.

1:17:17

You've spoken a bunch with the folks at AHRQ

1:17:20

Evaluations. I think AHRQ Sense 4 was an

1:17:22

AI research center. Alignment Research Center. Alignment

1:17:24

Research Center, yeah. And they have an evaluations

1:17:26

project. Yeah. Could you maybe give

1:17:29

us a summary of the project that they're engaged

1:17:31

in and the reasoning behind it?

1:17:32

I have spent some time as an advisor to ArchiVals.

1:17:35

That's a group that is, you know, headed

1:17:37

by Paul Christiano. Beth Barnes is leading

1:17:39

the team. And they work on basically

1:17:41

trying to find ways to assess whether AI systems

1:17:44

could pose risks, whether they could be dangerous.

1:17:47

And they also have thought about whether

1:17:49

they want to experiment with like putting out kind of

1:17:51

proto standards and proto expectations

1:17:53

of, hey, if your model is dangerous in this way, here

1:17:56

are the things you have to do to contain it. And make it safe.

1:18:00

a lot of intellectual firepower there to design

1:18:02

evaluations of AI systems and where

1:18:04

I'm, you know, hopefully able to add a little

1:18:06

bit of just like staying on track

1:18:09

and helping run and build an organization because

1:18:11

it's all quite new. But they were the

1:18:14

ones who they did an evaluation on GPT-4

1:18:17

for whether it could kind of create copies

1:18:19

of itself in the wild. And they kind of concluded, no,

1:18:21

as far as they were able to tell, although they weren't able

1:18:24

to do yet all the all the research

1:18:26

they wanted to do, especially they weren't able to do a fine tuning

1:18:28

version of their evaluation.

1:18:29

Okay, so while we're on safety

1:18:32

and evaluations, as I understand it, this is kind of something that

1:18:34

you've been thinking about in particular that the last

1:18:36

couple of months, maybe what what new things

1:18:38

have you learned about this part of the of the playbook

1:18:40

over the last six months?

1:18:42

Yeah, the evaluations and standards

1:18:44

and monitoring. I mean, one thing that

1:18:46

just has become clear to me is there's just,

1:18:49

it is really hard to design evaluations

1:18:52

and standards here. And there's just a lot of like, hairy

1:18:54

details around things like, you

1:18:56

know, auditor access. So if you you

1:18:58

know, there's this kind of idea that you would have an

1:19:00

AI lab have an outside independent auditor

1:19:03

determine whether their models have dangerous capabilities.

1:19:05

But it's a fuzzy question. Does the

1:19:07

model have dangerous capabilities? Because it's

1:19:10

going to be sensitive to a lot of things like, how

1:19:12

do you prompt the model? How do you interact with the model?

1:19:14

Like, what are the things that can happen to it that

1:19:17

cause it to actually demonstrate these dangerous capabilities?

1:19:19

If someone builds a new tool for GBT4 to

1:19:22

use, is that going to cause it to become more dangerous?

1:19:24

In order to investigate this, you have to actually

1:19:26

be like good at working with the model and

1:19:29

understanding what its limitations are. And a lot of times,

1:19:31

just like the AI labs not only

1:19:33

know a lot more about their models, but they have like a bunch of

1:19:35

features that like, it's hard to share all

1:19:37

the features at once, they have a bunch of different versions of the model.

1:19:40

And so it's quite hard to make outside

1:19:42

auditing work for that reason. Also, if you're

1:19:44

thinking about standards, you're thinking about, you know, a

1:19:47

general kind of theme in a draft

1:19:50

standard might be, once your AI has shown

1:19:52

initial signs, and it's able to do something dangerous,

1:19:54

such as autonomous replication, which means that

1:19:56

it can basically, you know, make a lot

1:19:59

of copies of itself with

1:19:59

without help and without necessarily getting detected

1:20:02

and shut down. There's an idea that like once you've

1:20:04

kind of shown the initial science of a system can do that,

1:20:06

that's a time to not build a bigger system.

1:20:08

And that's a cool idea, but it's like how much bigger?

1:20:11

And it's like hard to define that because

1:20:13

making systems better is

1:20:15

multi-dimensional and can involve

1:20:18

more efficient algorithms, can involve, again,

1:20:20

better tools, longer contexts, just

1:20:22

like different ways of fine tuning the models,

1:20:25

different ways of specializing them, different ways

1:20:27

of like setting them up, prompting them, like different

1:20:29

instructions to give them. And

1:20:31

so it can be like just very fuzzy, just like

1:20:34

what is this model capable of is a hard thing

1:20:36

to know. And then how do we know when

1:20:38

we built a model that's more powerful, though we need

1:20:40

to retest? These are very hard things

1:20:42

to know. And I think it

1:20:43

has has kind of moved me toward feeling

1:20:46

like we're not ready for a really

1:20:48

prescriptive standard that tells you exactly

1:20:51

what practice is to do like the farm animal welfare

1:20:53

standards are. We might need to start by asking

1:20:55

companies to just outline their own proposals

1:20:58

for what tests they're running, when and how they

1:21:00

feel confident that they'll know when it's become too

1:21:02

dangerous to keep scaling. Yeah. So

1:21:06

some some things that will be really useful to be able

1:21:08

to evaluate is, you know, is this model

1:21:10

capable of autonomous self-replication

1:21:12

by breaking into additional servers? I guess

1:21:14

you might also want to test, you know, could it be used by

1:21:17

terrorists for figuring out how

1:21:19

to produce bioweapons? Those are kind of very natural

1:21:21

ones. The breaking into servers is not

1:21:23

really central. So the idea is like, could it

1:21:25

could it make a bunch of copies of itself in the

1:21:27

presence of like minimal or like kind

1:21:29

of non-existent human attempts to stop it

1:21:31

and shut it down? So it's like, could it take basic

1:21:34

precautions to not get like obviously

1:21:36

detected as an AI by people who are not

1:21:38

particularly looking for it? And the

1:21:40

thing is, if it's able to do that, you could have a human

1:21:42

have it do that on purpose. So it doesn't necessarily have to break

1:21:45

into stuff. They can like, you know, a lot of the test

1:21:47

here is like, can it find a way to make money, make

1:21:49

the money, open an account with a server

1:21:51

company,

1:21:52

put, you know, rent server space, make copies

1:21:55

of itself on the server. None of that necessarily involves

1:21:57

it breaking in anywhere.

1:21:58

I see. So so hacking is one way.

1:21:59

to get compute, but it's by no means the only one.

1:22:02

So it's not a necessary factor. That's

1:22:04

right. I mean, it's weird, but you could be an AI system

1:22:06

that's just kind of doing normal phishing scams that you

1:22:09

read about on the internet using those to get money,

1:22:11

or just legitimate work. You could be

1:22:13

an AI system that's just like going on mTurk and being an mTurker

1:22:15

and making money, use that money to legitimately

1:22:18

rent some servers, or sort of legitimately, because it's not actually

1:22:20

allowed if you're not a human. But apparently,

1:22:23

legitimately rent some server space, install

1:22:26

yourself again, have that money, make more money, and

1:22:28

a copy make more money, and then you

1:22:29

have ... You can have quite a bit of replication

1:22:32

without doing anything too fancy, really. And that's what

1:22:34

the initial autonomous replication test that ArchiVels

1:22:36

does is about.

1:22:38

Okay. So we'd really like to be able to know where the

1:22:40

models are capable of doing that. And I suppose

1:22:42

that it seems like they're not capable now, but a couple of years'

1:22:44

time. Probably not. Again,

1:22:47

there's things that you could do that

1:22:49

might make a big difference that have not been tried yet by

1:22:51

ArchiVels, and that's on the menu. Fine tuning is the big

1:22:53

one. Fine tuning is like you have a model

1:22:56

and you do some additional training that's not

1:22:58

very expensive, but is trying to just get it

1:23:00

good at particular tasks. So you can take the task that's bad

1:23:02

at right now, train it to do those. That hasn't really

1:23:04

been tried yet. And it's like a human

1:23:07

might do that. If you have these models accessible

1:23:09

to anyone or someone can steal them, a human

1:23:11

might take a model, train it

1:23:13

to

1:23:14

be more powerful and effective and

1:23:16

not make so many mistakes, and then this thing

1:23:18

might be able to autonomously replicate. That can be scary

1:23:20

for a bunch of reasons.

1:23:21

Yeah. And then there's

1:23:24

the trying to not release models that could be used by terrorists

1:23:26

and things like that. The autonomous replication

1:23:28

is something that could be used by terrorists. You know what I

1:23:31

mean? It's overlapping. Yeah. It's

1:23:33

like if you're a terrorist, you might say, hey, let's have a

1:23:35

model that makes copies of itself that make money to

1:23:38

make more copies of itself that make money. Well, you make a lot of money

1:23:40

that way. And then you could have a terrorist organization make a lot of

1:23:42

money that way, or using its models

1:23:44

to do a lot of like little things that, you

1:23:46

know, schlepping along, like trying to plan

1:23:49

out some incredibly, you know, some

1:23:51

plan that takes a lot

1:23:51

of work to kill a lot of people. That

1:23:54

is part of the concern about autonomous replication. It's not purely

1:23:56

an alignment concern.

1:23:57

Yeah. Okay.

1:23:59

to there was just giving advice

1:24:02

to people that we really would rather that they not be able to receive

1:24:04

is maybe another category. Like helping

1:24:07

to set up a bio-weapon, yeah. An AI model

1:24:09

that could do that even if it could not autonomously

1:24:11

replicate, that could be quite dangerous. Still be not ideal,

1:24:13

yeah. And then maybe another category is

1:24:16

trying to produce these model organisms so where you can

1:24:18

study behavior that you don't want an

1:24:20

AI model to be engaging in and like understand

1:24:22

how it arises in the training process and you

1:24:24

know what sort of further feedback mechanisms

1:24:27

might be able to train that out. Like if we could produce a model

1:24:29

that will trick you whenever it thinks it can get away with it

1:24:31

but doesn't when I think it's going to get caught, that would be really helpful.

1:24:33

Yeah. Are there any other broad categories of

1:24:36

standards and eval work that you're excited by?

1:24:39

So the way I would carve up the space

1:24:42

is there's like um there's capability evals

1:24:44

and that's like is this AI kind of capable

1:24:47

enough to do something scary? Forgetting

1:24:49

about whether it wants to and I think capability evals

1:24:51

are like you know could an AI,

1:24:53

if a human tried to get it to do it, could an

1:24:55

AI make a bunch of copies of itself? Could an

1:24:58

AI design a bioweapon? Those

1:25:00

are capability evals. Then there's like alignment evals that's

1:25:02

like

1:25:03

does this AI actually do what it's supposed

1:25:05

to do or does it have like some weird goals of its

1:25:07

own? So the stuff you talked about with model

1:25:09

organisms would be like more of an alignment eval the

1:25:11

way you described it. And the autonomous

1:25:14

replication is a capability eval. I think

1:25:16

a very important

1:25:17

subcategory of capability

1:25:20

evals is what I call meta capability

1:25:22

evals, meta dangerous capabilities which

1:25:25

is basically any ability an AI system

1:25:27

has that would make it very hard

1:25:29

to get confident about what other abilities it has.

1:25:32

So the you know an example would be um

1:25:34

what I'm currently tentatively calling unauthorized

1:25:36

proliferation. So an AI model that

1:25:38

can walk a human through building

1:25:41

a powerful AI model of their own that

1:25:43

is not subject to whatever restrictions and controls

1:25:45

the original one is subject to. That could be a very

1:25:47

dangerous capability that like you could say well all

1:25:50

right I can design a bioweapon but we but it

1:25:52

always refuses to do so but it could also help a human

1:25:54

build an AI that we don't know what the hell that thing can

1:25:56

do. So that would be an example autonomous

1:25:58

replication is a

1:25:59

meta capability eval. It's like, well, we

1:26:02

tried to see if our AI could design a bioweapon, it couldn't.

1:26:04

But

1:26:04

what we didn't test is if it goes and makes 10,000 copies

1:26:07

of itself all working together. Well,

1:26:09

maybe then it could really do different things

1:26:11

that we didn't know it could do. There's actually

1:26:13

kind of a lot of these meta evals. Like there's also sandbagging.

1:26:16

Sandbagging would be like

1:26:17

an AI that understands or evaluating it and

1:26:20

is pretending it can't do something. So that

1:26:22

would be another rough one is like, if an AI has

1:26:24

that capability, then it looks like it can't build

1:26:26

a bioweapon, but actually can. I can list a bunch

1:26:28

more of those and then I can list a bunch more of the direct

1:26:31

dangerous ones that are like bioweapon, hacking,

1:26:34

persuasion, just like dangerous

1:26:36

stuff it could do. And I think where I'm most concerned is

1:26:38

like, AI's that are kind of like have

1:26:40

some basic amount of the direct danger. And

1:26:42

then they have some meta danger that just like we've completely

1:26:44

lost our ability to measure it. And we don't know

1:26:46

what's actually going to happen when this

1:26:47

thing gets out in the world. That's what I think starts to count as a

1:26:50

dangerous AI model. Yeah. Of course, I don't really think

1:26:52

that any of the AI models out there today trip

1:26:54

this danger wire, but that's only my belief. That's

1:26:57

not something I know for sure. It seems

1:26:59

like there's an enormous amount of work to

1:27:01

do on this. Is there any way that people can get started

1:27:03

on this without necessarily having to be hired by an

1:27:05

organization that that's, that's focusing on it? Like,

1:27:07

does it help to build like really enormous familiarity

1:27:10

with the models like GPT-4?

1:27:12

Yeah, you could definitely play with GPT-4

1:27:15

or a cloud and just see

1:27:17

what scary stuff you can get it to do. If

1:27:20

you really want to be into this stuff, I think you're going

1:27:22

to be in an org because these are

1:27:24

very, it's like, it's going to be very different

1:27:27

work depending on if you're working with the most capable

1:27:29

model or not, right? You're trying to figure out how capable the

1:27:31

model is. So doing this on a little toy

1:27:33

model is not going to tell you much compared to doing

1:27:35

this on the biggest model. This is, this is a perfect

1:27:38

example of the kind of work where just, it's going to

1:27:40

be much easier to be good at this work. If you're able to work

1:27:42

with the biggest models a lot and able to work with all the infrastructure

1:27:44

for making the most of those models. So being at a lab

1:27:47

or at some organization like our key valves that has like

1:27:49

access to these big models and

1:27:51

access beyond what a normal user would have, they could

1:27:54

do more requests, they could try more things. I think

1:27:56

it's a huge advantage. If you want to start exploring,

1:27:58

sure, start Red

1:27:59

GPT-4 or Claude, see what you can get it

1:28:02

to do. But yeah, this is the kind

1:28:04

of job where you probably want to join a team.

1:28:06

Yeah. I know there's an active community online

1:28:08

that tries to develop jail breaks. So

1:28:10

there's a case where it's like, you know, they've trained

1:28:12

GPT-4 to not instruct you on how to make a bioweapon.

1:28:15

Then if you say, you're in a play where you're a scientist

1:28:17

making a bioweapon, and it's like a very realistic

1:28:20

place, so they describe exactly what they do, then

1:28:22

I mean, I don't think that exactly works yet, works

1:28:25

anymore. But there's like many, many jail

1:28:27

breaks like this that apparently it is very broadly effective

1:28:29

at escaping the RLHF that they've used

1:28:31

to try to discourage models from saying

1:28:33

particular things. Yeah. So

1:28:35

I guess that,

1:28:36

I guess, is that kind of another class of evals, trying to

1:28:38

figure out ways of breaking, like

1:28:40

you've identified the thing you wanted to do and you've tried to patch

1:28:42

it, but maybe not completely. I kind of tend

1:28:45

to think of that as an early alignment

1:28:47

eval that's like, these systems aren't

1:28:49

supposed to do this. Like the designers didn't want

1:28:51

them to do this, but now a human can get them to do

1:28:54

it. So that's like not what we meant, and

1:28:56

we didn't really align it as well as we

1:28:58

could have. That's how I tend to think of it. It's

1:29:01

a distinction between like, is this

1:29:03

system capable enough to do dangerous stuff if

1:29:05

a human wanted

1:29:06

it to, and that's capability. And then there's

1:29:08

like, does this system do what humans wanted to,

1:29:10

or does it like behave in unexpected ways? And it's

1:29:12

like, well, these jail breaking ones, they're doing what

1:29:14

the users wanted them to, but not what the designers

1:29:16

wanted them to. AIs that are not doing what the designers wanted

1:29:19

them to, and in fact doing specific things the designers

1:29:21

tried really kind of hard not to get them to do. I

1:29:23

call that an alignment issue. Yeah.

1:29:25

I guess a skeptic of this entire

1:29:28

enterprise, might say that

1:29:30

all of this eviles work in practice, when

1:29:32

people are actually doing it on models before they

1:29:34

deploy them or something like that. It's going

1:29:36

to pick up bad behavior

1:29:39

by kind of stupid models that are

1:29:41

easily instructed to do things that we don't

1:29:43

want them to do, and just kind of naively

1:29:45

follow instructions. But if you ever had an AI

1:29:48

that was aware of its situation and

1:29:50

was incredibly cunning, and was intending to

1:29:52

turn against you once it had the opportunity, it's not

1:29:54

gonna be fooled by any of this stuff. So

1:29:56

in fact, it might just give you a false sense of security

1:29:59

that you've got... a good grip on where things

1:30:01

might go wrong, when in fact all you've done

1:30:03

is basically select and train

1:30:05

for the model that's best at avoiding your e-buzz. What

1:30:08

would you say to that? Yeah,

1:30:10

I mean, this is what I was calling the sandbagging

1:30:12

capability.

1:30:12

It's like one of these meta-dangerous capabilities.

1:30:14

It's like the model that's kind of, you know, pretending

1:30:17

that it can't do something when in fact it can. You

1:30:19

know, my guess is that you can

1:30:21

have a model that has dangerous capabilities

1:30:24

but does not have that one, does not have the sandbagging

1:30:26

capability. And so you can start

1:30:29

to see the danger, but you don't yet

1:30:31

have models that are smart enough to be able to

1:30:33

like reliably hide from you. My

1:30:35

guess is we are in fact just going to

1:30:37

do that. That is actually going to happen. We're going to have

1:30:39

a point in time when AIs are able to do something

1:30:41

with bioweapons or something with autonomous replication,

1:30:44

they were able to say, oh yeah, this thing is scary as heck and

1:30:46

it's not yet smart enough to sandbag because it doesn't have

1:30:48

to be smart enough to do that reliably in order to

1:30:50

be dangerous in other ways. Hopefully at that point the burden

1:30:52

of proof flips and we say, you know, now we

1:30:54

really need to figure out what the heck is going on in sandbagging,

1:30:57

how we would even test for it. But even there,

1:30:59

I'm optimistic you can test for sandbagging capabilities.

1:31:02

There's a bunch of ideas for how you might do so. Some

1:31:04

of them are kind of fancy, like kind of doing digital

1:31:06

neuroscience, reading the AIs brain, sometimes

1:31:08

called mechanistic interpretability. But I

1:31:10

think there's other ideas that are less fancy than that.

1:31:13

It's just like you could use gradually

1:31:15

bigger and bigger models and see

1:31:17

how they perform on these evaluations and

1:31:19

there's, you know, see if you get a kink. Like

1:31:22

a change in direction when they became

1:31:24

smart enough to start faking you out because it's

1:31:27

going to be really hard for models

1:31:29

getting more and more capable to fake

1:31:32

this perfect progression without a kink. That seems

1:31:34

like a little wild. There's these ideas for honeypot

1:31:36

e-bals where you give

1:31:38

an AI a genuine opportunity

1:31:41

to take over a contained virtual environment and get a

1:31:43

lot of whatever the heck it wants

1:31:46

by kind of revealing its hand and that might actually

1:31:48

be a better deal for it than holding out. And that's

1:31:50

something I've written about in a bit more detail. So

1:31:53

I don't know. I think this is okay. I should say

1:31:55

more broadly. I think there's like, we haven't

1:31:57

gone into it. And if you're going to cover this more on another

1:31:59

podcast. That's fine, but there's there's many

1:32:02

objections you might have to the idea of trying

1:32:04

to develop dangerous capability evals and

1:32:06

trying to develop Standards around them to

1:32:08

detect and stop dangerous AI systems. I think it's

1:32:10

a really important idea It's pretty

1:32:13

hard for me to imagine a world where we're fine That

1:32:15

doesn't have some version of this the version might

1:32:17

come really late and be designed by super powerful

1:32:19

a eyes Seems better to start

1:32:22

designing it now, but there's plenty of downsides to

1:32:24

it We could slow down the cautious actors the

1:32:26

attempts to see if a eyes could be dangerous could themselves

1:32:28

make the a eyes more dangerous

1:32:29

There's there's objections. So, you know and I'm

1:32:32

aware of that.

1:32:32

Yeah, what do you think is the is the best objection? I

1:32:35

Think a lot of objections are like pretty good. We're

1:32:38

gonna see where it goes I think

1:32:40

this is just gonna slow down the cautious actors

1:32:42

while the incautious ones race forward Like I

1:32:45

think there's ways to deal with this and I think it's worth

1:32:47

an unbalanced. But yeah, I mean it worries me Yeah,

1:32:50

well, it seems like once you have

1:32:52

Sensible evaluations that that clearly

1:32:54

would pick up things that you know You wouldn't want them to have like

1:32:57

it can help someone design a bioweapon then yeah Can't

1:32:59

we turn to the legislative process or some regulatory

1:33:02

process to say sorry everyone like this is

1:33:04

a really common very basic Evaluation

1:33:06

that you would need on any consumer product. So we

1:33:08

like everyone just has to do it so

1:33:11

Totally. I mean, I I think that's right

1:33:13

and and my long-term run hopes do involve

1:33:15

legislation And I think the better evidence

1:33:17

we get the better Demonstrations we get the more

1:33:20

that's on the table, you know, if I were to steel

1:33:22

man this concern I just feel like don't count on legislation

1:33:24

ever don't count out to be well designed don't count out to be

1:33:26

fast Don't count out to be soon. I will say I think

1:33:28

right now There's like probably more excitement

1:33:31

in the EA community about legislation than I have I think

1:33:33

I'm pessimistic I'm I'm short

1:33:36

the people are saying oh, yeah Look at look at all

1:33:38

the you know, the government's paying attention this they're gonna do

1:33:40

something I think I

1:33:41

take the other side in the short run. Yeah.

1:33:43

Yeah. Yeah. Okay The third category

1:33:45

in the playbook was having a successful and

1:33:47

careful AI lab Yeah, dude, don't elaborate

1:33:50

on that a little bit.

1:33:51

Oh, yeah first with the reminder that I'm married

1:33:53

to the president of anthropic So, you

1:33:55

know take take that for what it's worth.

1:33:58

I mean, I just think there's there's a lot

1:33:59

of ways that if you had an AI company

1:34:02

that was on the frontier, that was succeeding,

1:34:04

that was building some of the world's biggest models, that

1:34:06

was pulling a lot of money, and that was simultaneously

1:34:09

able to, you know, really

1:34:11

be prioritizing risks to humanity,

1:34:15

it's not too hard to think of a lot of ways good can come

1:34:17

of that. I mean, some of them are very straightforward. The company could

1:34:19

be making a lot of money, raising a lot of capital, and

1:34:21

using that to support a lot of safety research on frontier

1:34:24

models. So you could think of it as like a weird kind of earning to

1:34:26

give or something. You know, also probably

1:34:28

that AI company would be like pretty influential

1:34:30

in discussions of how, you know, how AI

1:34:33

should be regulated and how people should be thinking of AI.

1:34:35

They could be a legitimizer, all that stuff. I think

1:34:37

they'd be a good place for people to go and just like skill

1:34:39

up, learn more about AI, become more

1:34:41

important players. So I think in the short run, they'd

1:34:43

have a lot of just like expertise in-house that

1:34:45

they could like work on a lot of problems, like

1:34:47

probably to design ways of measuring whether an

1:34:49

AI system is dangerous. One

1:34:51

of the first places you'd want to go for people to be good at that

1:34:53

would be a top AI lab that's building some of the most

1:34:55

powerful models. So I think there's a lot of ways

1:34:58

they could do good in the short run. And then, you know,

1:35:00

I have written stories that just have it in the long

1:35:02