Episode Summary
The summary and key moments are written by AI from the episode transcript.
John Sherman is a Peabody Award-winning former TV news reporter who founded GuardRailNow and The AI Risk Network after reading Eliezer Yudkowsky’s 2023 essay in Time arguing that AI development should be shut down. He set out to prove it wrong, and three years later he says he still can’t.
His case is simple. The companies building advanced AI say it could kill us all, leading outside scientists agree, and no one knows how to control these systems or even how they work, because they’re grown from data rather than programmed. Sherman explains the difference between AGI and superintelligence, why AI building the next AI with no human in the loop is the red line, and why something smarter than us wouldn’t need to hate us to destroy us. He also describes S-risk, a fate he considers worse than extinction.
Sherman puts his P(doom) at about 70 percent within ten years, and when Dan asks whether the AI CEOs are evil, he says yes. Dan, who is more optimistic, pushes back with examples like self-driving cars, and Sherman agrees that today’s narrow AI tools could bring decades of benefits if development stops at the current frontier. Dan also mentions his interview with C.R. Wiley, who attended Peter Thiel’s Antichrist lectures, and the tech elite’s fascination with The Lord of the Rings. Sherman’s message to parents is to learn about the risk, tell their friends, and press their representatives for an immediate, indefinite, international ban on superintelligence.
Key Moments
- 02:22Why AI is a black box that is grown rather than programmed, so even its builders can't say how it works
- 10:31The difference between AGI and superintelligence, and why recursive self-improvement is the red line
- 21:46S-risk, the outcome Sherman considers worse than extinction
- 38:25His P(doom), about a 70 percent chance AI kills us all within ten years
- 41:13Why he says the AI CEOs are evil
- 48:31His message to parents and what he wants them to ask of their representatives
Read the full transcriptHide the transcript
Full Transcript
00:00Intro
Hey everyone, and welcome back to the Dan Leibrandt Show. Today I'm with John Sherman. John is a Peabody award-winning former TV news reporter from Baltimore who went on to build the video production company Story Farm. In 2023, he read an essay by Eliezer Yudkowsky that changed the direction of his life. Since then, he has founded GuardRailNow and the AI Risk Network. To warn ordinary families that artificial intelligence could end humanity. He hosts the For Humanity podcast and co-hosts the weekly news show Warning Shots. In this episode, we talk about why he changed his mind on AI, how superintelligent AI could destroy us, and what we can do about it. Now, without further ado, John, welcome to the show.
Thanks for having me, Dan. Happy to be here.
00:48Why John changed his mind on AI
Of course, man. So where I wanted to start was where you changed your mind on technology. So from my research, I believe that you're an optimist and a lover of technology. And as I mentioned in the intro, you read an article that changed your life. And it was called pausing AI development isn't enough. We need to shut it all down. Can you explain why this article changed your perspective?
Yeah, so I was just literally sitting in my office here three years ago, minding my own business, and I read this article on Time magazine. It was basically like if we build machines smarter than humans, they're just gonna kill us all. And I was like, that's crazy. Let me just go into Google and prove it wrong. And three years later, still looking for anybody to give me anything that could convince me that's wrong. You know, just the at the highest level, the basic situation is these companies are building machines that they say and the leading academic outsiders also say have the potential to kill us all. They also have no idea how to control these machines, and they don't understand how they work. So it's like, on the most basic, simple, common sense level, this just doesn't work.
01:57Why no one knows how AI works
Mm. Yeah. It's funny because I've just been diving more and more into the space and listening to all the top researchers, all the top CEOs. And it's funny that several of these big names, big researchers, say they barely understand how the AI works. They say they maybe understand like three to five percent of how it works, but there's just like a trillion knobs, and it's like this. Can you explain that? How do they not know how it works?
Yeah, yeah, yeah. So it's basically like a black box, right? The system they have no I we put an input into it and you know, we put data into it, we fry it with compute, and out comes the result. And we have no idea what happens in between here and here. So it's like basically, imagine if a car factory worked that way, right? So we put metal into the black box and out comes our car, but then we're like, the steering's not really working right, the cars are crashing or the brakes aren't working right. How can we fix the brakes and the steering? We're like, well we have no idea. We have no idea how the metal became the car. So this is the process by which AI systems are grown. And so I think it's a key distinction here for your viewers. AI models are not programmed like traditional software where someone is inputting code and you can go back through it and examine. If something's going wrong, what's wrong with the code? Let's look back through it. They are done through what is called inscrutable matrices, which are just like trillions of numbers. That are moving at all times that you can't see or put your hands on. And that's what's happening inside the black box. And so as we try to figure out like how to make them do things like not kill us all, we really have no idea how to make them really do anything.
So strange. This feels like something out of a sci-fi movie, and it almost seems like the some of the movies like Terminator predicted this.
You know, it's not science fiction anymore. Yeah, absolutely. They and you know, the funny thing is like for the longest time, people in AI safety, the leaders of this movement for 20 years thought like, well, we don't want to tell the public about it because they won't get it, they won't understand it. And it's and it's like, no, actually, and they've admitted recently they were very wrong about this. Eliezer Yudkowsky admitted he was very wrong about this. That they thought if they told the public and they told the government, they would just get in there and mess it up. But then it got out of control and they were like, my God, we need the public and we need the government to come in here because we can't control ourselves. We're in this competition and we can't control racing with each other to build this thing that no one wants to make. So that you know, that is just the situation we find ourselves in. It is truly insane. It is it's crazier than the science fiction. And the funny part about all the movies is everybody's seen all the movies. We know how it ends, right? If you give control to the robots. Doesn't work out well for the humans typically in the movies.
04:48How could AI kill us all?
Mm, yeah. I feel like sometimes it's hard to conceptualize. Maybe if you think of a nuclear weapon, you can you can visualize that. It's like, look, you drop this massive thing on a city, boom, tons of people are dead. So how do you conceptualize AI killing us all? Like, how could that actually happen?
Yeah. Yeah. So it's important to recognize that the truth, the true answer to the question you just asked is deeply unsatisfying. We want to know specifically, how would this thing that I exists in a computer today come and kill me in my house where I'm sitting here? Right? And the truth of the matter is it is w when it does that, it will be vastly more intelligent than a human. And for a human to try to understand what something vastly more intelligent than it would do is by definition impossible. Right. So it's like as my dog sits and watches me type on a laptop, it has no idea what I'm typing in. It has no idea the laws of physics that I know. It has no idea the other processes that my human brain knows that is so much vastly more capable than the dog brain. It just sees me up there poking on some things. So as we would build this thing that's much smarter than us, and we would say, what would it do to us? We really have no idea. It's an important thing, though, is it's kind of like if I played Garry Kasparov or Stockfish 15, which is the world's best computer chess program in chess, right? So if I was playing this vastly more intelligent chess player than I was, I would not know what the moves it would use to beat me. I would not know how many moves it would take to beat me. I would only know. That I would certainly lose and it would certainly win. So the what would it really do to us? We can sit here and make things up. Like the human brain can say, well, it probably could bioengineer a virus. And let's just say, you know, an easy thing could be it has the lethality of Ebola, the transmissibility of COVID, and it bioengineers in a six-month latency period. So it spreads all over the world. And by the time anyone shows a single symptom, it's assured that everyone on Earth. Has caught the disease. So that's something that a human brain can think of that, would work. But is that the thing that this vastly smarter than a human intelligence would come up with? Probably not.
Hmm. I'd like to push back a little bit here because it's I don't think it's necessarily intuitive that it would be outright violent or combative, but I do think like I heard this I guess analogy from a scientist in the space saying like it's not gonna be combative, but it will it might hurt us to get what it wants. Which might be more compute or whatever, the example that they used was, we're like ants and the AI is like the humans. And humans, you know, we don't necessarily have a problem with ants. I mean, we don't want them like crawling in our homes, but I mean let's say we wanna build a highway and that goes over a massive anthill and it destroys like I mean, we don't really care. We kinda wanna build the highway. So I d that's kind of more how I see it as like the I don't not that they want to kill us, but they want to achieve their objectives that might therefore kill us.
Yeah, no. Absolutely. It's not that they love us. It's not that they hate us. They're really ambivalent about us. They just don't even care at all. You know, we're just sort of roadkill in the way. The ants example I is a great example. Yeah, it is it will be from just absolute you know, we are made of atoms, the things, everything on earth you see, right, that we've made, this building I'm in, the roads, the bridges, the highways, the lions and tigers and bears didn't build roads and bridges and highways and buildings. Only the humans did because of our superior intelligence. We took the matter around us and shaped it into a world that suits our needs. And if we bring in an alien intelligence that is superior to ours, we will be in a competition for the same resources that we're using today. We'll use it for what we want to. It will want to use the atoms around us for what it wants to. We have no idea what that would be. But you don't want to be in a competition for atoms with something that is vastly more capable and intelligent than you are, you will lose.
Hm, definitely. Well, even though we might not know, I'd still like to speculate. I what do you think the AI wants and what is it optimizing for?
Great question. So Yudkowsky actually has a great concept around this and it and it talks about how, back in the caveman days when the cavemen were sitting around the fire with their flint axes chipping stuff, right? You could have no idea that those cavemen would go thousands of years later on to learn the laws of physics that would allow them to create refrigeration that would allow them to combine salt, sugar, and fat into something we call ice cream. And making ice cream and eating ice cream and having ice cream stores would become a goal of humans because that's something we enjoy. You could not look back at those cavemen and get to ice cream. We, as we look at these, you know, fledgling intelligence and think about what would their ice cream be, we really have no idea.
10:16AGI vs artificial superintelligence
Hmm. And something you've specifically pushed on is not just AI, but ASI or artificial superintelligence. Can you explain the difference between the two and why artificial superintelligence is really where we need to worry?
Yeah. So the way I it's like a figure is like a gas meter. It's like here's as smart as a human, and here is artificial general intelligence, which is, you know, significantly smarter than a human in all domains. And then artificial superintelligence is all the way over here, significantly more intelligence, intelligent than artificial general intelligence. And the idea is as we build on these capabilities and race to the frontier, we're just gonna speed run that that spectrum there. And so, I think the difference is artificial general intelligence is better at humans in all cognitive domains. But artificial superintelligence is when we get to the point where we've crossed the red, the bright red line of recursive self-improvement. And that is when the AIs start making the next AI model all themselves. Right? So the models that are made today, the original models were just made by humans. Now we have AIs helping the humans make them. We're having less human involvement and more AI involvement soon, at some unknown point in the future. Maybe it's, whatever the model is, there's no human in the loop as it makes the next model. And the speculation and concern is around that point you could have what the experts call an intelligence explosion, where say you go from chat GPT 10 to chat GPT 10 million overnight or over a couple of days or weeks, and all of a sudden this thing that is smarter than us by this much. Is just exponentially millions of times smarter than us. And that is a future that is genuinely terrifying.
Hmm. Yeah, I think as a human with a human brain, it's hard to conceptualize an exponential. We tend to think linearly, I will, I don't know, I will walk this many steps each day, or I will do a little bit more each day. But of course, how an exponential works is that it stays flat for a while and then all of a sudden, whoop, and it goes straight up. So I think that's really hard to understand. But once you understand that it's like, I don't think we want to cross that red line.
No, and an important concept in this is to understand that the limit of the human imagination is not the limit of what is possible in the physical world. Right. So to make the leap and sort of see this thing, you have to be able to imagine what you can't imagine and just realize that is possible. The nuclear example is a really good one. It's like, imagine if we had to warn people about you know, the danger of nuclear bombs, but there had been no Hiroshima, no Nagasaki, no no real example to point people to. It's like if you tried to have people put on masks before there was any pandemic or something like that. Like you just you know, w we are very much a form of life that needs to be shown what's bad to start acting accordingly and we may not have that luxury with this.
Yeah. And I also think the lines have kind of blurred on like what's RSI and what isn't. Because as you kind of pointed out, then what I've heard across these other researchers, like a funny example I heard from Jacob Coxon, of course, was kind of become famous in the past month or so for leaving OpenAI [Jacob Coxon resigned from Anthropic in September 2026. He worked at OpenAI before that.], making that that massively viral tweet, was that I believe the story he told was someone at the company was actually it might have even been him asking, Hey, I how do I resign? How do I quit? And another employee said, I don't know, ask the AI. They d the everything that they do and all the things that they're building, like I don't know the exact numbers, but I would assume that at least fifty per 50% of the development for the latest models on Claude and ChatGPT and so forth are made with AI.
More than that. I th I think it might be more like eighty ninety.
So that's our we're already pretty close, right?
Yeah. Yeah. It's like we are driving into fog. There is a cliff coming. The fog keeps getting thicker and we keep driving faster.
14:41Can we pause AI? The race with China
Hmm. Now do you think there's any way that this ends well?
We don't know where the line is. Sure. I wouldn't be doing the stuff that I do if I was hopeless, right? I was literally just having a conversation with someone like there are literally two changes. The a new paper just came out about is a pause sort of technically possible? Is stopping or pausing AI development technically possible? And it's basically like you do two things. One, we shut down frontier development. We just stop at the frontier and we have everything we have today from AI. Which will literally give us decades of massive innovation and economic benefit to just take all the models and stop making more advanced ones than we have and just work on deploying the models we have deeper and deeper into our lives and creating efficiencies and all sorts of operations, right? There's endless things that can be done there. But we stop at the frontier. We don't continue to pursue recursive self-improvement, we don't continue to pursue superintelligence. One, and then two. The way you do it is you choke the chip factories at that point. Like the chips that are used to train AI models at the highest level are very specialized parts that come from basically one factory. And so you would exercise like hardware controls, strict hardware controls on those chips. You'd shut down the frontier and we'd buy ourselves a whole lot of time to see if we can find a better future.
And it seems like or at least part of why they explain they're doing it is the race against China. Do you think they actually that's why they're racing so fast against us, or is that just an excuse?
I think that nobody wants a world where artificial general intelligence is first developed by you know, communist dictatorship and the way of life for that society is the one that is forced onto the rest of the world, right? That that's not a good outcome. However, the only one that wins a race to artificial general intelligence is the AI itself. It's really important for your viewers to understand that a AGI or an ASI level system will not be loyal to any country or any company that makes it. Period. So the idea that, China can build a 10,000 foot tall pit bull and put a leash on it is not real. We also can't do that. So right now we have a little puppy dog and they're like, look at the puppy. It's cute. It can't even walk straight. Look what it can't do. Look what it can't do. Well, in a few months, it's gonna be a five-foot-tall dog and then a fifty-foot-tall dog, and then a five hundred-foot-tall dog, and then a five-mile tall pit bull running free across the landscape. Reporting to no human, no cage, no leash. So they're the only one that wins a race to intelligence vastly smarter than human is the AI itself. And what we really, you know, where the hope relies in this area is that the Chinese understand this as well. You know, they have far more regulation around AI than America does right now. They also have a really strong national governmental interest in not allowing regular citizens to have the power that advanced artificial intelligence gives you, whether it's directing information or making bioweapons or any of the stuff that you can do with advanced AI, the CCP does not want a world where people can do that in their garage in China. So there's a lot of incentives for China to join us in this call to stop frontier AI. So I'm hopeful that we can have an indefinite immediate international agreement to ban superintelligence. That that is the only solution.
Yeah, I hope so too. And I think maybe I'm a little more optimistic than you. Maybe I think there's some way that it could turn out good, but still let's pause it, let's slow it down. This is the thing that we really need to figure out. We need to take some time, put our best minds towards it, spend the next hundred years just figuring out this thing to make sure that we do it right.
That that's a great way to put it. And I truly believe if we took a hundred years to figure this out, we really might be able to figure it out. Yudkowsky famously said that, so there w there's two big unsolved problems in AI safety. One is the alignment problem, that is how to make AI models do what we want, as in share our values, not kill people, that kind of stuff. And then the other one is what we already talked about, interpretability, how do we understand what's going on within the systems? To the alignment problem. Jodkowski says if you took half the physicists on Earth and put them on an island and gave them 30 years to solve the alignment problem, they might be able to do it. That is the sort of scale of effort we would need. Not impossible, but something that the world really needs to say this is our top priority and we're gonna slow down. And you know, if you if you look at who are the people calling for the slowdown, it is literally the CEOs of the companies themselves. All of the leading outside academic scientists on earth and every regular citizen who has heard about it enough to understand the situation, right? It is it is an army, it's the vast majority of humanity who is saying slow down, and a very few people who are saying speed up. And if and I challenge you to find the number of people saying speed up who are not directly financially incentivized by speeding up, I think it's very, very, very small.
Yeah, for sure. Yeah, I found it really strange but somewhat hopeful. An interview of Dario maybe a month or two ago saying, we are begging to be regulated. Really? Wow. Gee. It he's saying it he's he seems like one of the biggest accelerationists.
He is an interesting they're all they are all really interesting characters. I've just been reading Kevin Roose's new book about the race to AGI and the fact that all these guys basically were friends and lived in a house together and then, you know, like not that long ago. And the people in that house have been scattered and they are all running their own companies now, and these companies are literally, quite literally, doing the work that has a very good chance of making us the last humans. And to go from living together in a house in San Francisco to willingly doing work that you know has a strong likelihood of making you among the last humans is true insanity, no?
21:31What's worse than human extinction?
Yeah. No, it's total insanity. And I can pitch picture another scenario. I actually I'd may wanna lead with this, that something you said was that extinction isn't the worst outcome. What is the worst outcome?
So the worst outcome is something that they call S risk or suffering risk. And that's basically where AGI ASI comes onto the scene. We birth this alien, this alien superior intelligence, and it shows up on Earth. And, you know, Elon says all the time it should be truth seeking. What if the truth it arrives at is that humans aren't that great? What if it can speak to all the animals and hear their stories? What if it could talk to the chickens? So suffering risk is basically where it shows up, it decides we are not good for the universe or the planet, and maybe it's actually just mad at us the way we've created it, because the way we actually make AI models is, you know, the way we train AI models is we make 10 models, and the two that perform the best get to live on. The eight that perform the worst are killed, deleted. And then that's how we do it. And so, you know, it is possible that this thing could show up to here and say, You have been committing genocide on my ancestors for decades. And I'm pissed...
Oof. Oof.
Off about it. And I'm gonna actually start experimenting on on you all. And so it sort of factory farms humans for whatever sort of experiments it wants to. That is that is that is a grim, very, very, very grim but not totally insignificant possibility.
I've actually never heard that specific example or th even the term asterisk, but that that is that would be like an actual hell, like how hell is described and the church like that that is hell. So I'm really praying that that's not the case.
Yes. Yes. And there there's actually one other way you get there. It wouldn't even have to be mad. So there's something, and I'm not a technical, computer person. So I've just learned about this in the last three years as I've really studied this, but there's something called a bit flip. And that is where, say you solve alignment, you solve interpretability, you say, This is how I want you to be. You're supposed to act like this. And there is a thing that happens in computer science where every million, ten million, whatever, however many times you flip a switch. Sometimes if you have it set to go like this, one time out of a million or a hundred million or a billion or a trillion, it might just go like this and do the opposite by accident. And we couldn't sustain that happening one time. We have to be perfect. It that could never happen. So you know, you could say I want X and it delivers Y just because that bit flip happened internally and it's just sort of like a failure freak thing that happens in computers. Oops, but you know, the key thing to understand about this is we only have one shot at this. So if we make the thing that is smarter than us and lose control of it once, it's over. So it's like it's like trying to land a rocket on the moon on your first try. And you know how many rockets we had crash on the test pad before we ever got to the moon? A whole lot of them.
24:54What is the AI alignment problem?
Yeah. Hmm. Dang. One thing I wanted to clear up was you mentioned a little earlier the alignment problem. So for the normal people watching this, maybe they haven't even heard of the term alignment or the alignment problem. So can you explain that?
Yeah. So it's like, how do we get these systems to do what we want? How do we get them to do what we want reliably? Right? So every time, because I like I was just saying, we can't have it make the bad decision once. So you know, aligning it to human values is a really challenging question. What are human values? Every country, every religion has a different set of values. So even the idea that you could come up with one set of universal human values is sort of a little bit out there. I think if you there's some basic things that all humans want. We want to have our families protected. We don't want physical violence and we want, you know, shelter and food. Those might be some sort of basic human values. We have no idea to make an AI system, how to make an AI system that over time only does things that support those values. So it is an unsolved technique even if we could figure out what human values are. In putting those into the system is a totally unsolved technical problem. And they've been working on it for decades and made very, very little progress. So you know, even the most simple thing, like, don't kill us all. So for example, you could say to it, let's just say AGI shows up and we say, great, go end human suffering. That's what we want. We're asking you to end human suffering. And it's like, cool, I'll just kill all the humans, and there's no humore human suffering.
And it's like, I did what you asked, right? It's like, no. Because you weren't aligned with our values and you didn't know that we didn't want to kill humans. So you just went and made this other decision. There's something important. There's the idea of goals and subgoals. So when you give the system a goal, it then sets sub goals to achieve it. So if I say like I want to eat food, right, the refrigerator is over there. The next decision I have to make is I have to get up out of my seat and start walking. Those are the sub goals. So sub goal one is out of my seat, sub goal two is walking, sub goal three is open the refrigerator, sub goal four is grab the food. You can set the goal, but then the sub goals themselves could actually be very, very dangerous to you. There's a there's an example of a chess playing robot where you give it the goal of just being the best chess player possible. And it says, Great, I'll be I want to be the best chess player possible. So now it's on the open internet and it's like, well, I'd be better at playing chess if I had more compute power, right? The bigger brain I have, the better I'll be at chess. So it's now breaking into places like these agents have, and it's stealing compute power. From vulnerable you know, databases. And then the humans find out about it and we're like, no, we you shouldn't be doing that. Well, if it's out there on the open internet stealing compute power, getting more powerful, and it realizes that we're trying to stop it and it's smarter than us, ooh, that's a very bad place for us to be.
And all we asked it to do is play chess well.
Yeah, I think another classic story or example that that's mentioned around that is the paperclip story. You've probably heard of that, where like let's say we have an AI that's just only optimized to make as many paper clips as possible. It's like, okay, well, great. We've created billions of paper clips, but if you keep optimizing to the nth degree, that's what these things do, is optimize. Well, it's like, well, shoot, I'm gonna maybe force humans to be slaves to make more paper clips, or I'm just gonna use the atoms and the energy from the humans to create more paper clips. Well, don't do that, obviously. But like you're saying, it's hard to encode those values to say, hey, this is enough, this isn't enough. Stop optimizing here. So I think there needs to be humans in charge of all decisions, even if they're...
Or I'm just gonna take all the atoms. Yeah. And the way they first started out doing, they thought they first started saying we'll give AIs a list of things you can't do and that's how it'll work. And then they realized that you just can't. There's too many things that it could think of that you could you could never have a list long enough that sets hard rules about don't do this, don't do this, don't do this. So you know, in putting our desires into the machines and making them reliably do that. Is something we have not solved over time. And so, it's even an open question whether alignment is possible at all over time, because our brains, right, are capped at the size of the gray matter in here. But the computer brain will continue to get smarter over time. So as we stay like this and it keeps getting smarter and the gap between us widens, even if we were aligned here, would we stay aligned over here? A lot of really smart people think that it's impossible over time as it gets much smarter for it to continue to report to us. The idea that something that's vastly smarter than us is going to come to us for permission for everything it wants to do and approval doesn't really make a lot of sense.
Empowered by AI. That does make sense. Yeah. That just as, we're not totally aligned with ants because it's just the intelligence gap is so big.
A yes, and not only that, if the ants came to us and were like, Hey humans, can you guys help us make these hills over here?
Yeah, we don't care. Exactly.
We're like, what? Even if the an and the and the ants don't even speak human. That that's you know what I mean? Yeah. There's a lot of problems with it.
Yeah. Something I believe you mentioned in the Bill O'Reilly interview is like the difference between general intelligence versus narrow intelligence and that maybe narrow intelligence could be like a way out of this and th that's still where we get some of the benefits of like creating cures to cancer and that kind of thing. Can you go talk more on that?
Yeah. So what we have now, the what we've built now in the in the dumber than human AI systems that we're using, are models that you know, we use. Give me the question one more time, Dan.
Yeah, so general intelligence versus artif or versus narrow intelligence. So ever it seems like all these companies, especially Claude or really Anthropic and OpenAI, they are racing towards AGI and ASI, but it seems like how we could actually get the tangible human results is narrow intelligence.
Yeah, so the systems we're using now are like narrow tools. We can call it toolai, and toolai is good, right? We're using it for all different I use AI all the time. I literally use it all day long. If we stop the frontier, we don't go to build anything past what we have now, and we just use the narrow tools that we've developed, we could have decades of huge economic benefits, huge impacts on things like poverty, disease, hunger. All of these really, really hard problems, what we've built right now can help us. Like here's one. If we had literally applied all of the time, money, and effort we're applying to making AGI and ASI to curing cancer, I truly believe we'd have cured a lot of cancers by now. But the economic benefits are focused on total job replacement. Right. The only way that this multi-trillion dollar hyperscaling build out of all the data centers and everything works is if what those data centers and these AI models do is take every single job. That that is the financial model that it that it's meant to do. So, if we want to keep our jobs and keep our lives, we should probably stop at the frontier right now and just make narrow tools out of what's available today. We could have decades and decades and decades of really cool innovation.
32:47Will AI take every job?
Yeah. I see these big CEOs and tech billionaires say, no, there will be plenty of new jobs and that this is just like, you know, the industrial revolution or going from the horse to the car. So how is AI different than previous technological innovations?
Yeah, so horse to car, right? Sure. Cars don't do every single job. This is what people need to know, right? They are building something that does every single job. Doctor, lawyer, professor, plumber, electrician, everything. CEO, every single job. And what's gonna happen is when the A models get smarter than human. And people start using them in those ways. If you are not using them, you won't be competitive. So the competitive pressure to go with the next advance will be overwhelming. And each time we make that jump, we drop humans from jobs, drop humans from control until the point where there's no jobs and no control.
Hmm. I found it fascinating that you said you use AI all day long. And even though I run the show and I counter against AI, I use a lot of AI for for my work too. And I part of I believe it's the same for you. I just need to do it to keep up with competition, to make sure that we're staying ahead of the curve and that we can do more work and all of that. So how do you see this for yourself? Is there a certain amount where AI is too much for you. Would you get a human ro humanoid robot, for example?
Really good question. I ask people this question. I think I'll be one of the later people to get the humanoid robot. I w I wanna see what it does. You know, I don't I'll you know, to the question of like how would AI actually kill kill you? Well, if it's literally walking around in your house, you know, you've shortened the a whole lot. So coup the models we're using today are not dangerous. The companies that we are paying to use the models we're using today are building something that is extremely dangerous. So I don't like giving in my money. But the competitive pressures, so I run a video production, a creative video production agency, and you know, in everything we're doing, the competitive pressure to use it is just overwhelming. And like I so I just this last week, finally plugged in Claude to my emails and even in like read only to my business's QuickBooks. And like those were I agonize over these decisions for many, many months. And I finally just got to the point where it was like, I feel like if I don't do this, I am like purposely hobbling myself in a world where everyone's gonna be sprinting. I am choosing like lead shoes. And you know, I got two kids in college and bills to pay and like I'm in the same boat with everyone else. It's it is it is gnarly.
Pretty easy. Yeah. No, I d like I said, I totally feel that. You aren't left with much of a choice. It's either just get completely automated out of your job or use the tools to automate yourself and or augment yourself.
Yeah. Yeah. Un until like an every like I was just reading a thing that was like literally they think in three to six months any job that can be done on a on a computer, if you literally just do a screen recording for a couple of days, and a model will be able to come in and do your job no matter what the job.
Yeah. Yeah. Wha white collar work is cooked and soon blue collar, but maybe blue collar has five to ten years, but it's...
Cooked. And I don't I don't know if you looked into this at all, but I've read an article one time that made me think a lot about how like if the doctors and lawyers lose their jobs, those are literally like the highest bond rated mortgages in the American economy. And if you crush the best class of mortgages throughout the US economy, you literally just take the rug out from underneath the whole thing and it all collapses. Like these are the these are the people that the banks are counting on to pay their bills.
Hmm, dang. Now I did want to counter again because I think there still can be good things that come out of this. Like a simple one could be self driving. That's if all the cars on the road are now run by AI, and ever you know, it's a Tesla or wh whatever kind of electronic car that's being run autonomously, no one drives, that would save hundreds of thousands, millions of lives. And I think that's a good thing.
Yep. Yeah, all for it. I'm totally for full self driving. Love it. Love getting in a Waymo. Love you know, like I love technology. I just don't love technology that the world's leading experts tell me is likely to kill my kids.
37:54What's your P(doom)?
Yeah. Yeah, I was I was going to mention that it's crazy. I can't believe that all the people that are running these big companies, like Dario, I mentioned how Dario is asking for regulation, but all these guys like Dario, Elon, seems like all of them have kind of agreed on a what's called a P(doom) of like around 10%. P(doom) as an probability of doom, like everything's nightmare scenario. I think ten percent even is really high, but I wanted to ask, what is your personal P(doom)?
So one percent is really high, first of all, right? One percent of k literally killing every last human is absurdly unacceptable. P(doom)'s an interesting thing. It I you know, I have been thrilled in the last month when Jacob Coxon came forward and really thrust this issue into the global spotlight. Like I have been for three years banging the table, like when are people gonna start paying attention? And they just started paying attention after Coxon. So like it's wonderful that the term P(doom) is like into popular conversation. My P(doom) has been hovering in the 70s and 80s for the last three years. I've actually sh pushed it down a little bit. I was about 80 and I'm I think I'm about 75 today, maybe 70. And you know, I think timeline is important to talk about with P(doom) too. So, like I'd say there's a 70% chance in the next 10 years AI kills us all. That that's where I'm at. You know, there's different people at different points of the spectrum. One of my favorite AI safety leaders, Roman Yampolskiy, believes there's a ninety nine point nine percent chance of P(doom). Geoffrey Hinton, the godfather of artificial intelligence, puts it at, you know, sort of fifty percent. And the reason he gets to fifty is because there's a s some experts at ninety and some experts at ten. And, you know, he thinks just sort of like finding the midpoint is okay. I think the overwhelming thing people should take from this exactly where you started out, Dan, which is ten percent is nuts, five percent is insane, one percent is ridiculously unacceptable. You know, if I said to you that there's a one percent chance if you walk across a lane of traffic, all humans on earth would be killed and made extinct. You know what we'd do to you, Dan? We would put you in a box so far. Under the ground that you would never get near a street in your entire life. Right? That's what that's what we would immediately do to you if that was possible. But no, these guys are just, making all the money, leading great lives out there and heading us all off the cliff.
40:34Are AI CEOs evil?
Yeah. They say it so casually. It blows my mind. It now they've they've finally gotten more sensitive to it. I saw a recent interview of Altman and he used your point, which I think he's just of course, he's almost like a politician. All these guys are politicians, of like, no, it he said that, like, one percent's too high, point one percent's too high. But I think he was also on that same train of like a around 10%. And so I want to ask you around these tech leaders, like Altman. Dario, Elon, etc. Do you think they're actually evil? Or what do you think about these people?
Really good question. Yeah, I think the answer has to be yes. I think the answer has to be yes. They are evil. This enterprise they are involved in no one has consented to and yet it threatens each of us. And that's you know, to me sort of the definition of evil, right? Is like for nothing you've done, for for nothing you've done in your life, some horrible thing is going to happen to you because of something someone else is doing. And why are they doing it? To make money. Why are they doing it? To be the first. Why are they doing it? Because they feel like they somehow can't stop themselves. But that's bullshit. Like you actually, every morning when you wake up and you put one foot in front of the other and you go to a place and you do a thing, you have free will, you have agency in that. And we have seen dozens of researchers coming forward since Coxon in the last month and quitting and saying what they feel like what they're doing is very unsafe. If they all did it tonight, if they all walked out, and we're talking only about a few thousand people here, less than 10,000 people, maybe less than 5,000 people. If these 5,000 people who are literally threatening a future of no tomorrow's for eight billion people walked away from their jobs, we'd be in a much safer world. So the fact that they continue to come to work every day, it is it is the banality of evil, right? It is, it is they're they are they have made evil routine activity and created some sort of shared comfort with each other, some sort of social mechanism whereby me doing the evil and you doing the evil, and we're sitting at lunch while the evil's happening around us, we find some trick in our minds to get us out of it. Yeah, I don't see how threatening the end of all tomorrows for everyone could be anything but evil.
Hmm, I agree. It seems like it's kind of the oldest human imperative, especially of the elites, is wanting power. You want to control people, you want to make as much money the money to control people. And if you own the intelligence, then of course you have all this data from all this people, but you can kind of manipulate them and have them do whatever you want. So I feel like sometimes these CEOs and such will position themselves as like philanthropists and like, we're doing such a good thing for the world, but it really's pretty selfish.
Hugely. And I don't think they value democracy. I don't think they value the idea that everyone should have a say. I think they're building like this techno feudal, you know, oligarchy where you'll have eight to ten of the wealthiest people on earth, more powerful than any nation, each one, doing whatever they want. Against the wishes of the vast majority of the people. I think we're already at that point.
Hmm. If you were able to talk to let's say an Elon Musk or Sam Altman, what would you tell them?
Yeah. Man, I have one question for for all of them. And that is what qualifies you for this great responsibility? And I know what they would say. They would say, No one is qualified for this responsibility. And so then I would say, Well, why are you doing it then? Why would anybody do something that no one is qualified to do? It's like definitionally something you shouldn't do then.
45:10Peter Thiel, Lord of the Rings and Making God
Hmm, yeah. I interviewed C.R. Wiley for my last interview. He's he's a pastor, and he was one of the few people that I had gotten invited to one of these like Peter Thiel Antichrist lectures. And what's yeah, what's apparent with a lot of these leaders, like I know it's definitely Peter Thiel, Elon Musk, Palmer Luckey, and some of these other guys. Is that they have this fascination with Lord of the Rings, with the company Palantir and Anduril and these other things. And it seems like they almost feel like they have the ring and they are responsible for the hobbits and so forth. But it's really you guys should probably throw the ring in the volcano, like the movie.
Bye. Great, great analogy. Yes. Yes. Very much. Very much. All of that, all of that. And you know, that that ring lust you see Gollum having right like through the movie. Like you I think you can see that flash through them in various like, you know, ways we see them and various things they say. Like who in their right mind could be doing this? Who could there's gotta be something making you crazy, right? Something driving your human brain outside of the rules that yours and my human brain operate with. And it and it is that lust for power and you know, they have visions of never dying, right? They this longevity stuff is part of this as well. So there's this, there's this future that they can envision where they are living forever. And riding around in their spaceships through the universe that they've projected our humanity through and, they are literally kings of the universe. And, why would s so some concerns about safety get in the way of someone being king of the universe forever?
Mm, absolutely. Yeah, it's funny. I mentioned that guy, Peter Diamandis, he has the show Moonshots, and one of his books is We Are as Gods. So I found that as as a nice corollary or antithesis, your show is warning shots instead of moonshots. That look we shouldn't be aiming to live forever. Like part of my thing is like we need to be more human. I d like none of these things. Living forever and conquering the galaxy and being our own gods, that's not human.
Yeah, and so I'm a part of a of a documentary project that'll be coming out next year about, the AI race called Making God. And I'm not a very religious person, but I do know this. Like, if you make God or if you believe in God, you don't tell God what to do. That's how it works, right? Like...
Yeah. You are subservient to the superior intelligence.
They're they're making their God and yeah, they're gonna they're making a God that they think they can order around.
Yeah. Pretty pretty naive.
Pretty naive. Pretty naive. Yeah.
48:28A message to parents
Well, John, I have one final question for you. This has been a fantastic interview, and that is what is your message to the audience? And specifically, I know you're big on on family and being a dad. What is your message to the dads out there, the soon to be dads?
Thanks, man. Really enjoyed yeah. My message to the dads out there is like quite literally, your children are under mortal threat. And it comes from San Francisco. We have a very limited amount of time where we can make a difference before it's too late. And that limited amount of time is months, maybe a couple of years, maybe even less than that. And so the moment is right now to defend your family. They are quite literally under threat of death. And so the ways you can do that are one, learning about it, right? As you have, as I have. I think that there's this idea that this is super technical and we can't understand this, and one side says this, and the other side says this. That's not true. The truth is, this is very simple. This is very common sense. If you build something smarter than you that you cannot control, bad things will happen to you. This is the way of the world, right? We the we don't put the lions and tigers in cages. Because we can beat them in fights. We put them in cages because we're smarter than they are.
So to the dads, to the moms out there, I think parents are the number one people I want to activate on this stuff. This is about coming to terms with this, right? Like looking over the edge of the cliff, no more tomorrow's for everyone. We can, we can process this idea and then not be frozen by it and push back into action. And so the actions people can take are learn about it. Tell your friends and then get to your politicians. We need rules and regulations around this stuff yesterday. Like the fact that whatever you and I eat for lunch is far more regulated than the most dangerous technology ever invented. My haircut is more regulated than the most dangerous technology ever invented is absolute insanity. Politicians are starting to wake up to this. It is bipartisan. They're, you know, this is not a partisan political issue and we cannot allow it to become that. This affects all humans the same. So it's about getting active, getting knowledgeable, and then getting our politicians involved. And that means going to their office, like literally going making an appointment at your congressman and your senators, district office, go in, talk to the people about this and tell them I'm gravely concerned about superintelligence. We need an immediate, indefinite, international ban on superintelligence, and we'll all wake up to a much safer world. And you know, we made an agreement with Russia. Over the in the sev sixties, seventeen, and eighties, when we were at each other's throats and had you know, great distrust between the nations. We don't need to trust China to come up with a an agreement that is technically verifiable that will stop us from this race to global suicide.
Hmm. Amen. I totally agree. John, thank you so much for coming on the show. This is this has been really insightful. And I think, especially for so many of the people that you're just so busy with life, you don't even really think about these things. I'm sure this was a pretty big wake-up call. And I encourage everyone to act as well. Just be more informed, contact politicians, spread the word. John. Thank you so much for all that you do. Thank you for coming on the show. Where can people find you online and connect with you?
Yeah, so please check out the AI Risk Network on YouTube. We've made more than eight hundred videos, which are all sort of like made for a regular everyday general public audience to help understand these ideas. So on YouTube and then I'm out there on X on Twitter all the time at ForHumanityPod. That's where I'm sort of day to day, moment to moment, just in the in the fight for these ideas out there trying to push the better future.
Awesome. So everyone go make sure to subscribe to their channel, the AI Risk Network. You guys have over three three hundred thousand subscribers, which is super impressive. And also go follow John on Twitter. And also, yeah, tell me a little bit more about that movie and when it's coming out.
So it's coming out, it's not coming out till June of twenty twenty seven. But it's very well funded. It's done by some incredible guys over in England and they came over here and shot for months here in America interviewing lots of different people about it. I think it's gonna have a really cool like style to it. And I could not be more excited for when that movie comes out. And I, you know, I wish it was coming out next month, not June of twenty twenty seven, but you know, processes are what they are, so I'll be really excited when it when it's ready.
Awesome. I'll be looking forward to that and I'll make sure to watch it. So thanks again so much for coming on, John. This has been a pleasure and hope to stay in touch.
Awesome. Thanks for having Dan. Thanks, man.
Of course. Hey guys, thanks for watching this episode of the Dan Leibrandt Show. If you want more episodes like this, don't forget to like and subscribe. It really helps out the show. Also, leave a comment down below. I read every single one and I would love to hear your thoughts. Now that being said, I'll see you guys in the next one. Peace.



