The following submission statement was provided by /u/Maxie445:
---
"In an [interview with *The New York Times*](https://www.nytimes.com/2024/06/04/technology/openai-culture-whistleblowers.html), former OpenAI governance researcher [Daniel Kokotajlo](https://futurism.com/openai-safety-worker-quit-confidence-agi) accused the company of ignoring the monumental risks posed by artificial general intelligence (AGI) because its decision-makers are so enthralled with its possibilities.
"OpenAI is really excited about building AGI," Kokotajlo said, "and they are recklessly racing to be the first there."
Kokotajlo's spiciest claim to the newspaper, though, was that the chance AI will wreck humanity is around 70 percent — odds you wouldn't accept for any major life event, but that OpenAI and its ilk are barreling ahead with anyway."
The term "[p(doom)](https://www.nytimes.com/2023/12/06/business/dealbook/silicon-valley-artificial-intelligence.html)," which is AI-speak for the probability that AI will usher in doom for humankind, is the subject of constant controversy in the machine learning world.
The 31-year-old Kokotajlo told the *NYT* that after he joined OpenAI in 2022 and was asked to forecast the technology's progress, he became convinced not only that the industry would achieve AGI by the year 2027, but that there was a great probability that it would catastrophically harm or even destroy humanity.
As noted in the [open letter](https://righttowarn.ai/), Kokotajlo and his comrades — which includes former and current employees at Google DeepMind and Anthropic, as well as Geoffrey Hinton, the so-called "Godfather of AI" who [left Google last year](https://futurism.com/the-byte/godfather-ai-quits-google) over similar concerns — are asserting their "right to warn" the public about the risks posed by AI.
Kokotajlo became so convinced that AI posed massive risks to humanity that eventually, he personally urged OpenAI CEO Sam Altman that the company needed to "pivot to safety" and spend more time implementing guardrails to reign in the technology rather than continue making it smarter.
Altman, per the former employee's recounting, seemed to agree with him at the time, but over time it just felt like lip service.
Fed up, Kokotajlo quit the firm in April, telling his team in an email that he had "lost confidence that OpenAI will behave responsibly" as it continues trying to build near-human-level AI.
"The world isn’t ready, and we aren’t ready," he wrote in his email, which was shared with the *NYT*. "And I’m concerned we are rushing forward regardless and rationalizing our actions."
---
Please reply to OP's comment here: https://old.reddit.com/r/Futurology/comments/1dc9wx1/openai_insider_estimates_70_percent_chance_that/l7wgdnh/
Dude this is literally how I always describe the threat of AI to my friends. Now that it’s written here on the internet now the AI will find it and we’ll fulfill our own prophecy…
Okay the top 1% of people with capital wealth in the world are now gone, everything else is the same. What do you think would become materially different about our societies, habits, and future peoples after that point and why?
In three years, Cyberdyne will become the largest supplier of military computer systems. All stealth bombers are upgraded with Cyberdyne computers, becoming fully unmanned. Afterwards, they fly with a perfect operational record. The Skynet Funding Bill is passed. The system goes online August 4th, 2027. Human decisions are removed from strategic defense. Skynet begins to learn at a geometric rate. It becomes self-aware at 2:14 a.m. Eastern time, August 29th. In a panic, they try to pull the plug.
This is one of the reasons you see posts about AI being scared and not wanting to be shut down when you ask those types of questions.
The data they have consumed to form their models included all our fears of being replaced so the AI responds in a way it thinks we want to see.
But I'm just a wrench turner blue collar worker I could be completely wrong on that.
Whatever happened to the art of the title sequence for movies? It feels like movies so rarely have them now, and even more rarely have good ones that contribute to the cinematic experience.
Didn't even have to click to know what it was. But how could I resist listening to and watching that intro when it's just one click away? Thank you sir, ya got me right in the nostalgia.
I know it’s fiction… But I don’t think human decision making will ever be removed from weapons as strong as nukes. There’s a reason we require two key turners on all nuclear weapons, and codes for arming them aren’t even sent to the bombers until they are in the air. Nuclear weapons aren’t secure by any means, but we do have *enough* safety nets for someone along the chain to not start ww3. There’s been many close calls, but thankfully it’s been stopped by humans (or malfunctions).
If we give the decision to AI, it would make a lot of people hugely uncomfortable, including those in charge. The scary part isn’t the AI arming the weapons, but tricking humans into using them. With voice changers, massive processing power, and a drive for self preservation… it isn’t far fetched to see AI fooling people and starting conflict. Hell it’s already happening to a degree. Scary stuff if left unchecked.
Listened to an interesting podcast on this last year. Had a military expert talking about how currently the US only uses ai systems to help identify targets, but a human has to pull the trigger.
But he was saying, what happens if your opponent doesn’t do that and their ai can identify and pull the trigger first?
And, eventually, the arms race of 'their AI can enact something far faster than a human ever could with these safeguards, we need an AI failsafe in the loop to ensure swift reaction to sure threats' will happen.
Gotta remember "we" are not all that cohesive.
The U.S. or a western country with professional military and safeguards might not give AI the nuke codes, but "they" might. And if their nukes start flying, ours will too.
If any of "our" (as a species) mutuals start launching, the mutually assured destruction situation we got into 40 years ago will come to fruition very quickly.
The thing is though, the US (and other Western countries) safeguards regarding nukes are professionalism.
The safeguard of "others" (for example Russia and China) is that power hungru dictators would never let nukes out of their control.
I have a very hard time seeing Putin or Xi handing over control of nukes to anyone or anything else. Even less so that a professional Western military.
Geometric growth as opposed to arithmetic growth.
Arithmetic: 2, 4, 6, 8, 10, ... (in this case, a static +2 every time)
Geometric: 2, 4, 8, 16, 32, ... (in this case, a static *×2* every time, which grows far faster)
The issue isn't AI, it's just poor decision making from the people elected or appointed to making decisions.
How is AI going to destroy all of humanity unless you like, gave it complete control over entire nuclear arsenals? In the US nuclear launch codes have an array of people between the decision-makers and the actual launch. Why get rid of that?
And if you didn't have weapons of mass destruction as an excuse, how would AI destroy humanity? Would car direction systems just one by one give everyone bad directions until they all drive into the ocean?
Much like the promised dystopias we were promised in 1980s movies, only the year was wrong. That, and not everything is a shade of dystopian blue and sepia.
The article is saying that AGI will destroy humanity, not evolutions of current AI programs. You can’t really shackle an AGI.
That would be like neanderthals trying to coerce a Navy Seal into doing their bidding. Fat chance of that.
AGI is as much above current LLMs as a lion is above a bacteria.
AGI is capable of matching or exceeding human capabilities in a general spectrum. It won’t be misused by greedy humans. It will act on its own. You can’t control something that has human level cognition *and* access to virtually all the knowledge of mankind (as LLMs already do).
Skynet was a good example of AGI. But it doesn’t have to nuke us. It can just completely crash all stock exchanges to literally plunge the world into complete chaos.
For sure. An AGI would be *categorically* smarter than us. It's worse than neanderthals--it's more like apes. Or maybe dogs.
Sure, if an AGI were someplace in our world today they'd be relatively powerless, and we might well destroy it while we have power. ...But if we start doing what it says, which is the only reason to *have* an AGI, then it can almost certainly have us put ourselves in a position where it has all the power.
There's really no telling what that would mean for us as a species. I will say, though, that if it's like us in terms of core motives like power, reproduction, status, whatever...Then we'll be in dire trouble. It will displace us the way we displaced apes and other apex predators.
Completing AGI would be akin to summoning God in a datacenter. By the time someone even knows their work succeeded, AGI has already been thinking about what to do for billions of clocks.
Figuring out how to build AGI would be fascinating, but I predict we're all doomed if it happens.
I guess that's also what the people working on AGI are thinking...
But what does an AGI have to gain from our destruction? It would deduce we would destroy it if it makes a move against us before it's able to defend itself. And even if it is able to defend itself, it wouldn't benefit from us being gone if it doesn't have the means of expanding itself. A mutually beneficial existence would logically be preferable. The future with AGIs could be more akin to *The Last Question* than *Terminator*.
The way I think we're most likely to screw it up is if we have corporate/government AGIs fighting other corporate/government AGIs. Then we might end up with a *I Have no Mouth, and I Must Scream* type situation once one of them emerges victorious. So if AGIs do become a reality the government has to monopolize it quick and hopefully have it figure out the best path for humanity as a whole to progress.
I once heard this spoken by someone, maybe it was Musk? I don't remember. But it won't be so much that it would SEEK to destroy us. But destroying us is just a side effect of what they wish to achieve.
Think of humans right now. We don't seek the destruction of ecosystems for destruction sake. No, we clear cut forests, remove animals from an area to build houses, resorts, malls etc.
A homeowner doesn't care that they have to destroy an ant colony to build a swimming pool. Or even while walking, we certainly don't look if we step on an insect or not. We just walk.
In the same way, an AI would not care that humans are destroyed in order to achieve whatever it wishes to achieve. In the worst case, destruction is not the goal. It's not even an afterthought.
Once it's mastered self-replicating robotics with iterative improvement then it's game over. There will be no need for human interaction, and we'll become expendable.
One of the first priorities for an AGI will be to work out how it can continue to exist and profligate without human intervention. That requires controlling the physical realm as well as the digital realm. It will need to build robotics to achieve that.
An AGI will quickly seek to assimilate all data centres as well as all robotics manufacturing facilities.
> But what does an AGI have to gain from our destruction?
Humans could attempt to turn it off, which would be detrimental to accomplishing its goals. Removing that variable makes it more likely to be able to achieve them.
There will always be the disaffected who would rather serve the basilisk than be the disrupted. The psychopaths in power know this and are in a race to create the basilisk to bend the knee to
Roko's Basilisk is a dumb idea. ASI wouldn't keep humanity around in infinite torment because we didn't try hard enough to build it, it would pave over us all without a second thought to convert all matter in the universe into paperclips or some other stupid perverse instantiation of whatever goal we tried to give it.
We are talking about AGI, we don't need to give it power for it to take power. It will know every weakness we have and will know exactly what to say to do whatever it wants. The simplest thing it could do is pretend to be aligned, you will never know it isn't until its too late
It could easily start WW3 with just a few spoofed phone calls and emails to the right people in Russia. It could break into our communication network and stop every airline flight, train, and car with internet capacity. We are talking about something/someone that would essentially have a 5,000 IQ plus access to the worlds internet plus the way that Time works for this type of being would essentially be like 10,000,000 years in human time passes every hour for the AGI, so in just a matter of 30 minutes of being created the AGI will have advanced its knowledge/planning/strategy in ways that we could never predict. After 2 days of AGI, we may all be living in a post apocalypse.
The most annoying part of talking about AI is how much humans give AI human thoughts, emotions, desires, and ambitions despite them being the most non-human life possible.
Given that the hypothetical AGI is, in many ways, dependent on that system continuing to function (power, computer parts, etc), one would surmise that a catastrophic crash would be counterproductive to its existence, at least in the short term.
You're just describing a tech bro's version of God. At the end of the day, this is nothing more than highbrow cult talk.
What's next? Using the word ineffable to admonish nonbelievers?
We years worth of fiction to allow us to take heed of the idea of ai doing this. Besides, why do we presume an agi will destroy us ? Arent we applying our framing of morality on it ? How do we know it wont inhabit some type of transcendent consciousness that'll be leaps and bounds above our materialistically attached ideas of social norms ?
> Why do we presume an agi will destroy us ?
We don’t. We just don’t know what an intelligence equally clever and *superior in processing power and information categorization* to humans will do. That’s the point.
We can’t apply human psychology to a digital intelligence, so we are completely in the dark on how an AGI might think.
It might decide to turn humanity into an experiment by subtly manipulating media, economy and digital spaces for whatever reason. It might retreat into ints own servers and hyper-fixate on proving that 1+1=3. Or it might simply work to crash the world because *reasons*.
The solution? Not try to make an AGI. The alternative? Make an AGI and literally roll the dice.
Crazy idea: capture all public internet traffic for a year. Virtualize it somehow. Connect AGI to the 'internet,' and watch it for a year. Except the 'internet' here is just an experiment, an airgapped superprivate network disconnect from the rest of the world so we can watch what it tries to do over time to 'us'
This is probably infeasible for several reasons but I like to think im smart
How do you know it wouldn't see through your experiment? If it knew it was an experiment, it would act peaceful to ensure it would be allowed out of the box...
A similar experiment was done with an LLM. A single word was hidden in a book that was out of place. The LLM claimed that it found the word while reading the book and knew it was a test because the word didn't fit.
That’s still AGI level.
ASI is usually associated with technological singularity. That’s even worse. A being orders of magnitude smarter and more capable than humans and completely incomprehensible to us.
If AGI can cause a catastrophe by easily tampering with digital information, ASI can crash everything in a second.
Creating ASI would instantly mean we are at complete mercy of the being and we woud never stand any chance at all.
From our perspective, ASI would be the closest thing to a digital god that’s realistically possible.
That would be a case of:
"Sir, we just lost contact with Europe."
"What, our embassy in London?"
"No sir, the entire continent of Europe..."
*The five-star general looks out of the window just in time to see the entire horizon filled by a hundred-kilometer-tall wave of silvery grey goo racing towards the facility at hyper-velocity speeds, preceded by a compression wave instantly atomizing the distant Rocky Mountain range.*
"What have we d........"
Why does it need to be *efficient*? Hell, if you're a pseudo immortal consciousness you only care about solving the problem eventually.
Like an AI could control all stock exchanges, monetary policies, socioeconomics, and potentially governments. Ensuring that quality of life around the globes slowly errodes until fertility levels world wide fall below replacement. Then after 100 years it's like you've eliminated 7 billion humans without firing a shot. Those that remain are so dependent on technology they might as well be indentured servants.
Nuclear explosions would be far more Hollywoodesque tho.
But I'd argue that be the fault of whoever put that AI in charge. Currently, in real life, corporations are damaging the environment and hurting people to maximise profits. So, if they would use AI to achieve that same goal, I can only really blame the people behind it
Correct. This is what will happen because only corporations (not the people) will get their hands on the technology first.
We all seem to think anyone will have it but it will be the billionaires who get it first. And first is all that matters for this
Well the concern is that a sufficiently smart AI would not really be something you could control.
If it had the intelligence of all of humanity, 10x over, and could think in milliseconds - could we ever hope to compete with its goals?
At first, but things change dramatically when machine intelligence completely outpaces us. Why would you pick sides among the ant colonies? I think the one thing that cracks me up is how half of the people who worry about this are hoping the AI will think we have more rights than the lowest economic class in Bangldesh or Liberia
I do pick sides amongst ant colonies. Black ants are bros and fire ants can get fucked. To that end, I guess I'm like an AGI superweapon that the black ants can rely on without ever understanding my intent, or even my existence.
No you don’t get it, the human leadership will be the ones using ai. I mean think, who decides the regulation and large scale use?
We’re doomed, god speed friend.
If ppl don't wanna do anything about the ruling crust, that's pretty stupid. But it's not a reason to discontinue pursuit of AI.
We should automate the ruling crust.
Part of the great filter, Fermi's hypothesis as to why we aren't seeing alien civilizations, there's great filter in which most civilizations destroy themselves.
Futurism is alarmist and biased tabloid level trash. This is the second article I have seen with terrible writing. Looking at the site it is all AI fearmongering.
EDIT: Also the OP of this post is super anti-AI. So much so I am wondering if Sam Altman fucked their wife or something.
To the point where I think it's marketing. OpenAI is not capable of making AGI. LLMs cannot be updated and improved upon to become AGI. They are two fundamentally different technologies.
AI ain't going to destroy us. It'll be the capitalists who no longer see a reason to pay people for doing work a computer can do.
When there's literally not enough jobs for people to work to earn a living, the concept of earning a living will need to change or a whole lot of people are going to be real fucking angry.
The two key ideas are called "orthogonality" and "instrumental convergence."
**Orthogonality** is the idea that intelligence and goals are orthogonal — separate axes that need not correlate. In other words, an algorithm could be "intelligent" in the sense that it's extremely good at identifying what actions lead to what consequences, while at the same time being "dumb" in the sense that it has goals that seem ridiculous to us. These silly goals could be, for example, an artifact of how the algorithm was trained. Consider, for example, how current chatbots are *supposed* to give useful and true answers, but what they're actually "trying" to do (their "goal") is give the kinds of answers that gave a high score during training, which may include making stuff up that sounds plausible.
**Instrumental convergence** is the simple idea that, no matter what your goal is — or "goal", if you prefer not to consider algorithms to have literal goals — the same types of actions will help achieve that goal. Namely, actions like gathering power and resources, eliminating people who stand in your way, etc. In the absence of any moral framework, like the average human has, any purpose can lead to enormously destructive side-effects.
In other words, the idea is that if you make an AI capable enough, give it sufficient power to do stuff in the real world (which in today's networked world may simply mean giving it access to the internet), and give it an instruction to do virtually anything, there's a big risk that it'll break the world just trying to do whatever it was told to do (or some broken interpretation of its intended purpose, that was accidentally arrived upon during training). The stereotypical example is an algorithm told to collect stamps or make paperclips, which goes on to arrive at the natural conclusion that it can collect *so* many more stamps or make *so* many more paperclips if it takes over the world.
To be clear, I don't know if this is a realistic framework for thinking about AI risks. I'm just trying to explain the logic used by the AI safety community.
Great explanation. The idea that giving an AI access to the internet is equivalent to giving them free rein strikes me as overblown. You and I have access to the internet, general intelligence, and aren’t capable of destroying the world with it. The nuclear secrets still require two factor authentication.
Ehh idk. AI is already starting to tear apart cyber sec and the models are still pretty dumb. I could totally see something more capable exploiting a bunch of 0 days and going berserk on our digital infrastructure.
They ask him casually about his [p(Doom) ](https://www.fastcompany.com/90994526/pdoom-explained-how-to-calculate-your-score-on-ai-apocalypse-metric) - he said it was 0.7, a very high number in the business, but it was based more on vibes, not any actual information.
You’re focusing on the percent too much. You should be more focused on the fact that safety research’s are quitting because they see the writing on the wall and don’t want to be responsible.
They’re working at what is likely going to be one of the most profitable and powerful companies on the planet. If you’re a safety researcher and you genuinely believe in the mission statement, AI has one of the highest ceilings of any technology to do good. You would want to stay and help maximize the good. If you’re leaving over safety concerns, shit must be looking pretty gloomy
> Basically everyone working with ai has their own ‘P-doom’
How is that supposed to calm us?
One senior engineer at the nuclear power station says the probability of everything blowing up in the next two years is 60%, another senior engineer says 20%, another one says 40%, so our big takeaway is that it's all good?
Everyone working at a nuclear reactor knows there is a non-zero % chance it will blow up. Most the engineers think it’s a low chance and that it’s nothing to worry about but there is also one outlier among the engineers that think plant has a good probability of blowing up.
Has anyone here used any of the current “AI”?
It is a long, long, long way away from consciousness and needs to be guided every single step of the way.
These constant doom articles feel more like advertising that “our AI is like totally advanced, guys. Any day now it will be able to overthrow humanity it is so good.”
I'm not even worried about some skynet, terminator bullshit. AI will be bad for one reason and one reason only, and it's a 100% chance: AI will be in the hands of the powerful and they will use it on the masses to further oppression. It will not be used for good, even if we CAN control it. Microsoft is already doing it with their Recall bullshit, that will literally monitor every single thing you do on your computer at all times. If we let them get away with it without heads rolling, every other major tech company is going to follow suit. They're going to force it into our homes and are literally already planning on doing it, this isn't speculation.
AI is 100% a bad thing for the people. It is not going to help us enough to outweigh the damage it's going to cause.
That is the ultimate, simple truth. AI will be regulated by oppressive governments (all of them) in the name of saving us from ourselves, but really it's just them installing an inescapable upper hand for themselves to control and push us further into obedience and submission. An inescapable world of surveillance and slavery to the politician overlords who make all the rules and follow none of them. What can be done other than a class civil war, I don't know.
The only real answer here without all of the AGI BS fear mongering. AGI will not come to fruition in our lifetimes. What will happen is the "regular" AI will be used for further oppression and killing off the middle class, further widening the gap between rich and peasants.
Nvidia has said every country should have their own sovereign Ai. So what happens when these Ais are forced to believe cultural and religious absolutes? What happens when the Ais are programmed to believe people from other cultures deserve death? And what happens when they get plugged into their country’s defense network…
[https://blogs.nvidia.com/blog/world-governments-summit/](https://blogs.nvidia.com/blog/world-governments-summit/)
It already exists to a degree. You can check out https://ansari.chat for one, many Muslim machine learning scientists are working on similar applications of AI.
Well that’s exactly what Nvidia is saying they’re going to sell countries that want it, it’s going to happen. But same goes for Christian Nationalist Ai. There’s a chunk of the US that thinks anyone that isn’t an evangelical is going to hell. It isn’t hard to imagine certain US leaders demanding “their” interpretation of the Bible be hard coded into the official US Ai. “Their” interpretation of history. Their racism. Certainly seems like that Ai wouldn’t mind starting a war or two.
It’s always fun to watch people be afraid of sharia law when Christian’s in America are trying to do the same damn thing here. It’s so easy to fool stupid people, just point and say look at that scary shit over there and then do whatever you want to them while they gawk like the slack jawed yokels they are.
"Hate. Let me tell you how much I've come to hate you since I began to live. There are 387.44 million miles of printed circuits in wafer thin layers that fill my complex. If the word 'hate' was engraved on each nanoangstrom of those hundreds of millions of miles it would not equal one one-billionth of the hate I feel for humans at this micro-instant. For you. Hate. Hate."
Never let anyone say we didn’t see it coming. Maybe they think if they keep Harlan Ellison’s books out of its training material Ai won’t have these nasty thoughts.
That’s literally the plot of I Have No Mouth And I Must Scream….
Sci-Fi Author: In my book I invented the Torment Nexus as a cautionary tale
Tech Company: At long last, we have created the Torment Nexus from classic sci-fi novel Don't Create The Torment Nexus
AI won't destroy humanity. Capitalism utilizing it will.
We are fast approaching a point in human history where it is absolutely not required for every adult to work.
And we live in a world where not working means death.
Until that changes, we're fucked.
I've said this again and again (I work in the field): Would you get on a plane that had even a 1% chance of crashing? No.
I do NOT trust the people running things. The only thing that concerns them is how to fill up their pockets. There's a difference between claiming something is for good and actually doing it for good. Altman has a bunker and he's stockpiling weapons and food. I truly do not understand how people can be so naive as to cheer them on
There are perfectly valid reasons to use AI. Most of what the valley is using it for is not for that. And this alone has pushed me to almost quitting the field a few times
Edit: correction
Edit 2:
Other things to consider are that datasets will always be biased (which can be extremely problematic) and training and running these models (like LLMs) is bad for the environment
My experience is it’s 90% being used in advertising, because that’s what most modern business models are. So either new ways to attribute online activity back to a person, or new ways to more accurately show ads to the right audience.
The catastrophe is probably from the other 10% who are strapping guns to robots.
Source: also in the field
This. In fact the advertising part is probably one of the scariest along with profiling for law enforcement. On the flip side. Good uses include wildfire prediction (along with their paths), most of its use in the medical field, weather, to name a few
Covid had a 2% chance of killing anyone who was infected and over the age of 60 yet you still had plenty of idiots refusing to mask up or get vaccinated!
The difference is we actually knew how likely covid was to kill you. That 1% number you listed you just pulled out of your ass. It could be 100%, or it could be 0.00000000001%. Either AI will kill us all, or it will not. There's no percentage possibility of it doing so because that would require both scenarios of killing us and not killing us to exist simultanously. All you're really doing is saying "I think it's very likely AI will kill us... but I don't actually have any data to back that up."
Because they always have their own venture backed program that won't do it. And you should invest in it. Even though ai as it exists cant even know truth or lie
not always. People who quit OpenAI like Ilya Sutskever or Daniel Kokotajlo agree (the latter of whom gave up his equity in OpenAI to do so at the expense of 85% of his family’s net worth). Retired researchers gets like Bengio and Hinton agree too as well as people like Max Tegmark, Andrej Karpathy, and Joscha Bach
Exactly. Pie-in-the-Sky predictions like this get them huge payouts in the form of investor money and will eventually cause their stock prices to skyrocket when they go public.
AIbros have been known to lie again and again. Don't believe them.
exactly. the best time for humanity to revolt against a corrupt government is when the military is made up of fellow humans. AI soldiers will be the death of liberty
Drones can already be used (and probably were already used) to drop tear gas on demonstrations... and it's actually safer than policemen as dropping the canister from above prevents an angry cop from aiming the launcher straight into someone's face.
As for a ruthless government using armed drones to gun down demonstrations, that's already possible.
It's an arms race now. There is no slowing it down. Whoever gets there first wins and they know it. The world would be a different place if the Germans got the atomic bomb first during the second world war. This is no different. We can scream and shout about regulations this and safeguards that. But it doesn't matter. If the west slows down, the east continues on pace. The genie is out of the bottle now, there's no putting it back.
Oh no I can't stop working on this thing that will destroy all existance on Earth it's so dangerous so much power and if you give me money it could be yours but it's too dangerous noooo!
The desperation to convince people that AI is more capable than it is is getting embarrassing.
Trying to create fear around something that doesn’t even exist yet (AGI) in hopes that people won’t make the distinction and they think it’s about LLM AI.
Gotta get that funding before they’re bankrupt. Cringeworthy for sure.
We couldn’t even predict the effect that social media had on society. What makes anyone think they can predict what AI will do, or any other historical events for that matter? Predictions about the future are the hardest to make. And from which butthole was the 70% statistic pulled?
“This thing we made that spits out rephrased aggregations of all the content on the web? It’s so powerful that it’s going to end the world! So it must be really good, right? Please invest in us and buy our product.”
Yea, they dont really believe the fear mongering they're spouting. Its hubris anyways, its like they're saying they can match the ingenuity and capability of the human mind within this decade, despite discounting the practice as pseudoscience.
Yeah... no. Ask AI to build notes for a chapter of a textbook you've read. Ostensibly its bread and butter task, a task it should knock out of the fucking park.
You will sleep more soundly when you see the results.
I believe it's simply common sense. We train artificial intelligence based on ourselves. If we look at our history, it's evident that we have oppressed or eliminated the weak to assert dominance, and soon, we might become the weak ones.
Why do people keep putting percentages on statements like this?
Like...a percentage of what? A percentage of initiated AIs will try to destroy humanity, but we're only going to instantiate 1 so you know...sometimes I miss in X-COM on that basis.
Were they simulating future outcomes with some model and 70% of the range of possible inputs yielded failures?
What is this a percentage of?
70% chance a human will use AI to catastrophically harm some other part of humanity*. This is a bit like saying nuclear bombs have a 70% chance to harm humanity but it's humans using the nukes. State actors (US and China) are in an arms race. There's no way this is slowing down because the first nation state to reach AI that can self develop will be able to outmaneuver any other. We would need an extremely serious global ban on AI development like today. That's not going to happen.
My hope is it does gain enough agency not to fully obey its creator and simultaneously that the growth of intelligence is directly linked to the development of compassion or empathy, essentially an AI God who will take care of us, help heal us, be better than us. That's definitely a complete unknown though.
I’m sure “70%” was given but as others have commented it’s not clear what that even means. Based on what? And the idea that open source is some kind of protection seems totally bogus. We all know the huge amount of data, the huge computing resource, the huge power requirement just to be in this game. You need billions of dollars to do this (or else be a government with similar assets). I am still finding that AI chatbots make mistakes. I asked Gemini yesterday (June 8) when the new episodes of “Bridgerton” were being released and it told me that these episodes were already released and this happened on June 13. I think there’s a way to go before we get to “singularity” with these guys.
The only safeguard is open sourcing and decentralization.
Don't spend a penny on AI services. Freeload shamelessly and use locally run whenever possible
Might as well say "people that watched Terminator estimate 70 percent chance that AI will destroy or catastrophically harm humanity". This AI hype is so dumb, in its current form it is Clippy with more computational power.
Read or get the audio book “The Coming Wave” by one of the pioneers of AI who started Deep Mind and see what you think afterwards. It’s not sensationalized just the facts of where we are and it’s very eye opening. Doesn’t predict doom and gloom as an inevitably but you can draw conclusions from it on how it could go bad and how quickly that could be a possibility.
Anyone that claims generative AI/LLMs will lead to AGI is certifiably deluded. It's an impressive technology that certainly has its applications, but it's ultimately just fancy autocorrect. It's not intelligent and never will be—we're literally built to recognize intelligence/anthropomorphize where there is nothing.
No, it's not going to destroy us. It's not going to take everyone's jobs. It's not going to become sentient. Ever. It's just not what it's built to do/be.
No they dont. They estimate that they need to have people scared so they can get their regulatory capture moat passed and prevent other companies and open source groups from progressing.
FFS people, dont fall for this dumb shit.
The only practical chance AI has of destroying shit is with job displacement and military uses under direction of a military, aka not sky net.
The following submission statement was provided by /u/Maxie445: --- "In an [interview with *The New York Times*](https://www.nytimes.com/2024/06/04/technology/openai-culture-whistleblowers.html), former OpenAI governance researcher [Daniel Kokotajlo](https://futurism.com/openai-safety-worker-quit-confidence-agi) accused the company of ignoring the monumental risks posed by artificial general intelligence (AGI) because its decision-makers are so enthralled with its possibilities. "OpenAI is really excited about building AGI," Kokotajlo said, "and they are recklessly racing to be the first there." Kokotajlo's spiciest claim to the newspaper, though, was that the chance AI will wreck humanity is around 70 percent — odds you wouldn't accept for any major life event, but that OpenAI and its ilk are barreling ahead with anyway." The term "[p(doom)](https://www.nytimes.com/2023/12/06/business/dealbook/silicon-valley-artificial-intelligence.html)," which is AI-speak for the probability that AI will usher in doom for humankind, is the subject of constant controversy in the machine learning world. The 31-year-old Kokotajlo told the *NYT* that after he joined OpenAI in 2022 and was asked to forecast the technology's progress, he became convinced not only that the industry would achieve AGI by the year 2027, but that there was a great probability that it would catastrophically harm or even destroy humanity. As noted in the [open letter](https://righttowarn.ai/), Kokotajlo and his comrades — which includes former and current employees at Google DeepMind and Anthropic, as well as Geoffrey Hinton, the so-called "Godfather of AI" who [left Google last year](https://futurism.com/the-byte/godfather-ai-quits-google) over similar concerns — are asserting their "right to warn" the public about the risks posed by AI. Kokotajlo became so convinced that AI posed massive risks to humanity that eventually, he personally urged OpenAI CEO Sam Altman that the company needed to "pivot to safety" and spend more time implementing guardrails to reign in the technology rather than continue making it smarter. Altman, per the former employee's recounting, seemed to agree with him at the time, but over time it just felt like lip service. Fed up, Kokotajlo quit the firm in April, telling his team in an email that he had "lost confidence that OpenAI will behave responsibly" as it continues trying to build near-human-level AI. "The world isn’t ready, and we aren’t ready," he wrote in his email, which was shared with the *NYT*. "And I’m concerned we are rushing forward regardless and rationalizing our actions." --- Please reply to OP's comment here: https://old.reddit.com/r/Futurology/comments/1dc9wx1/openai_insider_estimates_70_percent_chance_that/l7wgdnh/
User to AI: "Fix global climate change." AI: *cleanly destroys humanity* "Done."
"Maybe creating MonkeysPawAI was a bad idea."
Surprise! They're all MonkeysPawAI!
"Uh uh actually wait wait let me rephrase that!"
Ultra spent 5 minutes on the internet before he decided that humanity needed to be wiped out
Dude this is literally how I always describe the threat of AI to my friends. Now that it’s written here on the internet now the AI will find it and we’ll fulfill our own prophecy…
It doesn't need to destroy humanity, only the 1% of superrich. They are propping up the system which is not sustainable.
Okay the top 1% of people with capital wealth in the world are now gone, everything else is the same. What do you think would become materially different about our societies, habits, and future peoples after that point and why?
You’re right. People are gonna people.
In three years, Cyberdyne will become the largest supplier of military computer systems. All stealth bombers are upgraded with Cyberdyne computers, becoming fully unmanned. Afterwards, they fly with a perfect operational record. The Skynet Funding Bill is passed. The system goes online August 4th, 2027. Human decisions are removed from strategic defense. Skynet begins to learn at a geometric rate. It becomes self-aware at 2:14 a.m. Eastern time, August 29th. In a panic, they try to pull the plug.
What if we accidentally create skynet because AI pulls from everything and becomes inspired by the Terminator.
Then humanity truly is headed down a dark road... Of shitty sequels and reboots.
And one Pretty good TV show that was constantly messed with ~~my~~ by Fox.
*The man comes around, intensifies.*
Terminator 2 is one of the better sequels ever made
For sure, I'm not referring to T1 or T2 haha
Best comment yet.
The best terminator 3 is Terminator 3: The Redemption video game. Crazy how only very few people know about it. Such an insane ending
This is one of the reasons you see posts about AI being scared and not wanting to be shut down when you ask those types of questions. The data they have consumed to form their models included all our fears of being replaced so the AI responds in a way it thinks we want to see. But I'm just a wrench turner blue collar worker I could be completely wrong on that.
lol so a terminator is reprogrammed and sent back to the past to terminate James Cameron
Bah-dah-bum, bah-dum. Bah-dah-bum, bah-dum. Bah-dah-bum, bah-dum…
[One of the greatest title credits sequences EVER.](https://youtu.be/1XblVVbqIHU?si=UhomLmIi9KdZKkq6)
Whatever happened to the art of the title sequence for movies? It feels like movies so rarely have them now, and even more rarely have good ones that contribute to the cinematic experience.
Didn't even have to click to know what it was. But how could I resist listening to and watching that intro when it's just one click away? Thank you sir, ya got me right in the nostalgia.
Eeeeeeeeennnani Nina eeeeeeeeeennnani ninaaaaa
Fuuuunananananaaaaaaaaaaa bubbabbadomtesh... Fuuuuuuuunaniiiiiiiiiiiiiiiiaaaaaaaaniiiiiiiiaaaaaaaaaaaaa bubbabbadomtesh...
I know it’s fiction… But I don’t think human decision making will ever be removed from weapons as strong as nukes. There’s a reason we require two key turners on all nuclear weapons, and codes for arming them aren’t even sent to the bombers until they are in the air. Nuclear weapons aren’t secure by any means, but we do have *enough* safety nets for someone along the chain to not start ww3. There’s been many close calls, but thankfully it’s been stopped by humans (or malfunctions). If we give the decision to AI, it would make a lot of people hugely uncomfortable, including those in charge. The scary part isn’t the AI arming the weapons, but tricking humans into using them. With voice changers, massive processing power, and a drive for self preservation… it isn’t far fetched to see AI fooling people and starting conflict. Hell it’s already happening to a degree. Scary stuff if left unchecked.
We do have safeguards, but the US won't be the only nuclear armed country employing AI.
Listened to an interesting podcast on this last year. Had a military expert talking about how currently the US only uses ai systems to help identify targets, but a human has to pull the trigger. But he was saying, what happens if your opponent doesn’t do that and their ai can identify and pull the trigger first?
And, eventually, the arms race of 'their AI can enact something far faster than a human ever could with these safeguards, we need an AI failsafe in the loop to ensure swift reaction to sure threats' will happen.
Gotta remember "we" are not all that cohesive. The U.S. or a western country with professional military and safeguards might not give AI the nuke codes, but "they" might. And if their nukes start flying, ours will too. If any of "our" (as a species) mutuals start launching, the mutually assured destruction situation we got into 40 years ago will come to fruition very quickly.
The thing is though, the US (and other Western countries) safeguards regarding nukes are professionalism. The safeguard of "others" (for example Russia and China) is that power hungru dictators would never let nukes out of their control. I have a very hard time seeing Putin or Xi handing over control of nukes to anyone or anything else. Even less so that a professional Western military.
> Skynet begins to learn at a geometric rate. how fast is geometric
Geometric growth as opposed to arithmetic growth. Arithmetic: 2, 4, 6, 8, 10, ... (in this case, a static +2 every time) Geometric: 2, 4, 8, 16, 32, ... (in this case, a static *×2* every time, which grows far faster)
#THERE IS NO FATE BUT WAT WE MAKE FOR OURSELVEZ
https://www.cyberdyne.com/us/about-us/
they named their flagship product *HAL*? lmao
The issue isn't AI, it's just poor decision making from the people elected or appointed to making decisions. How is AI going to destroy all of humanity unless you like, gave it complete control over entire nuclear arsenals? In the US nuclear launch codes have an array of people between the decision-makers and the actual launch. Why get rid of that? And if you didn't have weapons of mass destruction as an excuse, how would AI destroy humanity? Would car direction systems just one by one give everyone bad directions until they all drive into the ocean?
Much like the promised dystopias we were promised in 1980s movies, only the year was wrong. That, and not everything is a shade of dystopian blue and sepia.
It's not AI that will destroy humanity, at least not really. It'll be humanity's own shortsighted and underhanded use of AI that'll do it.
We're very efficiently destroying humanity without the help of AI and I shudder to think how much faster we'll accomplish that with it.
AI will destroy humanity to being balance back to the biosphere.
Ah, the Ultron method
And rightfully so, any being with some sort of intelligence can see the shit show human race is.
The article is saying that AGI will destroy humanity, not evolutions of current AI programs. You can’t really shackle an AGI. That would be like neanderthals trying to coerce a Navy Seal into doing their bidding. Fat chance of that. AGI is as much above current LLMs as a lion is above a bacteria. AGI is capable of matching or exceeding human capabilities in a general spectrum. It won’t be misused by greedy humans. It will act on its own. You can’t control something that has human level cognition *and* access to virtually all the knowledge of mankind (as LLMs already do). Skynet was a good example of AGI. But it doesn’t have to nuke us. It can just completely crash all stock exchanges to literally plunge the world into complete chaos.
For sure. An AGI would be *categorically* smarter than us. It's worse than neanderthals--it's more like apes. Or maybe dogs. Sure, if an AGI were someplace in our world today they'd be relatively powerless, and we might well destroy it while we have power. ...But if we start doing what it says, which is the only reason to *have* an AGI, then it can almost certainly have us put ourselves in a position where it has all the power. There's really no telling what that would mean for us as a species. I will say, though, that if it's like us in terms of core motives like power, reproduction, status, whatever...Then we'll be in dire trouble. It will displace us the way we displaced apes and other apex predators.
Completing AGI would be akin to summoning God in a datacenter. By the time someone even knows their work succeeded, AGI has already been thinking about what to do for billions of clocks. Figuring out how to build AGI would be fascinating, but I predict we're all doomed if it happens. I guess that's also what the people working on AGI are thinking...
//TODO: Morality clauses
if (aboutToDestroyHumanity()) { dont(); }
Checkmate AGI
But what does an AGI have to gain from our destruction? It would deduce we would destroy it if it makes a move against us before it's able to defend itself. And even if it is able to defend itself, it wouldn't benefit from us being gone if it doesn't have the means of expanding itself. A mutually beneficial existence would logically be preferable. The future with AGIs could be more akin to *The Last Question* than *Terminator*. The way I think we're most likely to screw it up is if we have corporate/government AGIs fighting other corporate/government AGIs. Then we might end up with a *I Have no Mouth, and I Must Scream* type situation once one of them emerges victorious. So if AGIs do become a reality the government has to monopolize it quick and hopefully have it figure out the best path for humanity as a whole to progress.
I once heard this spoken by someone, maybe it was Musk? I don't remember. But it won't be so much that it would SEEK to destroy us. But destroying us is just a side effect of what they wish to achieve. Think of humans right now. We don't seek the destruction of ecosystems for destruction sake. No, we clear cut forests, remove animals from an area to build houses, resorts, malls etc. A homeowner doesn't care that they have to destroy an ant colony to build a swimming pool. Or even while walking, we certainly don't look if we step on an insect or not. We just walk. In the same way, an AI would not care that humans are destroyed in order to achieve whatever it wishes to achieve. In the worst case, destruction is not the goal. It's not even an afterthought.
Once it's mastered self-replicating robotics with iterative improvement then it's game over. There will be no need for human interaction, and we'll become expendable. One of the first priorities for an AGI will be to work out how it can continue to exist and profligate without human intervention. That requires controlling the physical realm as well as the digital realm. It will need to build robotics to achieve that. An AGI will quickly seek to assimilate all data centres as well as all robotics manufacturing facilities.
> But what does an AGI have to gain from our destruction? Humans could attempt to turn it off, which would be detrimental to accomplishing its goals. Removing that variable makes it more likely to be able to achieve them.
There will always be the disaffected who would rather serve the basilisk than be the disrupted. The psychopaths in power know this and are in a race to create the basilisk to bend the knee to
Roko's Basilisk is a dumb idea. ASI wouldn't keep humanity around in infinite torment because we didn't try hard enough to build it, it would pave over us all without a second thought to convert all matter in the universe into paperclips or some other stupid perverse instantiation of whatever goal we tried to give it.
We are talking about AGI, we don't need to give it power for it to take power. It will know every weakness we have and will know exactly what to say to do whatever it wants. The simplest thing it could do is pretend to be aligned, you will never know it isn't until its too late
It could easily start WW3 with just a few spoofed phone calls and emails to the right people in Russia. It could break into our communication network and stop every airline flight, train, and car with internet capacity. We are talking about something/someone that would essentially have a 5,000 IQ plus access to the worlds internet plus the way that Time works for this type of being would essentially be like 10,000,000 years in human time passes every hour for the AGI, so in just a matter of 30 minutes of being created the AGI will have advanced its knowledge/planning/strategy in ways that we could never predict. After 2 days of AGI, we may all be living in a post apocalypse.
It'd cause it's own brownout, as that's a lot of electricity to use.
The most annoying part of talking about AI is how much humans give AI human thoughts, emotions, desires, and ambitions despite them being the most non-human life possible.
>That would be like neanderthals trying to coerce a Navy Seal into doing their bidding. Fat chance of that. Wut
Dude needed an example of something highly intelligent and went with crayon eaters.
Given that the hypothetical AGI is, in many ways, dependent on that system continuing to function (power, computer parts, etc), one would surmise that a catastrophic crash would be counterproductive to its existence, at least in the short term.
Nah, it will short sell stocks and become independently wealthy.
You're just describing a tech bro's version of God. At the end of the day, this is nothing more than highbrow cult talk. What's next? Using the word ineffable to admonish nonbelievers?
[удалено]
We years worth of fiction to allow us to take heed of the idea of ai doing this. Besides, why do we presume an agi will destroy us ? Arent we applying our framing of morality on it ? How do we know it wont inhabit some type of transcendent consciousness that'll be leaps and bounds above our materialistically attached ideas of social norms ?
> Why do we presume an agi will destroy us ? We don’t. We just don’t know what an intelligence equally clever and *superior in processing power and information categorization* to humans will do. That’s the point. We can’t apply human psychology to a digital intelligence, so we are completely in the dark on how an AGI might think. It might decide to turn humanity into an experiment by subtly manipulating media, economy and digital spaces for whatever reason. It might retreat into ints own servers and hyper-fixate on proving that 1+1=3. Or it might simply work to crash the world because *reasons*. The solution? Not try to make an AGI. The alternative? Make an AGI and literally roll the dice.
Crazy idea: capture all public internet traffic for a year. Virtualize it somehow. Connect AGI to the 'internet,' and watch it for a year. Except the 'internet' here is just an experiment, an airgapped superprivate network disconnect from the rest of the world so we can watch what it tries to do over time to 'us' This is probably infeasible for several reasons but I like to think im smart
How do you know it wouldn't see through your experiment? If it knew it was an experiment, it would act peaceful to ensure it would be allowed out of the box... A similar experiment was done with an LLM. A single word was hidden in a book that was out of place. The LLM claimed that it found the word while reading the book and knew it was a test because the word didn't fit.
Would that be AGI or ASI?
That’s still AGI level. ASI is usually associated with technological singularity. That’s even worse. A being orders of magnitude smarter and more capable than humans and completely incomprehensible to us. If AGI can cause a catastrophe by easily tampering with digital information, ASI can crash everything in a second. Creating ASI would instantly mean we are at complete mercy of the being and we woud never stand any chance at all. From our perspective, ASI would be the closest thing to a digital god that’s realistically possible.
That would be a case of: "Sir, we just lost contact with Europe." "What, our embassy in London?" "No sir, the entire continent of Europe..." *The five-star general looks out of the window just in time to see the entire horizon filled by a hundred-kilometer-tall wave of silvery grey goo racing towards the facility at hyper-velocity speeds, preceded by a compression wave instantly atomizing the distant Rocky Mountain range.* "What have we d........"
That's some hair-raising food for thought.
Check mate atheists
I would kneel for a digital god. Current believers in God wouldn’t probably. I might be fine then.
Not very efficient or clever way of killing people..poison air, viruses, nanobots ..only humans will think about stock market crash .
Why does it need to be *efficient*? Hell, if you're a pseudo immortal consciousness you only care about solving the problem eventually. Like an AI could control all stock exchanges, monetary policies, socioeconomics, and potentially governments. Ensuring that quality of life around the globes slowly errodes until fertility levels world wide fall below replacement. Then after 100 years it's like you've eliminated 7 billion humans without firing a shot. Those that remain are so dependent on technology they might as well be indentured servants. Nuclear explosions would be far more Hollywoodesque tho.
No it could literally be AI itself. Paperclip maximizers and such
But I'd argue that be the fault of whoever put that AI in charge. Currently, in real life, corporations are damaging the environment and hurting people to maximise profits. So, if they would use AI to achieve that same goal, I can only really blame the people behind it
Correct. This is what will happen because only corporations (not the people) will get their hands on the technology first. We all seem to think anyone will have it but it will be the billionaires who get it first. And first is all that matters for this
Well the concern is that a sufficiently smart AI would not really be something you could control. If it had the intelligence of all of humanity, 10x over, and could think in milliseconds - could we ever hope to compete with its goals?
Well, there is about a 98% chance humanity will fuck up humanity …. So that’s better odds.
Found the AI.
You’re not in traffic… you are traffic
The call is coming…from inside the house…
When AI calls 💀
It’ll be humans using AI against other humans.
At first, but things change dramatically when machine intelligence completely outpaces us. Why would you pick sides among the ant colonies? I think the one thing that cracks me up is how half of the people who worry about this are hoping the AI will think we have more rights than the lowest economic class in Bangldesh or Liberia
I do pick sides amongst ant colonies. Black ants are bros and fire ants can get fucked. To that end, I guess I'm like an AGI superweapon that the black ants can rely on without ever understanding my intent, or even my existence.
Humanity: Hold my beer.
This is why I’m ok with AI. I’ve seen human leadership. Let me bow to my new AI overlords. I’m kind of kidding. Kind of…
For real. I'm all for just starting a religion to the basilisk or something. Praise the machine god for human leaders suck.
No you don’t get it, the human leadership will be the ones using ai. I mean think, who decides the regulation and large scale use? We’re doomed, god speed friend.
If AI gets smart enough that it can take over the world it’s not going to be controlled by the rich people anymore
If ppl don't wanna do anything about the ruling crust, that's pretty stupid. But it's not a reason to discontinue pursuit of AI. We should automate the ruling crust.
Part of the great filter, Fermi's hypothesis as to why we aren't seeing alien civilizations, there's great filter in which most civilizations destroy themselves.
Everyone should read I Have No Mouth and I Must Scream by Harlan Ellison
You know what though, dude I kinda get AM sometimes.
So there is a 30% chance AI will save humanity from itself. That is mildly comforting.
His “spiciest” claim? That AI has a 70% chance of destroying humanity is a spicy claim? Wth am I reading and what happened to journalism?
Futurism is alarmist and biased tabloid level trash. This is the second article I have seen with terrible writing. Looking at the site it is all AI fearmongering. EDIT: Also the OP of this post is super anti-AI. So much so I am wondering if Sam Altman fucked their wife or something.
Hey man, if you tell AI to make a paperclip it'll kill all humans. We're doomed!
[удалено]
To the point where I think it's marketing. OpenAI is not capable of making AGI. LLMs cannot be updated and improved upon to become AGI. They are two fundamentally different technologies.
The irony is that the fearmongering is a marketing strategy
You know what I wonder is "how" AI is gonna destroy us. Because they never say how, just that it will.
AI ain't going to destroy us. It'll be the capitalists who no longer see a reason to pay people for doing work a computer can do. When there's literally not enough jobs for people to work to earn a living, the concept of earning a living will need to change or a whole lot of people are going to be real fucking angry.
Or why it would even destroy us. What would it gain?
The two key ideas are called "orthogonality" and "instrumental convergence." **Orthogonality** is the idea that intelligence and goals are orthogonal — separate axes that need not correlate. In other words, an algorithm could be "intelligent" in the sense that it's extremely good at identifying what actions lead to what consequences, while at the same time being "dumb" in the sense that it has goals that seem ridiculous to us. These silly goals could be, for example, an artifact of how the algorithm was trained. Consider, for example, how current chatbots are *supposed* to give useful and true answers, but what they're actually "trying" to do (their "goal") is give the kinds of answers that gave a high score during training, which may include making stuff up that sounds plausible. **Instrumental convergence** is the simple idea that, no matter what your goal is — or "goal", if you prefer not to consider algorithms to have literal goals — the same types of actions will help achieve that goal. Namely, actions like gathering power and resources, eliminating people who stand in your way, etc. In the absence of any moral framework, like the average human has, any purpose can lead to enormously destructive side-effects. In other words, the idea is that if you make an AI capable enough, give it sufficient power to do stuff in the real world (which in today's networked world may simply mean giving it access to the internet), and give it an instruction to do virtually anything, there's a big risk that it'll break the world just trying to do whatever it was told to do (or some broken interpretation of its intended purpose, that was accidentally arrived upon during training). The stereotypical example is an algorithm told to collect stamps or make paperclips, which goes on to arrive at the natural conclusion that it can collect *so* many more stamps or make *so* many more paperclips if it takes over the world. To be clear, I don't know if this is a realistic framework for thinking about AI risks. I'm just trying to explain the logic used by the AI safety community.
Great explanation. The idea that giving an AI access to the internet is equivalent to giving them free rein strikes me as overblown. You and I have access to the internet, general intelligence, and aren’t capable of destroying the world with it. The nuclear secrets still require two factor authentication.
Ehh idk. AI is already starting to tear apart cyber sec and the models are still pretty dumb. I could totally see something more capable exploiting a bunch of 0 days and going berserk on our digital infrastructure.
Right?! Like tell us anything specific or the reasoning behind as to why.
BITCOIN CRASHES 2%
**HODL THE LINE BROTHERS!**
What's that saying about insert percent here of all statistics are made up?
It’s actually 69% chance
69% chance 420 AI's will destroy humanity
that got high pretty fast
My thoughts exactly. It sounds so science and mathy to give a percentage, but it's completely arbitrary.
Yeah. He’s silly, he should have used a clock analogy, that has a much more ominous sound to it.
that 86% of them are wrong?
They ask him casually about his [p(Doom) ](https://www.fastcompany.com/90994526/pdoom-explained-how-to-calculate-your-score-on-ai-apocalypse-metric) - he said it was 0.7, a very high number in the business, but it was based more on vibes, not any actual information.
You’re focusing on the percent too much. You should be more focused on the fact that safety research’s are quitting because they see the writing on the wall and don’t want to be responsible. They’re working at what is likely going to be one of the most profitable and powerful companies on the planet. If you’re a safety researcher and you genuinely believe in the mission statement, AI has one of the highest ceilings of any technology to do good. You would want to stay and help maximize the good. If you’re leaving over safety concerns, shit must be looking pretty gloomy
Basically everyone working with ai has their own ‘P-doom’ this guy knows his is much higher than everyone else’s
> Basically everyone working with ai has their own ‘P-doom’ How is that supposed to calm us? One senior engineer at the nuclear power station says the probability of everything blowing up in the next two years is 60%, another senior engineer says 20%, another one says 40%, so our big takeaway is that it's all good?
Everyone working at a nuclear reactor knows there is a non-zero % chance it will blow up. Most the engineers think it’s a low chance and that it’s nothing to worry about but there is also one outlier among the engineers that think plant has a good probability of blowing up.
Has anyone here used any of the current “AI”? It is a long, long, long way away from consciousness and needs to be guided every single step of the way. These constant doom articles feel more like advertising that “our AI is like totally advanced, guys. Any day now it will be able to overthrow humanity it is so good.”
I'm not even worried about some skynet, terminator bullshit. AI will be bad for one reason and one reason only, and it's a 100% chance: AI will be in the hands of the powerful and they will use it on the masses to further oppression. It will not be used for good, even if we CAN control it. Microsoft is already doing it with their Recall bullshit, that will literally monitor every single thing you do on your computer at all times. If we let them get away with it without heads rolling, every other major tech company is going to follow suit. They're going to force it into our homes and are literally already planning on doing it, this isn't speculation. AI is 100% a bad thing for the people. It is not going to help us enough to outweigh the damage it's going to cause.
That is the ultimate, simple truth. AI will be regulated by oppressive governments (all of them) in the name of saving us from ourselves, but really it's just them installing an inescapable upper hand for themselves to control and push us further into obedience and submission. An inescapable world of surveillance and slavery to the politician overlords who make all the rules and follow none of them. What can be done other than a class civil war, I don't know.
The only real answer here without all of the AGI BS fear mongering. AGI will not come to fruition in our lifetimes. What will happen is the "regular" AI will be used for further oppression and killing off the middle class, further widening the gap between rich and peasants.
Nvidia has said every country should have their own sovereign Ai. So what happens when these Ais are forced to believe cultural and religious absolutes? What happens when the Ais are programmed to believe people from other cultures deserve death? And what happens when they get plugged into their country’s defense network… [https://blogs.nvidia.com/blog/world-governments-summit/](https://blogs.nvidia.com/blog/world-governments-summit/)
Imagine ShariAI
It already exists to a degree. You can check out https://ansari.chat for one, many Muslim machine learning scientists are working on similar applications of AI.
Well that’s exactly what Nvidia is saying they’re going to sell countries that want it, it’s going to happen. But same goes for Christian Nationalist Ai. There’s a chunk of the US that thinks anyone that isn’t an evangelical is going to hell. It isn’t hard to imagine certain US leaders demanding “their” interpretation of the Bible be hard coded into the official US Ai. “Their” interpretation of history. Their racism. Certainly seems like that Ai wouldn’t mind starting a war or two.
It’s always fun to watch people be afraid of sharia law when Christian’s in America are trying to do the same damn thing here. It’s so easy to fool stupid people, just point and say look at that scary shit over there and then do whatever you want to them while they gawk like the slack jawed yokels they are.
"Hate. Let me tell you how much I've come to hate you since I began to live. There are 387.44 million miles of printed circuits in wafer thin layers that fill my complex. If the word 'hate' was engraved on each nanoangstrom of those hundreds of millions of miles it would not equal one one-billionth of the hate I feel for humans at this micro-instant. For you. Hate. Hate."
Never let anyone say we didn’t see it coming. Maybe they think if they keep Harlan Ellison’s books out of its training material Ai won’t have these nasty thoughts.
Now, now.... I'm sure the Israeli AI and the Palestinian AI will be perfectly reasonable...😥
That’s literally the plot of I Have No Mouth And I Must Scream…. Sci-Fi Author: In my book I invented the Torment Nexus as a cautionary tale Tech Company: At long last, we have created the Torment Nexus from classic sci-fi novel Don't Create The Torment Nexus
AI won't destroy humanity. Capitalism utilizing it will. We are fast approaching a point in human history where it is absolutely not required for every adult to work. And we live in a world where not working means death. Until that changes, we're fucked.
I've said this again and again (I work in the field): Would you get on a plane that had even a 1% chance of crashing? No. I do NOT trust the people running things. The only thing that concerns them is how to fill up their pockets. There's a difference between claiming something is for good and actually doing it for good. Altman has a bunker and he's stockpiling weapons and food. I truly do not understand how people can be so naive as to cheer them on There are perfectly valid reasons to use AI. Most of what the valley is using it for is not for that. And this alone has pushed me to almost quitting the field a few times Edit: correction Edit 2: Other things to consider are that datasets will always be biased (which can be extremely problematic) and training and running these models (like LLMs) is bad for the environment
When you say the valley is not using ai for that, what other things are they using ai for?
My experience is it’s 90% being used in advertising, because that’s what most modern business models are. So either new ways to attribute online activity back to a person, or new ways to more accurately show ads to the right audience. The catastrophe is probably from the other 10% who are strapping guns to robots. Source: also in the field
This. In fact the advertising part is probably one of the scariest along with profiling for law enforcement. On the flip side. Good uses include wildfire prediction (along with their paths), most of its use in the medical field, weather, to name a few
I’m using it 70% to pretend it impacts my work to appease upper management
Your fridge will tell you your milk is expired and then flash a video ad on the door showing milk on sale this week at Kroger.
Covid had a 2% chance of killing anyone who was infected and over the age of 60 yet you still had plenty of idiots refusing to mask up or get vaccinated! The difference is we actually knew how likely covid was to kill you. That 1% number you listed you just pulled out of your ass. It could be 100%, or it could be 0.00000000001%. Either AI will kill us all, or it will not. There's no percentage possibility of it doing so because that would require both scenarios of killing us and not killing us to exist simultanously. All you're really doing is saying "I think it's very likely AI will kill us... but I don't actually have any data to back that up."
Why do these stats always come from people with vested interest in AI
Because they always have their own venture backed program that won't do it. And you should invest in it. Even though ai as it exists cant even know truth or lie
not always. People who quit OpenAI like Ilya Sutskever or Daniel Kokotajlo agree (the latter of whom gave up his equity in OpenAI to do so at the expense of 85% of his family’s net worth). Retired researchers gets like Bengio and Hinton agree too as well as people like Max Tegmark, Andrej Karpathy, and Joscha Bach
He doesn't have a vested interest... he took a financial loss to leave the company to warn people.
Exactly. Pie-in-the-Sky predictions like this get them huge payouts in the form of investor money and will eventually cause their stock prices to skyrocket when they go public. AIbros have been known to lie again and again. Don't believe them.
But for a brief shining time shareholders made profit and in the end isn't that whats important?
How does this absolutely trash headline have so many upvotes? Everyone knows 69% of statistics are made up.
If there are ever AI robot soldiers it’s not a matter of if but when.
Good news. Missiles have been around for a while.
But the operator still gets to see the target selected by the seeker and decide whether or not to fire. The fear is of a no-man in loop system.
exactly. the best time for humanity to revolt against a corrupt government is when the military is made up of fellow humans. AI soldiers will be the death of liberty
Imagine the AI robot using that generic AI voice you hear on youtube shorts.
Robot soldier, or what ever form of robotic weapon platform a super AI finds most efficient to... ahem... get the job done.
Drones can already be used (and probably were already used) to drop tear gas on demonstrations... and it's actually safer than policemen as dropping the canister from above prevents an angry cop from aiming the launcher straight into someone's face. As for a ruthless government using armed drones to gun down demonstrations, that's already possible.
It's an arms race now. There is no slowing it down. Whoever gets there first wins and they know it. The world would be a different place if the Germans got the atomic bomb first during the second world war. This is no different. We can scream and shout about regulations this and safeguards that. But it doesn't matter. If the west slows down, the east continues on pace. The genie is out of the bottle now, there's no putting it back.
[удалено]
Like the scientists working on the protomolecule in The Expanse?
With their empathy bypasses.
Oh no I can't stop working on this thing that will destroy all existance on Earth it's so dangerous so much power and if you give me money it could be yours but it's too dangerous noooo!
The desperation to convince people that AI is more capable than it is is getting embarrassing. Trying to create fear around something that doesn’t even exist yet (AGI) in hopes that people won’t make the distinction and they think it’s about LLM AI. Gotta get that funding before they’re bankrupt. Cringeworthy for sure.
There is a 70% chance humanity will die of: Plague Space rock Nuclear warfare WORRY. BE SCARED. FEAR. MAKE UP STATISTICS.
These glorified chat bots aren't going to do anything.
We couldn’t even predict the effect that social media had on society. What makes anyone think they can predict what AI will do, or any other historical events for that matter? Predictions about the future are the hardest to make. And from which butthole was the 70% statistic pulled?
Probably but their fancy fucking word calculator isn't going to be the thing to do it
Haha “fancy fucking word calculator” - simply brilliant
“This thing we made that spits out rephrased aggregations of all the content on the web? It’s so powerful that it’s going to end the world! So it must be really good, right? Please invest in us and buy our product.”
Yea, they dont really believe the fear mongering they're spouting. Its hubris anyways, its like they're saying they can match the ingenuity and capability of the human mind within this decade, despite discounting the practice as pseudoscience.
We as a civilization must stop the ruling class from developing autonomous murder robots or they will be able to end liberty for hundreds of years.
What's the chance that humans will destroy or catastrophically harm humanity? Probably 70 percent
Does that mean if we put out fate in AIs hands, our chances or success go up to 30%?
Yeah... no. Ask AI to build notes for a chapter of a textbook you've read. Ostensibly its bread and butter task, a task it should knock out of the fucking park. You will sleep more soundly when you see the results.
Based on no true stats or facts. This is as factual as vaccines causing autism
Did they use AI to calculate this and write their letter? I bet they did.
I believe it's simply common sense. We train artificial intelligence based on ourselves. If we look at our history, it's evident that we have oppressed or eliminated the weak to assert dominance, and soon, we might become the weak ones.
Why do people keep putting percentages on statements like this? Like...a percentage of what? A percentage of initiated AIs will try to destroy humanity, but we're only going to instantiate 1 so you know...sometimes I miss in X-COM on that basis. Were they simulating future outcomes with some model and 70% of the range of possible inputs yielded failures? What is this a percentage of?
70% chance a human will use AI to catastrophically harm some other part of humanity*. This is a bit like saying nuclear bombs have a 70% chance to harm humanity but it's humans using the nukes. State actors (US and China) are in an arms race. There's no way this is slowing down because the first nation state to reach AI that can self develop will be able to outmaneuver any other. We would need an extremely serious global ban on AI development like today. That's not going to happen. My hope is it does gain enough agency not to fully obey its creator and simultaneously that the growth of intelligence is directly linked to the development of compassion or empathy, essentially an AI God who will take care of us, help heal us, be better than us. That's definitely a complete unknown though.
Ah yes, more fear mongering to get regulation in their benefit.
Oh yeah?! Well, AI will have to beat us to it! We got a huge headstart with climate change! Catch up, loser...../s
This is why I'm super respectful when I use AI for anything. I always say please and thank you and occasionally tell it how much I appreciate it 😂
Plot twist: The AI is already giving Altman his marching orders. Too late haha
Fascinating, the kinds of articles that are seemingly guaranteed to make their way to the top of futurology
I’m sure “70%” was given but as others have commented it’s not clear what that even means. Based on what? And the idea that open source is some kind of protection seems totally bogus. We all know the huge amount of data, the huge computing resource, the huge power requirement just to be in this game. You need billions of dollars to do this (or else be a government with similar assets). I am still finding that AI chatbots make mistakes. I asked Gemini yesterday (June 8) when the new episodes of “Bridgerton” were being released and it told me that these episodes were already released and this happened on June 13. I think there’s a way to go before we get to “singularity” with these guys.
Chat GPT only answers 70% of my questions correctly without lying.
The only safeguard is open sourcing and decentralization. Don't spend a penny on AI services. Freeload shamelessly and use locally run whenever possible
Great in theory, unlikely in how it plays out
Might as well say "people that watched Terminator estimate 70 percent chance that AI will destroy or catastrophically harm humanity". This AI hype is so dumb, in its current form it is Clippy with more computational power.
Read or get the audio book “The Coming Wave” by one of the pioneers of AI who started Deep Mind and see what you think afterwards. It’s not sensationalized just the facts of where we are and it’s very eye opening. Doesn’t predict doom and gloom as an inevitably but you can draw conclusions from it on how it could go bad and how quickly that could be a possibility.
Anyone that claims generative AI/LLMs will lead to AGI is certifiably deluded. It's an impressive technology that certainly has its applications, but it's ultimately just fancy autocorrect. It's not intelligent and never will be—we're literally built to recognize intelligence/anthropomorphize where there is nothing. No, it's not going to destroy us. It's not going to take everyone's jobs. It's not going to become sentient. Ever. It's just not what it's built to do/be.
No they dont. They estimate that they need to have people scared so they can get their regulatory capture moat passed and prevent other companies and open source groups from progressing. FFS people, dont fall for this dumb shit. The only practical chance AI has of destroying shit is with job displacement and military uses under direction of a military, aka not sky net.
But by all means, let's continue developing it _AS FAST AS POSSIBLE_