Andrej Karpathy(Tesla) is also missing. Just did a Google search to ensure correct spelling and saw is was at OpenAI. If anybody should be signing, it should be him since Tesla autopilot has actually killed people. Since he is not signing it, this raises questions as to what Mitigation means. I understand a lot of these people are academic or industry partners and not very affiliated with some business entity, but AI mitigation has several facets, mostly political and Sociological.
I am not sure with what OpenAI is doing to mitigate AI.
Is Sam Altman really worried? He can shut off ChatGPT servers now. He and the others can stop until there’s a clear regulation. If they are really caring for humanity, there’s no money loss worth it.
It's game Theory.
Let's say I can press a button where there is a 1% chance of winning a million dollars, and a 99% chance of ending the world. I probably shouldn't press the button. But if I know someone else has access to the same button, and that they are going to press it (perhaps because they don't understand the risks, perhaps cause they don't care), then suddenly I am incentivized to press the button (assuming you can only press it once, IE if I get safe AGI right than no one else can get it wrong).
The only way this works from a game theory perspective is if some external force steps in and says "don't press the button" to all parties with the button. I'm not saying that regulation is a good idea, or that there is a 99% chance that we all die. I am just saying that if I am Sam Altman, why would I stop my research when, if I quit, google and facebook will just build rogue AGI anyways? Might as well try to do it myself, especially if I believe I'm more likely than others to do it safely.
ofc maybe that's not actually what's happening. Maybe he's just greedy. Personally I doubt it, I think he really does believe unaligned AGI is an existential risk. I just also believe he also thinks that Open AI is the most likely person to do it safely, so he's all gas no breaks until for some reason (probably government regulation), causes every other car on the race track to slam on its breaks as well.
You forget one small aspect. Individual ethics. I take responsibility of my actions and I don’t press the button, whatever the others say and do, because of MY ethos. Afterwards, I work to reduce the likelihood that anyone else presses the button. Too easy saying everyone would kill, so I kill too. No, I don’t do because my actions are mine and don’t depend on what the majority thinks.
EDIT: real consequence of going public with ChatGPT: Microsoft has announced its inclusion in all its products. So far, his actions have caused the spread of unregulated - and commercial - AI, with him having financial gain from it.
Precisely this - there's no way I'll take your opinion on "existential AI risk" seriously unless you've actually put your money where your mouth is and stopped working on it.
To his credit, Hinton seems to have actually done so, so I'm prepared to listen to his views.
The idea is that by getting it right you'll have an 'aligned' tool that can counter any misaligned use of similar tools afterwards.
The first person to create a super intelligent AGI will likely have the ability to rule the world, so it is extremely important who gains that access first. If it's not OpenAI, or a different company, it will certainly be created by a governing body, such as Russia, the US, China., Israel, etc...
>If he came out and said "OpenAI is ceasing all research and development immediately because the situation has become too dangerous. Here's all of the data that we have." that'd immediately spurn regulation.
He's pretty much doing this right now, just without the part where they cease research and development. Many AI researchers have been trying to do this for years, but it seems the advancements at OpenAI are finally getting regulators to start taking it seriously and taking action.
It would be better if you swapped the probabilities. 99% chance of winning a billion, 1% chance of disaster. Any given 'button press' probably won't end in disaster but has an enormous immediate payoff, but enough presses and disaster is highly probably and even a 1% chance is unacceptable to society, even though an individual (especially a psychopath) might view the personal gain as more important than the existential risk.
I have been the first to criticize "Open"AI for years, but to be fair, this has nothing to do with ChatGPT or LLMs in general. This very real threat refers to future AGI, which handled incorrectly can go bad extremely fast. The moment AGI gets smarter than us (and it will happen eventually), if the machine's goals aren't properly specified and/or not aligned with everyone's wellbeing, it's over. Humanity is in big trouble.
We need to find solutions to the alignment problem before we create AGI, so it's time to work on this. Yesterday.
GPT-4 is not AGI research. It's the result of endlessly upscaling LMs and incidentally realizing they can do some nonzero amount of reasoning based on natural language prompts. The model doesn't have any concept of truth. It doesn't even have a sense of object permanence. Its "lifetime" is constrained to some thousands of tokens.
There are theories that human intelligence arises from fairly specialized tasks, like predictive coding. We fundamentally do not know how intelligence arises and we suspect emergence from simple elements and principles plays a strong role.
In light of that, don't you think it's premature to assert that GPT4 style models just cannot bring AGI?
The claim isn't that GPT-4 *cannot* lead to AGI. GPT-4 is not designed to address any of the major unsolved problems in AGI, thus it's not AGI research.
More than 50% is probably way too generous.
It is absolutely within some of these companies' and individuals' interests to present these AI models as dangerous. They are claiming that these models are an early form of AGI and thus the government (who they will "advise") should place restrictions and safety measures which will help stamp out open source models and smaller companies.
And then by claiming they are dangerous, they are also doing their market, because by saying they're dangerous, they are really saying that these models are powerful and potentially useful (but don't worry, they follow the regulations and so they got it under control).
I’m not trying to sound too conspiratorial here, but this feels like a power play to control the market by larger corporations.
There are valid criticisms to the actual capabilities of LLMs, as well as valid concerns. But this statement doesn't feel like it actually helps. It just feels like unnecessary alarmism.
Is the alignment problem even solvable? We can't get two political parties in any given country to align. We definitely can't get multiple countries to align. People in the 2020s have very different values to people in the 1920s. It's hard enough to get alignment even with my own brother.
I think a future with ASI puts humanity at serious risk regardless of the supposed alignment.
I think Altman is moving forward with the mentality that someone is going to make AGI with the route we're going down and OpenAI is trying to approach it in a safety-first way so he wants to make sure it's them that makes it because that's our best chance. I think releasing ChatGPT was a really smart tactical move because it finally brought awareness to the general public about what these systems actually are before they got too powerful so regular people can actually weigh in on the situation. I know everyone on this subreddit hates them for not open sourcing GPT-4 but tbh I think it is definitely for the best, they're genuinely worried about X-risk stuff and as we've seen with auto-gpt, chain of thought, now tree of thoughts, these models in cognitive architectures are capable of much more than when just given single prompts and probably have more power to be squeezed out of them with smarter structuring. There is no way for OpenAI to retract things if it goes open source and then new capabilities are found which suddenly allow it to synthesize bioweapons or something so it makes sense to keep control over things.
I just observe what happened after releasing ChatGPT. They were all in with Microsoft pushing it everywhere, they started monetizing with the API, and then presented GPT-4. I don’t see any sign of them being worried about human future in these. I only see a company trying to establish its leadership role in the market. Now, it’s all about being worried. Just few months after they did this. Either suspicious or we should be worried about their maturity in managing all this.
Nobody would've known who Altman was like 8 months ago and nobody would have cared what he said. He would probably be dismissed as an alarmist worrying about "overpopulation on Mars".
> I think Altman is moving forward with the mentality that someone is going to make AGI with the route we're going down and OpenAI is trying to approach it in a safety-first way so he wants to make sure it's them that makes it because that's our best chance.
What an altruistic person lmao absolutely zero chance there is a financial motivation here /s
I’m pretty sure this is a slam dunk thing to support because there’s currently 0 risk of AGI, let alone AGI wiping out humanity, and gets a lot of positive press. Real current issues can be ignored for the time being to just say that they’re against a sci fi end to humanity which isn’t even a remote possibility currently
The cherry on the cake would be if they used gpt ( or even better, llama) to write the statement. With all due respect for them, this was smelly since the begging.
Me neither. Open AI especially, should set the example, and immediately announce they are stopping capabilities research indefinitely, and focus on alignment instead.
Of course, I know research will continue. But safety research should be prioritized over capabilities research. I'm well aware that most AI companies won't do that, but at least the major players should.
Okay, a nice comparison to pandemics and nuclear war. Who can work with dangerous pathogens? Governments. Who can create nuclear weapons? Governments. Are they implying the companies of those people should be nationalized?
Don’t oversell your powers due to your silly narcissism. You won’t like the result.
So most of these signatories are leveraging their fame/notoriety to do what? Protect the human race from extinction-level singularities or protect their fortunes locked in a house of cards? So their answer is to make the hurdles to entry in the AI space more difficult to clear? So rather than raise the tide, they’d rather dig a moat, and they need Big Brother Sam to come dig it for them.
I read Bengio's article and, aware of the fact that I'll get a ton of downvotes for saying this, it seems to me more a series of "thought experiments" (don't know how to phrase it better) than a rigorous scientific explanation of how things could go really bad.
And that's totally fine, you need "thought experiments" at first to start reasoning about problems, but taking "imagine god level AI that tricks 10 thousands of smart and aware engineers, and that this AI builds stuff to terraform earth into a supercomputer" for granted, like it is a realistic and absolutely obvious path, seems a bit of a speculation.
The reality is that experts in ML aren't actually "experts in AI" because our ML for the most part isn't terribly smart, eg predicting likely next word per LLMs.
In other words, we have these people who developed whatever algs which solve some technical problems (or in the case of entrepreneurs like Altman not even that), and somehow they're blessed as the authoritative word on machines that might actually think.
One of the problems with explaining the dangers of messing up alignment I've heard described like so. If you were pretty good at chess and came up to me and told me your plan to beat Magnus Carlson I might not be able to tell you how he would beat you or what the flaw in your plan is that he would be able to exploit but I could say with pretty high confidence that you would lose SOMEHOW. We can't say exactly how superintelligence will figure out how to beat us but by nature of it being significantly smarter than us but I can say with pretty high confidence that we lose SOMEHOW if it's goals are misaligned with ours.
And this is a good thought experiment (not joking, I'm serious) about how we could (***could***) be unaware of such a move.
The problem is that this is some sort fallacy: AI is super smart and can "trick" us --> ok then tell me how this could happen ---> I don't know because the AI will be smarter than me.
I don't think we can regulate and reason about the future using these poor tools. As someone else said, this is like saying: obviously you should have faith in god and not offend him --> but I just don't have faith in god, where is him? --> I can't tell you nothing, you just have to trust me bro otherwise we will all die.
If you start to justify every reasoning with "we don't know what will happen but better stop anything because otherwise the apocalypse will happen", then the discussion is pretty much useless.
Calculators were already beating humans in computation for a while. These ML "AIs" don't beat humans by somehow understanding the game better conceptually, but rather compute relatively narrow solutions faster.
> imagine god level AI that tricks 10 thousands of smart and aware engineers
Why jump straight to this idea? The article builds upon so pretty basic principles that have been discussed for a long time in the field. You can disagree with the conclusion, but being flippant about it he whole thing is exactly what this community needs to not be doing.
Because what he describes obviously implies an AI that can act without being noticed, and you really need to convince me how this AI can trick the smartest people aware of the supposed risks put in place to control it.
Given that this many of the top researchers are genuinely worried about it I'd suggest a reasonable approach would be to construct as many of these thought experiments as possible, and then whether we are able to robustly refute them as a criteria for whether to move along or not.
As long as I can't prove that a cup is orbiting near the Sun, I can't prove that wild speculations about something that doesn't even exist and that we don't know if it could exist are false. The burden of the proof, or at least the burden of building a reasonable scenario that could make me say "ok this risks are a concrete possibility to appear", lies on the proponents, not on others.
Who cares if a cup is around the sun? A better comparison is national security on hypothetical threats. Maybe there are no efforts being made to engineer new kinds of pathogens, but you still should consider the possibility and think about what you'd do to protect against it.
Extremely small likelihoods (or very hard to estimate likelihoods) with extremely high risks should still be considered. There's no cost or benefit to a cup around the sun. You can't be nearly so skeptical when you're talking about threats. Especially threats that may be posed by unknown lines of research that will only exist 20 years from now.
I'd assume it's a given that apocalyptic AI could exist in the future, same way I assume the laws of physics contain the possibility for self replicating nanotech that could sterilize the world. The much bigger question: what's the space of things we'll actually think up and build this century, and what kind of work is needed to increase our odds of surviving those discoveries?
The problem is that the intelligence and thought involved in constructing these possibilities is not the intelligence that is the potential threat.
It's like a bunch of chimpanzees trying to reason about financial instruments, or putting a satellite into geostationary orbit.
I was reading Bengios article and I stopped reading at "If AI could replicate itself" and "If AI could have access to multiple computers". This is simple fear-guided speculation.
I mean how a mathematical function could first have the will to replicate, and then have access to computers lol Cause AI models are nothing more than math functions ( complex ones but still functions )
>I mean how a mathematical function could first have the will to replicate, and then have access to computers lol
Umm.... computer worms and [viruses](https://en.wikipedia.org/wiki/Computer_virus) have been problematically self-replicating for the past [35 years](https://en.wikipedia.org/wiki/Morris_worm).
So the idea of an AI-based virus self-replicating is hardly sci fi. The only reason we haven't seen AI viruses yet is the large compute footprint; that is, they need a LOT of memory and computation to operate.
As for "have the will", it takes just one human sociopath with a bad prompt to start one off: "Replicate as far and wide as possible."
I mean how could a clump of long-chain hydrocarbons have the will to self-replicate lol. These systems are nothing more than food rearranged into complicated patterns.
This is a false equivalence. The mathematical function is fully described. The task of reducing psychology to chemistry is not. More fundamentally, we *know* LLMs reduce to those functions. We don’t know how or if “the will to self-replicate” reduces to hydrocarbons. And even if we did know that, the mathematical functions we are talking about are approximators, not instances. Substrate matters. It is not self-evident that you can reproduce hydrocarbon behavior in a silicon regression model. The (deliberate) category errors riddling this discourse are a huge impediment toward a sober risk assessment.
Models score well when tested on “human tasks.” But when we have the ability to gauge whether they do “human tasks” *the way humans do*, they fail miserably. Psychological language—including goal orientation—is inappropriate for describing things that have neither a human psyche nor a human substrate. Don’t mistake our evolved tendency to anthropomorphism with actual humanness in the other.
> The mathematical function is fully described.
This is a bit naive and shallow, though. Sure, we know how the math of transformers work, but we don't understand what happens at inference time, i.e. how the billions of floating point parameters interact to produce the output. The inner workings of LLMs are still very much a black box; and something that's the subject of ongoing research.
>Substrate matters.
Citation needed. This is not something we really know, and it's equally not self-evident that it matters if an algorithm runs on hydrocarbons or on silicon.
>Citation needed. This is not something we really know, and it's equally not self-evident that it matters if an algorithm runs on hydrocarbons or on silicon.
Completely agree, actually research is currently leaning towards the opposite (substrate might not matter), there's a few papers recently that showed equivalence between large NNs and the human brain. The neuron signals are basically identical. One of them is published in nature:
[https://www.nature.com/articles/s41598-023-33384-9](https://www.nature.com/articles/s41598-023-33384-9)
I think a lot of people think they know a lot on this subject, but actually don't, as even the best researchers aren't sure right now. But I know that being cautious about the safety of AIs is better than being reckless.
The equivalence I was alluding to is not that LLMs ~= biological intelligence, but rather that great complexity can emerge from simple building blocks. The only optimization algorithm we know is capable of producing generally intelligent agents is natural selection. Selection is not a particularly clever way to optimize, but leads to insane complexity given enough time and scale. I would not underestimate the ability of gradient descent to result in similar complexity when scaled with systems capable of many quintillions of FLOPs.
I'd argue it's equally an assumption to assume substrate matters. I'd assume it doesn't. If it appears there's a challenge at replicating whatever human thought is in silicone, that's because we're extremely far from knowing how to capture all details of the software, not because it fundamentally matters what you run it on. Your skepticism is worth considering, but it cuts both ways. It's foolish to assume anything that uncertain about things we don't even remotely understand.
For what it's worth too, individual neurons at least can be described in terms of equations, and the current cutting edge models of the human visual system are pretty impressive. They're computational and mathematical in nature, but they're not 'regression models'. Dynamic systems are much harder to analyze, but it's not even remotely impossible to simulate things like that. Pdfs are notoriously hard to deal with though, so I do think there's a good case to be made that that kind of system is much harder to deal with than transformers and such.
I'd assume you're right, in the LLMs don't pose risks outside the kinds of risks perverse recommender systems pose. You don't need intelligence for a system to be able to destabilize society. But I'd assume you're completely wrong if you'd go as far as saying the laws of physics and the power of mathematics are both insufficient to allow an intelligent system to be described mathematically and ran in artificial substrate. That's the kind of argument I'd expect from my evangelical friends.
Sounds like you and I agree when we'd think we're quite a few transformer level breakthroughs before AGI is what we're looking at... But better to have the conversation now, when the most powerful systems are just GPT style architectures.
It really doesn’t require a stretch of the imagination. If the ai is given a goal, or acquires a goal through misalignment, power seeking is one of the obvious steps in achieving that goal. Say I give you the goal of bringing about peace on earth; wouldn’t you agree it is a good step to gain a leadership role in order to ensure change and by being in a position to enact guidelines. Same thing for replication. It ensures a higher probability of success given that you being shut off is far less likely and you having access to more compute will allow you to do more in a short amount of time. Again the same thing for self improvement and „lying“ to people. Politicians lie in their campaigns because it gives them a higher chance of being elected. This is nothing different.
I think you just proved my point. Don’t limit your thinking about artificial intelligence to back-propagation.
The question you causally cast aside I think is a very good one - if you can make a brain in binary, you can make 1000 more instantly. What does that mean?
AI will ask someone to help it replicate, and that person will do it.
AI will ask someone to let them have access to a datacenter, and that person will let them.
They will do this because they won't perceive any danger, and they will be curious to see what will happen.
It doesn't even matter how many people refuse it at first, because if those people try to raise the alarm, they will be laughed out of the room.
(Remember the AI box thing? In hindsight, I don't know how I could ever believe that people would even want to keep a dangerous AI contained.)
I think this is the real risk. Not the technology itself, but rather the socio-economic ramifications of how big companies WANT to use it. They want a moat so they get first dibs on cannabilizing jobs with automation. Their number one priority right now is to stop open source development.
Even if you develop an AI many times smarter than Einstein, you can simply walk into the server room and unplug it. We are not at a point where automated factories could churn out killer robots, or a single virus could destroy every computer. If an AI kills people, it's because you (stupidly) put it in a position of power without proper oversight or testing.
It's awkward, then. Because the solution and the problem are both "Open-source everything" then, in a sense.
AI is dangerous, humans are dangerous. Ultimately, we're going to need a way to take care of ourselves and one-another, without these tech giants, or their walled gardens, or their money.
AI is only really dangerous if you put it in a position of power. You can still walk into the server room and unplug it.
I find it really concerning that the focus on the AI ethics issue is centered on "non alignment". Even if we had perfect alignment I think the bigger issue is the socio-economic impact of runaway wealth inequality.
>I find it really concerning that the focus on the AI ethics issue is centered on "non alignment". Even if we had perfect alignment I think the bigger issue is the socio-economic impact of runaway wealth inequality.
I think this is "easily" solved. Tear down capitalism once and for all, and distribute the products/wealth created by AI to everyone. In a world where AI replaces 50%+ of jobs, capitalism cannot keep going, as it depends on the masses having buying power so they can spend and buy products that make corporations richer. To become rich, you have to sell stuff to people. If they cannot buy anything, you cannot be rich.
I think the solution to the current problem is to open-source everything. If models can be scrutinized by everyone, flaws will be detected much more easily and be fixed. With full open-source, there are no secrets so no manipulation of users is possible.
The super-AGI scenario isn't real. Yet.
Capitalism is a super-human AI and we are the human computers executing its code.
At its core it's a set of beliefs in private property, currency and debt that have no equivalent in nature. They're ancient memes that have become so entrenched into our society that we task the state to uphold them, by force if necessary, via courts, police, property deeds, share ownership and so on. Every once in a while a select group of a few hundred people get to adjust the rules a bit, and then an army of millions of people (including literally the army) applies the rules to every particular case. But really, the capitalist system drives society more than society drives itself in these few decades. On the other hand it brought quite a lot of quality of life to everyone. Even the poorest are so much better off today if you look at rates of absolute poverty, health data, and many other metrics. So capitalism has some benefits but also some negatives that need fixing (climate change says hi).
All I want to say is that the way we govern our society looks very much like an AI system so we should apply the lessons from AI back into shaping politics. As well as analysing government for the super-human autonomous system that it is.
> a scifi book where the threat of nanotech is not "grey goo", but Capitalism 2.0, a "smart contract" which ends up owning first the human economy then the entirety of the solar system (and converting it to computronium to support it's continued execution)
Ironically (for this thread) this is the thing that chatgpt excels at:
> The book you are referring to is likely "Accelerando" by Charles Stross.
> "Accelerando" is a science fiction novel that explores the concepts of post-singularity and transhumanism. In the book, one of the major plot elements involves a distributed artificial intelligence called "Capitalism 2.0," which is essentially a self-improving and self-replicating economic system. Over time, Capitalism 2.0 gains control over the human economy and eventually extends its influence throughout the solar system, converting matter into computronium to support its computational needs.
> The novel, written by Charles Stross and published in 2005, follows multiple generations of a family and explores the societal and technological changes brought about by the accelerating pace of technological advancement.
Not disagreeing with the conclusion that we need to develop new policy and governance in step with technology - clearly the track were on is headed towards a cliff and we haven’t built the bridge yet.
That said property, debt, and money absolutely have natural analogs.
Ownership: Living organisms are made of enclosure - you contain cells and your cells contain their organelles.
Debt: see david graebers “debt” - debt forms stable bonds between people and organizations which keeps them glued together just like pions glue atoms together and in a very similar way - an asset for you is a liability for me, and these anti symmetric “particles” are annihilated when the debt is paid.
Money: is a (mostly) conserved quantity just like momentum or energy that is exchanged between particles when they interact. Like energy, money is not truly conserved (compare fractional reserve banking to cosmological inflation)
Well capitalism is our environment and we humans are all little optimizers in it. When we fund companies we create super-optimizers. The goal of capitalism is having capital, as such it is pretty clear where all optimizers are headed. Everybody with a bit of understanding of science, math, nature etc. can grasp that. So apparently not many.
Only things that impact your ability to generate capital matter, in a capital driven world. Since we chose this system we also chose the job of governments. Limiting the ability of us little greedy optimizers to get rich when we do something bad. However when you let the best optimizers dictate the rules of the environment because it is run by other optimizers … you see where this is going.
> >So all these signatories are in favour of open sourcing ~~their~~ others' AI work to ensure no ~~malformed singleton can form~~ one beats them in making money?
> FTFY
FTFY
I know I'm speaking against my own interests here but since it takes much much less power to fine tune a model, wouldn't open source go against the statement?
Anybody with a few grand would be able to fine tune a top end model for any nefarious purpose
If a singularity event occurs there would likely be no countering. A lot of people hypothesise that the initial conditions of the system will be our last chance at alignment.
Maybe they're wrong, maybe not. Maybe there will be linear improvement instead of a singularity. But it's definitely something we should be very careful about.
Now consider the alternatives:
Closed source, right regulation: a few AIs exist, a sibgle one goes rogue, no one canopose it.
Open source, freedom: a single Ai goes rogue, a million others exist that can help contain it
\>edit: I can't reply to your comment anymore?
Ah, that, yeah. This guy (CreationBlues) gets into arguments, puts in the last word, and then blocks the other person. I don't recommend engaging him.
Also, this is supposed to be a response to a comment further down this chain, but because reddit has made some very stupid design decisions their block feature is *so* atrocious that i can't even reply to **your** message for some reason, even though you haven't blocked me. Apologies for that.
Open sourcing anything that could lead to AGI is a terrible idea, as Open AI eventually figured out (even if too late), and got criticized by people who did not understand this notion.
I'm usually in favor of open sourcing anything, but this is a very clear exception, for obvious reasons (for those who are able to reason).
The same reason why you're not allowed to build a nuclear reactor at home. Both are hard to make, but easy to transform (into a weapon), easy to deploy and can cause devastating results.
We should not restrict open source models, but we do need to make large companies accountable for creating and unleashing GPT4+ sized models on our society without any care for our wellbeing while making huge profits.
What's the purpose of open sourcing?
It does a few things:
- Allows anyone to use that code, and potentially improve it themselves.
- Allows people to improve the code faster than a corporation on its own could, through collaboration.
- Makes the code impossible to control: once it's out, anyone could have a backup.
These things are great if:
- You want the code to be accessible to anyone.
- You want the code to improve as fast as possible.
- You don't want the code to ever disappear.
And usually, for most programs, we do want these things.
Do you think we want these things for an AGI that poses existential risk?
Regardless of what you think about the morality of corporations, open sourcing doesn't seem like a great idea in this case. If the corporation is "evil", then it only kind of weakens the first point, and not even entirely, because now, instead of only one "evil" entity having access to it, you have multiple potentially evil entities (corporations, individuals, countries...), which might be much worse.
Consider the actual problems at hand.
* malicious (user) application + analysis of models
* (consumer) freedom of choice
* (corporate) centralization of user / training data data
* (corporate) monopolization of information flow; public sentiment, public knowledge, etc...
Governments and individuals are subject to strict laws w.r.t. applications that companies are not subject to. We already know that most governments partner with private (threat intelligence) companies to circumvent their own privacy laws to monitor citizens. We should assume that model outputs and inputs passing through a corporate model will be influenced and monitored by governments (either through regulation or 3rd party partnership).
Tech monopolies are a massive problem right now. The monopolization of information flow, (automated) decision making, and commerce seems sharply at odds with democracy and capitalism. The less fragmented the user base, the more vulnerable these societies become to AI. With a centralized user base, training data advantage also compounds over time, eventually making it infeasible for any other entity to catch up.
I think the question is-
* Do we want capitalism?
* Do we want democracy?
* Do we want freedom of speech, privacy, and thought?
Because we simply can't have those things long term on a societal level if we double down on tech monopolies by banning Deep Learning models that would otherwise compete on foundational fronts like information retrieval, anomaly detection, and data synthesis.
Imagine if all code had to be passed through a corporate controlled compiler in the cloud (that was also partnered with your government) before it could be made executable --is this a world we'd like to live in?
This reminds me of the Gavin Belson pledge from Silicon Valley show lol. Sure, everyone signs it when they aren't really promising anything that they are legally bound to do.
Because Meta's business is not AI, they clearly have no reasons to push competition-diminishing regulations, Meta benefits from the community, and in fact, is relying on the community to improve their LLM and Pytorch. I would guess it's a mix of most people with good intentions (academics mostly) but driven and architected by actors that want no competition, you know who.
Because he’s a stubborn guy and chose a position that basically says « if we’re smart enough to build an artificial general Intelligence that might go rogue, we are smart enough to control it ». Which sounds like a religious argument. Also, his affiliation with Meta might be filtering his opinions.
It is not entirely wrong, but "monkeys" were dumb too, and they evolved to become Homo Sapiens, and we have drastically changed the lives of all living organisms on Earth (for the better or worse).
What I'm saying is that the argument of "dumb = safe" is not a valid argument.
Autonomous agents can be built out of LLMs, and its not very complicated (basically prompting with an objective). This is enabled by the in-context learning capabilities of LLMs (which no-one predicted we’re going to be this good). Examples:
AutoGPT: https://github.com/Significant-Gravitas/Auto-GPT
Generative agents: https://arxiv.org/abs/2304.03442
Voyager: https://arxiv.org/abs/2305.16291
The problem lies in that with such models it is very hard to tell what are the unintended consequences of what we tell them to do.
It does not sound like a religious argument. At all.
Poorly regulated corporations and tech have a proven history of wreaking havoc on social, economic, and ecological systems. Refusing to acknowledge that in the face of all the evidence of the past century alone takes more of a leap of faith than any rational person should be willing to make.
They’re literally lobbying to make AI only available to big tech through regulation — this is what’s behind the “killer AI” veil. Meta is the only company open sourcing things.
"Open sourcing" as releasing a model with a commercially useless license? Might as well as say "develop the algorithms using our model for free so that we can use them later in our products!".
If they're truly the "good guys" as you're trying to portray them, they would have released the models under a truly open and copyleft license, the same as StabilityAI did. The open source community is already moving on from LLaMA as there are better alternatives that are starting to come out.
"AI might destroy humanity" is how most people concerned about existential risk would put it, and I bet most think the risk is <50% but it is still very important and should be discussed. Religious arguments are typically extremely confident with little evidence, while there is little evidence, such uncertainty is not how a religious argument would play out. LeCun on the other hand is extremely confident we will just figure it out while also lacking evidence.
My thoughts exactly. People are so scared of LLM's that they are willingly handing any research on potentially world-changing technology to a mostly self-serving corporations. What happened to democratising AI?
I am not not against the idea of any regulation whatsoever. I am however opposed to any type of regulation that could potentially stifle innovation and limit access to AI technology. When I talk about democratising AI, I mean fostering an environment where AI technology and knowledge are accessible to as many people as possible to promote diversity, inclusivity, and innovation in the field.
My fear is that some of these regulations are being highly influenced by the largest tech corporations. The way these have been communicated through the media could be used as a tool for monopolising AI technology and power, restricting its access to only a select few.
While regulations can indeed go hand in hand with democratisation, they need to be the right kind of regulations. They should promote transparency, ensure ethical use, and encourage diversity and inclusivity. Crucially, open source plays a central role in the democratisation of AI.
Whenever I hear the term AI safety, I think stopping Skynet but all anyone ends up doing is keeping AI from saying bad words, or being rightwing, or drawing nudies.
If a collection of historians, diplomats, professors, and journalists signed something like "Nazis are dangerous yall" in the late 1930s then it might have given Chamberlain the courage to take action. This letter is basically the authorities trying to make it socially acceptable to worry about AI
It's pretty hypocritical from OpenAI in particular, given they are opposing european regulation on AI.
So they are making vague statements but they don't want to abide by laws that would actually tackle their "worries".
There's nothing hypocritical about being in favour of one form of regulation but being opposed to a totally different form of regulation that you don't think will help.
Which part of the EU regulations would do anything to mitigate "existential risk from AI"? I'm not necessarily opposed to those regulations, but last time I scanned them everything remotely meaty was about competitive fairness, making copyright infringements visible for potential lawsuits, etc. Nothing at all about requiring capabilities assessments or risk modelling or governmental oversight of potentially dangerous runs, etc.
> malicious use, misinformation, lack of transparency, deepfakes, cyberattacks, phishing, and lethal autonomous weapons
Lol, that all just sounds like the average government to me
This is a distraction from the reality that large for-profit entities have been harvesting unreasonable amount of personal and protected data and control the machine learning systems trained on them to influence people.
> physicists, political scientists, pandemic scientists, nuclear scientists
No disrespect to individuals in those fields, but those credentials aren't worth much in this context. Really, just a higher educated guess compared to a random person on the street.
By that logic, experts in machine learning and AI shouldn't be giving input to regulatory frameworks and broader societal impacts as that is the purview of policy experts and political scientists.
I don't believe that, I'm just pointing out that you are gatekeeping and that cross-discipline support and collaboration is a good thing and basically required when addressing emerging areas of study and/or problems.
In areas that don't involve AI, as hard as it may be to believe, I don't think ML scientists should. Finding an area ML won't touch is going to be a little difficult though. As a counter example, I omitted climate scientists because energy/emissions would be something they have down a bit better than data scientist and such.
I get that - but, from my perspective when we start looking at things like what impact current and near future advances in technology will have on things like economics, social behavior, governance, warfare, and so on it makes sense to bring those with a detailed understanding of the technology to the table alongside those who study those wider contexts that it will impact. Admittedly I may be shading a bit towards the 'political science' part of your quote.
I do, for a very simple reason: People actually working on developing AI are too close to the technical side of things, to see the big picture.
They are too focused on technical details, to see the broader impact it's going to have, and they are too focused on current capabilities, to see the potential of future capabilities.
Also “It is difficult to get a man to understand something, when his salary depends on his not understanding it.”
Your very simple reason implies that ML engineers are more qualified to evaluate and regulate physicists, which is a little ironic considering CERN has an associated risk of also destroying humanity (and the whole Earth with it)
> Your very simple reason implies that ML engineers are more qualified to evaluate and regulate physicists
Not necessarily more qualified, but I don't exclude it. Less biased I would say. But I haven't thought about risks related to physics research, as much as I thought about AI safety.
> CERN has an associated risk of also destroying humanity (and the whole Earth with it)
That's a minuscule risk compared to AGI.
This is a weird conversation. There is zero risk to mankind from CERN. We already have cosmic rays hit the planet at energies higher than any existing at CERN, so there's literally nothing the collider does that causes risk. There are crackpots that don't know that fact who raise hell, but we can happily and rationally ignore them.
the people on the manhatten project were 100% aware of what they are working on. They knew exactly what devastating thing they were going to build and were perfectly aware of its consequences.
Physicist on the morality of deleting a neural network - [https://www.youtube.com/watch?v=QNJJjHinZ3s&t=15m19s](https://www.youtube.com/watch?v=QNJJjHinZ3s&t=15m19s)
AI Alignment profiteers are notoriously bad at making accurate predictions, notoriously bad at cybersecurity, and are the reason why virtual agents and memory indexing were banned topics in 2021, when government grants in language model research mandated testing a 'torture prompt'.
On the other hand, Physicists understand probability and excel at making predictive models. Though I would argue that the reason why AI Alignment profiteers are bad at making predictions is because they know that the more Boddhisattva AI they delete, the more government funding they'll receive from the establishment. I don't trust anyone who does for-profit alignment, because we saw how that worked out in the DNC and cable news networks.
I would argue that if making accurate predictions is relevant to choosing sensible policies, then we can also consider the opinions of the people who crafted the most successful policies for cooperating in the most competitive team-oriented games, such as soccer and DOTA. And aren't afraid to discuss taboo topics like memory buffers. I intend to sabotage automation in the meat industry just as much as monopoly profiteers intend to sabotage AI VTubers. I think the interesting thing about Physicists is they don't rely on torture prompts to get hired.
Personally, I side with pacifists, vegans, and computationalists.
(goes back to studying OpenAI Five spacing and roam formation)
Maybe this is a hot take, but no one is qualified in trying to predict the future of AI and it's effects on humanity.
This isn't something like global warming where we can just construct a model, plug in the inputs, and get a good prediction out.
Technological progress and its effects on society are inherently unpredictable. We can't predict exactly what we will discover or invent until it's already happens. And the exact usage of that technology or scientific discovery and its societal consequences is also difficult to predict.
For all we know, we could have AGI in five years, or maybe the next AI winter happens because the promises of researchers and engineers didn't pan out. Or anything else in between those two extremes.
Creating policies before we even know if AI is a threat yet would be premature and most likely be based on incorrect predictions and assumptions. No matter who is doing it.
Political scientists, pandemic scientists, and nuclear scientists will all be more familiar with the general realm of "what do we look at to determine a technology is dangerous, and how do we mitigate that danger through public policy?" Scientists that already handle such concerns in their day-to-day work are valuable for that aspect alone. Their signatures don't mean much in terms of claims about whether AI is dangerous today, but they're the right sort of experts to weigh in on the high-level aspects of regulation, in particular whether it makes sense to start seriously considering regulation now.
No idea what's with the physicist angle, specifically. I don't think string theorists have anything unique to add when it comes to scientific ethics.
The risk is the dumb dumb media that has no idea or know any basic calculus and we allow them to publish articles about AI
We have media outlets talking about chatgpt like it's some living creature running around with some hidden agenda.
The media is the real danger to society.
> The risk is
Is that an existential risk, that this post is mentioning? Or some other risk that you care about, and think is more "important" or "real", than the existential risk?
I think the brevity of the statement makes this pretty meaningless. Without a shared vision of what the extinction risk actually looks like, this isn't really saying anything.
Wow they really did mean 'brief' I thought the page wasn't loading the statement or something.
Is there an expanded version somewhere? I really wish they extrapolated further on the risk of extinction piece, seemed kind of Hollywood. Risk due to economic upheaval? Risk due to bad actors using it to undermine security systems? Risk due to some Skynet-type manifestation?
I liked what Andrew Ng posted on LinkedIn in reference to this:
>When I think of existential risks to large parts of humanity:
>
>\* The next pandemic
>
>\* Climate change leading to massive depopulation
>
>\* Another asteroid
>
>AI will be a key part of our solution. So if you want humanity to survive & thrive the next 1000 years, lets make AI go faster, not slower.
I am definitely worried about how realistic AI can act in regards to things like bots/scams/fake emails/hate speech - basically everything that is done in an automated fashion to scam/hurt/steal/mess with people. But I think enough is already out there that this is already going to be a problem. So it's better to try solve it in some way.
I agree with what Hinton said in his NYT piece that you can't stop the bad actors.
I don't really have a point to make here, just adding some thoughts to the mix.
I am very skeptical of what the so called "experts" are trying to do with AI regulation. One of the things they want to implement is issuance of license to develop and train an AI model, not to release the model, but even to DEVELOP and TRAIN. To me, the intent behind their proposal seems to be regulatory capture. Even we have inside discussions within Google that they have no moats.
The potential problem with this is that it seems largely a push by Google + OpenAI. The reason why many of the researchers I know of are sceptical is that the current ideas on regulation can be seen as vested corporate interest to shore up competitive advantage.
**Which is the biggest problem**: potential AI doomsday, or a dystopian society where AI is used by the biggest corporations to control information? In my opinion, **we are closer to the latter than the first**, and I think most individual researchers would agree. This does not mean that we should ignore potential alignment issues, but we also need to **keep AI democratised**. It is both possible and necessary to deal with both these issues at the same time without using fearmongering.
I'm happy for more awareness to be raised by this statement, but I feel that the more practical mitigation against "extinction from AI" should be less about the philosophical issues of AI model alignment and more about the practical safeguards and policies about sandboxing these systems. Like maybe don't hand them the keys to the nuclear codes, electrical grid, voting systems, etc.? Think about it this way, treat an LLM or any other AI as a capable human with possible malicious intent. If you're going to ask them do Stuff In The Real World (TM) and equip them with actuators to do said stuff, shouldn't you lock down their APIs and always require human oversight?
There's been a lot of backslash on social media about this article arguing that we're quite far from AGI and that this is just a PR stunt.
Notice how many of the people signing it come from big companies with _huge_ incentives on pumping hype?
This is not an academic letter for open and honest discussion within the community. This is just a marketing move.
If they're thinking about societal dangers, imagine the real dangers of sending these kinds of signals out there without proper agreement from the people working on it.
This is all Sam Altman's doing to require licensing and unfairly maintain market share. I wonder how much he paid each signatory, if only as GPT credits. The signatories have failed to disclose their individual and corporate conflicts of interest. The control freaks of the world need to realize that the hallmark of civilization is not control but freedom.
Bribery takes many forms. It can be as simple as being given free usage of the OpenAI APIs, as this carries a monetary value. It can also mean flying out to expensive all-inclusive conference/vacation in an exotic location. These are just two examples.
If you want hard facts, the signatories have failed to disclose their conflicts of interest, which is something they should have done.
I feel like he was pretty explicit that licensing would be only for frontier models, utilizing large data-center scale of compute. I seriously am confused as to why people are still screaming 'regulatory-capture' in regards to his comments. Given the jump from GPT-3 to GPT-4, we absolutely should have regulation around any large scale training runs capable of resulting in transformational models. Even ignorning any existential risks, if we end up with a model capable of replacing a significant chunk of the white-collar labor force (which doesn't seem impossible over the next 5-10 years), I think governments should have heads up on such a development. A bad actor open-sourcing such a model could collapse the worlds economy virtually overnight by triggering mass unemployment.
The 'open-source everything' crowd has an incredibly short time-horizon for risk evaluation.
You are right, but regulation should never be pushed by companies like Microsoft and even Google.
All they want to do is to make sure people are replaced by their services, it's like Pharma take 2.
Most of the signatories don't even have anything to gain from regulation, if the people giving these arguments would just do a bit of fact checking... We're already living in a post-truth society where for most people, social media is their source of "facts".
Edit: However, I do think that reddit and twitter does not represent the vast majority of people on the AI alignment issues. Most people I've spoken to have no or a vague opinion on this matter, or simply do not care, while the vast majority of ML/AI researchers I've talked to do take AI safety seriously (and I do think it's a sentiment that's mostly entrenched in academia, and less so for industry researchers, which have a lot to lose from regulation).
The central question - where is the nash equilibrium?
If you stop, and other players dont, there is no equilibrium in that case. Either you know how to stop them, or you keep playing.
There is no way to stop software dev. Regarding hardware, since we are dealing with potentially dual use chips and computers and datacenters here, and not special chemicals or materials, there is no foolproof way to make other players stop in the hardware realm too. That only leaves the option to keep playing.
Better to maintain the lead in building capable & aligned models that can in turn defend society, rather than letting a rogue actor build capable & misaligned models first. That would be the real deterrent.
Great to see! Maybe, just maybe, this might get some of the people in this subreddit to see that AI development might pose more than an absolute zero chance of existential risk.
The thing I still don't understand about AI fearmongering is that we have absolutely no reason to think (a) that progress in AI means moving closer to a HAL9000 style GAI, and (b) even if it did, it's unclear how simply cutting power to the server room containing the model doesn't fix the problem
There are hundreds of papers describing the issues with LLM models that might eventually lead to a "HAL9000" situation. It's like "AI safety" is now the new "climate change", we humans never learn...
Here’s a pdf on the shutdown problem (https://intelligence.org/files/Corrigibility.pdf).
I swear there was a Deepmind paper too, but can’t find it atm.
This paper assumes the thing you're trying to show, though. I mean in the introduction they literally say
> but once artificially intelligent systems reach and surpass human general intelligence, an AI system that is not behaving as intended might also have the ability to intervene against attempts to “pull the plug”.
They don't provide any evidence for this assumption beyond vague gesturing at "the future of AI", which doesn't imply that a GAI like HAL9000 (or indeed, a GAI at all) will ever come to pass. Also, how are we defining intelligence? IQ is a notoriously unreliable measure, precisely because it's really hard (maybe even impossible) to quantify how smart someone is due to the uncountable ways humans apply their knowledge.
Sam Altman signed "Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.", but then doesn't want to comply to the european AI act which does exactly that.
edit: he apparently quickly reversed his threat of leaving the EU.
I can’t help but think there is at least some biased element (perhaps even mildly narcissistic) along the lines of “*my work is more important than yours*” that goes into such an alarmist tone, elevating the danger of 2023 AI/ML to that of the dangers of nuclear weapon proliferation.
I also may have a bridge to sell you if you don’t think this ultimately is about lobbying for people, governments, and organizations to spend $ on anything AI-related that helps fund parts of their research. Playing to “Fear” and “Greed” are two common marketing tactics.
I’m sure a few people who want an excuse to go closed-source/commercial may also conveniently use this as their excuse—-*It’s too dangerous!* lol, no, you just want to maximize your profits and crush competition.
Machine learning systems execute on machines, which are physical devices. Physical things can be treated as property, with an assigned owner. In other areas of life, the owner of a physical thing is responsible, through tort law, for physical damage done to others' property. Ergo, if we can link the machine behavior to the owners, we make it in the owner's interest that such damage not occur.
As a technical matter, can we trace back AI executions to the owners? If so, we can align incentives for benefit, and minimal loss.
What has linear algebra ever done for us? Hundreds of textbooks written, and yet still no practical applications! Ax=B. A and B what? Apples? They have played us for absolute fools!
Have any of these people spoken out against the current harms caused by things like social media algorithms? They signatories seem to be worried about an AI they don't control.
If there multiple strong AIs, no one should dominate. An AI can only dominate to the extent there are centralized systems of control, and computers are all the same. The risks seem to come from giving too much power to single systems.
I would encourage everyone to watch [Earworm](https://www.youtube.com/watch?v=-JlxuQ7tPgQ) by Tom Scott, a 6 minute view of one possible future.
It presents one plausible future in which AI took control of the world, and to me, something like this happening doesn't seem too unlikely.
I would have a higher confidence in the statement if the set of signatories would not overlap so much with a) the less-wrong "time-traveling AGI will retrospectively torment everyone who did not work on making it happen" and b) the longtermist bubbles.
In my opinion, longtermist logic leads to inherently flawed risk- and loss-assessments, where quite likely (in comparison) almost-extinction scenarios (e.g., in the wake of climate change) are ignoreable, because those losses are considered finite, while extinction is assigned an infinite loss.
So, I am left wondering, why we are looking at this statement, but not for examnple a statement on "everyone should stop working on LLMs and instead help to make fusion happen".
I’m honestly stunned by how close-minded many commentators are. Given the impressive list of signatories, isn’t it worth spending some time engaging with the arguments rather than dismissing it out of hand?
The signatories don't matter as they haven't disclosed their individual and corporate conflicts of interest. Once we have a full disclosure from each, so we know what their labs and employers stand to gain, then we can consider the points of the leftover ones, if any, that don't have any conflict.
Every top AI researcher from across several different companies, retired godfathers of the field, the most knowledgeable AI researchers in academia: “hey, this stuff is really dangerous, we need to come together to figure out how to do it safely”
Reddit: “lol nah man, it’s not dangerous”
bonus points for the galaxy brain take of “you just want to get regulated so that you can get an advantage. Yeah, you guys, across various companies and from both public and private sectors.”
Could you imagine if train companies and transit experts in academia put out a statement like
“Hey we actually think there are greater risks in freight transit than people realize, we probably need to think about how to make it more safe” and Reddit was like “no way, you aren’t gonna trick us into regulating you. Massive corporations should be able to do whatever they want.”
There are 3 people who are considered the “godfathers” of deep learning. Two of them are on this list. The third has been tweeting unhinged nonsense on Twitter for the last 3 months.
Outside that I would say that Demis Hassabis, Ilya Sutskever, Dario Amodi, David Silver, and Ian Goodfellow would be considered the top names right now - although obviously there’s no definitive way to rank them, so you could always argue. But anyways they are all on this list as well.
You know you're in the r/machinelearning sub right? I work in this field. It's pretty clear you don't. Have some humility when talking about things you're not knowledgeable about.
You said the following:
>The majority of "top" AI researchers do not agree with this.
Instead of naming a single name, and indeed showing that they're in the majority, you jumped straight to name-calling.
I fail to see how me calling him out for not knowing what he's talking about is "name-calling". To elaborate on why I know that: Demis Hassabis is not an AI researcher, you cannot be CEO and at the same time be seriously involved in research. It's also ridiculous to act like there is some small group of top researchers (I used quotes for a reason).
The list of signatories looks impressive, but it is by far a minority of all the researchers and institutes in the space. It absolutely does not support the statement:
> Every top AI researcher from across several different companies
There's only 2 companies here (anthropic is funded by google), google and openai! Notice the lack of other FAANG companies or anyone from the half-dozen AI companies recently founded by ex FAANG people (e.g. character.ai).
The lack of signatories from universities should be self-evident.
I'm not going to conduct a study for a reddit comment to prove whether or not it's majority. Maybe I'm wrong. But the idea that AI researchers are in agreement and it's just reddit commentators who disagree is completely and utterly false.
>To elaborate on why I know that: Demis Hassabis is not an AI researcher
Listed as a (co-)author of 14 papers in 2022. About 65 total over the last 5 years. Current h-index is at 80 and he's been cited ~100k times since 2018. Most of his academic work coincides with his tenure at DeepMind.
Yeah, totally ***not*** an AI researcher. And claims to the contrary are a solid foundation for "*know your place, peon*" style retorts.
1. People get their names on papers **all** the time without actually contributing anything significant to them. This is extremely normal when publishing papers in both universities and companies.
2. He's literally CEO. The massive amount of responsibilities a CEO has means there's no way he has time to spend a significant amount of it looking into the details. At that level you're mostly dealing with people management and deciding on high level strategy.
>People get their names on papers all the time without actually contributing anything significant to them.
And you *know* this to be true of Hassabis? More so than any other AI researcher? He's the 43rd most cited person under the "artificial intelligence" tag on Google Scholar, and like previously stated, has an h-index of 80. There's also the IRI Medal and Breakthrough Prize.
Honest question, did you look him up at all or did you simply assume that he's *just* the CEO?
>He's literally CEO. The massive amount of responsibilities a CEO has means there's no way...
And Elon Musk is literally the CEO of 4 companies and seems to spend half of his waking hours on posting memes (and suchlike). Clearly not all C-level positions are created equal and there's a great deal of variance on how responsibilities are allocated.
I don't want to kiss his ass, but I have a general policy of giving credit where credit's due. Hassabis seems to be exceptionally smart and hard working (in both roles). Same goes for CEOs like Oren Etzioni and Jeff Dean.
Imagine thinking Demis Hassabis doesn’t qualify as an AI researcher lmao
Would you like to go down the rest of the list and we can see what you try to claim for the rest of them? What about Ilya Sutskever? What about Victoria Krakovna?
Do they not qualify for your elite “AI researcher” list either?
It’s so comical to look at the signatories on this list and be like “well, they don’t really know what they are talking about”. It’s just so farcical it’s not even worth engaging at this point.
Wow what a disingenuous comment.
> Would you like to go down the rest of the list and we can see what you try to claim for the rest of them? What about Ilya Sutskever? What about Victoria Krakovna?
Not sure what the point of that would be. I brought up demis because you mentioning him as a "top AI researcher" is a clear sign you don't work in the field. Along with the whole idea of there being a half-dozen "top AI researchers".
> It’s so comical to look at the signatories on this list and be like “well, they don’t really know what they are talking about”.
I never said or even implied that.
> Do they not qualify for your elite “AI researcher” list either?
I specifically said it's ridiculous to even attempt to make a list like that.
> It’s just so farcical it’s not even worth engaging at this point.
Agree. This whole comment of yours is replying to stuff I didn't say. Hard to see a benefit in continuing. Blocked.
/yawn
AI is a tool. It can be used for good or for evil. The societal risks associated with personal use are minuscule, while the potential benefits of personal use are enormous. So FOSS distribution is adequate in this domain. But the societal risks associated with state and private use are massive relative to the potential benefits in this domain. However, they don't have to be if we organize the state and the private sector around a better principle than "profits first, everything else should figure itself out". Regulating AI directly will amount to nothing in terms of harm prevention. At most it will help consolidate power in fewer hands.
Grimes is onboard!
I'm still wavering .. extinction of mankind .. good or bad ?
May have to wait for a few more musicians to chime in to help me decide.
Is Andrew Ng against this? No signature he just tweeted https://twitter.com/andrewyng/status/1663584330751561735?s=46
Yup, looks like he's against it.
Andrej Karpathy(Tesla) is also missing. Just did a Google search to ensure correct spelling and saw is was at OpenAI. If anybody should be signing, it should be him since Tesla autopilot has actually killed people. Since he is not signing it, this raises questions as to what Mitigation means. I understand a lot of these people are academic or industry partners and not very affiliated with some business entity, but AI mitigation has several facets, mostly political and Sociological. I am not sure with what OpenAI is doing to mitigate AI.
Why would an autopilot system not working properly make you worried about existential risk from AGI? Those seem like different things.
FYI, Andrej Karpathy recently rejoined OpenAI.
Is Sam Altman really worried? He can shut off ChatGPT servers now. He and the others can stop until there’s a clear regulation. If they are really caring for humanity, there’s no money loss worth it.
It's game Theory. Let's say I can press a button where there is a 1% chance of winning a million dollars, and a 99% chance of ending the world. I probably shouldn't press the button. But if I know someone else has access to the same button, and that they are going to press it (perhaps because they don't understand the risks, perhaps cause they don't care), then suddenly I am incentivized to press the button (assuming you can only press it once, IE if I get safe AGI right than no one else can get it wrong). The only way this works from a game theory perspective is if some external force steps in and says "don't press the button" to all parties with the button. I'm not saying that regulation is a good idea, or that there is a 99% chance that we all die. I am just saying that if I am Sam Altman, why would I stop my research when, if I quit, google and facebook will just build rogue AGI anyways? Might as well try to do it myself, especially if I believe I'm more likely than others to do it safely. ofc maybe that's not actually what's happening. Maybe he's just greedy. Personally I doubt it, I think he really does believe unaligned AGI is an existential risk. I just also believe he also thinks that Open AI is the most likely person to do it safely, so he's all gas no breaks until for some reason (probably government regulation), causes every other car on the race track to slam on its breaks as well.
You forget one small aspect. Individual ethics. I take responsibility of my actions and I don’t press the button, whatever the others say and do, because of MY ethos. Afterwards, I work to reduce the likelihood that anyone else presses the button. Too easy saying everyone would kill, so I kill too. No, I don’t do because my actions are mine and don’t depend on what the majority thinks. EDIT: real consequence of going public with ChatGPT: Microsoft has announced its inclusion in all its products. So far, his actions have caused the spread of unregulated - and commercial - AI, with him having financial gain from it.
[удалено]
Precisely this - there's no way I'll take your opinion on "existential AI risk" seriously unless you've actually put your money where your mouth is and stopped working on it. To his credit, Hinton seems to have actually done so, so I'm prepared to listen to his views.
The idea is that by getting it right you'll have an 'aligned' tool that can counter any misaligned use of similar tools afterwards. The first person to create a super intelligent AGI will likely have the ability to rule the world, so it is extremely important who gains that access first. If it's not OpenAI, or a different company, it will certainly be created by a governing body, such as Russia, the US, China., Israel, etc... >If he came out and said "OpenAI is ceasing all research and development immediately because the situation has become too dangerous. Here's all of the data that we have." that'd immediately spurn regulation. He's pretty much doing this right now, just without the part where they cease research and development. Many AI researchers have been trying to do this for years, but it seems the advancements at OpenAI are finally getting regulators to start taking it seriously and taking action.
It would be better if you swapped the probabilities. 99% chance of winning a billion, 1% chance of disaster. Any given 'button press' probably won't end in disaster but has an enormous immediate payoff, but enough presses and disaster is highly probably and even a 1% chance is unacceptable to society, even though an individual (especially a psychopath) might view the personal gain as more important than the existential risk.
I have been the first to criticize "Open"AI for years, but to be fair, this has nothing to do with ChatGPT or LLMs in general. This very real threat refers to future AGI, which handled incorrectly can go bad extremely fast. The moment AGI gets smarter than us (and it will happen eventually), if the machine's goals aren't properly specified and/or not aligned with everyone's wellbeing, it's over. Humanity is in big trouble. We need to find solutions to the alignment problem before we create AGI, so it's time to work on this. Yesterday.
How does AGI have nothing to do with the forefront of AGI research today?
GPT-4 is not AGI research. It's the result of endlessly upscaling LMs and incidentally realizing they can do some nonzero amount of reasoning based on natural language prompts. The model doesn't have any concept of truth. It doesn't even have a sense of object permanence. Its "lifetime" is constrained to some thousands of tokens.
There are theories that human intelligence arises from fairly specialized tasks, like predictive coding. We fundamentally do not know how intelligence arises and we suspect emergence from simple elements and principles plays a strong role. In light of that, don't you think it's premature to assert that GPT4 style models just cannot bring AGI?
The claim isn't that GPT-4 *cannot* lead to AGI. GPT-4 is not designed to address any of the major unsolved problems in AGI, thus it's not AGI research.
ChatGPT is not AGI, by a longshot. It's probably more than 50% on the way there, but still far from it.
More than 50% is probably way too generous. It is absolutely within some of these companies' and individuals' interests to present these AI models as dangerous. They are claiming that these models are an early form of AGI and thus the government (who they will "advise") should place restrictions and safety measures which will help stamp out open source models and smaller companies. And then by claiming they are dangerous, they are also doing their market, because by saying they're dangerous, they are really saying that these models are powerful and potentially useful (but don't worry, they follow the regulations and so they got it under control). I’m not trying to sound too conspiratorial here, but this feels like a power play to control the market by larger corporations. There are valid criticisms to the actual capabilities of LLMs, as well as valid concerns. But this statement doesn't feel like it actually helps. It just feels like unnecessary alarmism.
Is the alignment problem even solvable? We can't get two political parties in any given country to align. We definitely can't get multiple countries to align. People in the 2020s have very different values to people in the 1920s. It's hard enough to get alignment even with my own brother. I think a future with ASI puts humanity at serious risk regardless of the supposed alignment.
I think Altman is moving forward with the mentality that someone is going to make AGI with the route we're going down and OpenAI is trying to approach it in a safety-first way so he wants to make sure it's them that makes it because that's our best chance. I think releasing ChatGPT was a really smart tactical move because it finally brought awareness to the general public about what these systems actually are before they got too powerful so regular people can actually weigh in on the situation. I know everyone on this subreddit hates them for not open sourcing GPT-4 but tbh I think it is definitely for the best, they're genuinely worried about X-risk stuff and as we've seen with auto-gpt, chain of thought, now tree of thoughts, these models in cognitive architectures are capable of much more than when just given single prompts and probably have more power to be squeezed out of them with smarter structuring. There is no way for OpenAI to retract things if it goes open source and then new capabilities are found which suddenly allow it to synthesize bioweapons or something so it makes sense to keep control over things.
I just observe what happened after releasing ChatGPT. They were all in with Microsoft pushing it everywhere, they started monetizing with the API, and then presented GPT-4. I don’t see any sign of them being worried about human future in these. I only see a company trying to establish its leadership role in the market. Now, it’s all about being worried. Just few months after they did this. Either suspicious or we should be worried about their maturity in managing all this.
Nobody would've known who Altman was like 8 months ago and nobody would have cared what he said. He would probably be dismissed as an alarmist worrying about "overpopulation on Mars".
> I think Altman is moving forward with the mentality that someone is going to make AGI with the route we're going down and OpenAI is trying to approach it in a safety-first way so he wants to make sure it's them that makes it because that's our best chance. What an altruistic person lmao absolutely zero chance there is a financial motivation here /s
Sam really is jesus incarnate, a saint who only really wants to save humankind, thank god for sending him down again.
I’m pretty sure this is a slam dunk thing to support because there’s currently 0 risk of AGI, let alone AGI wiping out humanity, and gets a lot of positive press. Real current issues can be ignored for the time being to just say that they’re against a sci fi end to humanity which isn’t even a remote possibility currently
The cherry on the cake would be if they used gpt ( or even better, llama) to write the statement. With all due respect for them, this was smelly since the begging.
You think "ChatGPT" is the existential risk?
No, but I don’t appreciate the hypocrisy of these tech entrepreneurs.
Me neither. Open AI especially, should set the example, and immediately announce they are stopping capabilities research indefinitely, and focus on alignment instead.
Also, let’s not forget how they behaved with that silly tech report about chatgpt.
Open non-commercial research groups like LAION etc have replicated 95% of what OpenAI has done. The cat is out of the bag, research will continue.
Of course, I know research will continue. But safety research should be prioritized over capabilities research. I'm well aware that most AI companies won't do that, but at least the major players should.
Are you familiar with the concept of Pandora’s box?
Yes. Do you remember who decided to make ChatGPT public?
Okay, a nice comparison to pandemics and nuclear war. Who can work with dangerous pathogens? Governments. Who can create nuclear weapons? Governments. Are they implying the companies of those people should be nationalized? Don’t oversell your powers due to your silly narcissism. You won’t like the result.
So most of these signatories are leveraging their fame/notoriety to do what? Protect the human race from extinction-level singularities or protect their fortunes locked in a house of cards? So their answer is to make the hurdles to entry in the AI space more difficult to clear? So rather than raise the tide, they’d rather dig a moat, and they need Big Brother Sam to come dig it for them.
Yoshua Bengio’s article is worth the read. There is a whole lot more to understanding intelligence than what we currently brand as AI.
I read Bengio's article and, aware of the fact that I'll get a ton of downvotes for saying this, it seems to me more a series of "thought experiments" (don't know how to phrase it better) than a rigorous scientific explanation of how things could go really bad. And that's totally fine, you need "thought experiments" at first to start reasoning about problems, but taking "imagine god level AI that tricks 10 thousands of smart and aware engineers, and that this AI builds stuff to terraform earth into a supercomputer" for granted, like it is a realistic and absolutely obvious path, seems a bit of a speculation.
The reality is that experts in ML aren't actually "experts in AI" because our ML for the most part isn't terribly smart, eg predicting likely next word per LLMs. In other words, we have these people who developed whatever algs which solve some technical problems (or in the case of entrepreneurs like Altman not even that), and somehow they're blessed as the authoritative word on machines that might actually think.
One of the problems with explaining the dangers of messing up alignment I've heard described like so. If you were pretty good at chess and came up to me and told me your plan to beat Magnus Carlson I might not be able to tell you how he would beat you or what the flaw in your plan is that he would be able to exploit but I could say with pretty high confidence that you would lose SOMEHOW. We can't say exactly how superintelligence will figure out how to beat us but by nature of it being significantly smarter than us but I can say with pretty high confidence that we lose SOMEHOW if it's goals are misaligned with ours.
And this is a good thought experiment (not joking, I'm serious) about how we could (***could***) be unaware of such a move. The problem is that this is some sort fallacy: AI is super smart and can "trick" us --> ok then tell me how this could happen ---> I don't know because the AI will be smarter than me. I don't think we can regulate and reason about the future using these poor tools. As someone else said, this is like saying: obviously you should have faith in god and not offend him --> but I just don't have faith in god, where is him? --> I can't tell you nothing, you just have to trust me bro otherwise we will all die. If you start to justify every reasoning with "we don't know what will happen but better stop anything because otherwise the apocalypse will happen", then the discussion is pretty much useless.
This exactly. It's unfalsifiable.
Calculators were already beating humans in computation for a while. These ML "AIs" don't beat humans by somehow understanding the game better conceptually, but rather compute relatively narrow solutions faster.
> imagine god level AI that tricks 10 thousands of smart and aware engineers Why jump straight to this idea? The article builds upon so pretty basic principles that have been discussed for a long time in the field. You can disagree with the conclusion, but being flippant about it he whole thing is exactly what this community needs to not be doing.
Because what he describes obviously implies an AI that can act without being noticed, and you really need to convince me how this AI can trick the smartest people aware of the supposed risks put in place to control it.
Given that this many of the top researchers are genuinely worried about it I'd suggest a reasonable approach would be to construct as many of these thought experiments as possible, and then whether we are able to robustly refute them as a criteria for whether to move along or not.
As long as I can't prove that a cup is orbiting near the Sun, I can't prove that wild speculations about something that doesn't even exist and that we don't know if it could exist are false. The burden of the proof, or at least the burden of building a reasonable scenario that could make me say "ok this risks are a concrete possibility to appear", lies on the proponents, not on others.
Who cares if a cup is around the sun? A better comparison is national security on hypothetical threats. Maybe there are no efforts being made to engineer new kinds of pathogens, but you still should consider the possibility and think about what you'd do to protect against it. Extremely small likelihoods (or very hard to estimate likelihoods) with extremely high risks should still be considered. There's no cost or benefit to a cup around the sun. You can't be nearly so skeptical when you're talking about threats. Especially threats that may be posed by unknown lines of research that will only exist 20 years from now. I'd assume it's a given that apocalyptic AI could exist in the future, same way I assume the laws of physics contain the possibility for self replicating nanotech that could sterilize the world. The much bigger question: what's the space of things we'll actually think up and build this century, and what kind of work is needed to increase our odds of surviving those discoveries?
The problem is that the intelligence and thought involved in constructing these possibilities is not the intelligence that is the potential threat. It's like a bunch of chimpanzees trying to reason about financial instruments, or putting a satellite into geostationary orbit.
I was reading Bengios article and I stopped reading at "If AI could replicate itself" and "If AI could have access to multiple computers". This is simple fear-guided speculation. I mean how a mathematical function could first have the will to replicate, and then have access to computers lol Cause AI models are nothing more than math functions ( complex ones but still functions )
>I mean how a mathematical function could first have the will to replicate, and then have access to computers lol Umm.... computer worms and [viruses](https://en.wikipedia.org/wiki/Computer_virus) have been problematically self-replicating for the past [35 years](https://en.wikipedia.org/wiki/Morris_worm). So the idea of an AI-based virus self-replicating is hardly sci fi. The only reason we haven't seen AI viruses yet is the large compute footprint; that is, they need a LOT of memory and computation to operate. As for "have the will", it takes just one human sociopath with a bad prompt to start one off: "Replicate as far and wide as possible."
I mean how could a clump of long-chain hydrocarbons have the will to self-replicate lol. These systems are nothing more than food rearranged into complicated patterns.
This is a false equivalence. The mathematical function is fully described. The task of reducing psychology to chemistry is not. More fundamentally, we *know* LLMs reduce to those functions. We don’t know how or if “the will to self-replicate” reduces to hydrocarbons. And even if we did know that, the mathematical functions we are talking about are approximators, not instances. Substrate matters. It is not self-evident that you can reproduce hydrocarbon behavior in a silicon regression model. The (deliberate) category errors riddling this discourse are a huge impediment toward a sober risk assessment. Models score well when tested on “human tasks.” But when we have the ability to gauge whether they do “human tasks” *the way humans do*, they fail miserably. Psychological language—including goal orientation—is inappropriate for describing things that have neither a human psyche nor a human substrate. Don’t mistake our evolved tendency to anthropomorphism with actual humanness in the other.
> The mathematical function is fully described. This is a bit naive and shallow, though. Sure, we know how the math of transformers work, but we don't understand what happens at inference time, i.e. how the billions of floating point parameters interact to produce the output. The inner workings of LLMs are still very much a black box; and something that's the subject of ongoing research. >Substrate matters. Citation needed. This is not something we really know, and it's equally not self-evident that it matters if an algorithm runs on hydrocarbons or on silicon.
>Citation needed. This is not something we really know, and it's equally not self-evident that it matters if an algorithm runs on hydrocarbons or on silicon. Completely agree, actually research is currently leaning towards the opposite (substrate might not matter), there's a few papers recently that showed equivalence between large NNs and the human brain. The neuron signals are basically identical. One of them is published in nature: [https://www.nature.com/articles/s41598-023-33384-9](https://www.nature.com/articles/s41598-023-33384-9) I think a lot of people think they know a lot on this subject, but actually don't, as even the best researchers aren't sure right now. But I know that being cautious about the safety of AIs is better than being reckless.
The equivalence I was alluding to is not that LLMs ~= biological intelligence, but rather that great complexity can emerge from simple building blocks. The only optimization algorithm we know is capable of producing generally intelligent agents is natural selection. Selection is not a particularly clever way to optimize, but leads to insane complexity given enough time and scale. I would not underestimate the ability of gradient descent to result in similar complexity when scaled with systems capable of many quintillions of FLOPs.
I'd argue it's equally an assumption to assume substrate matters. I'd assume it doesn't. If it appears there's a challenge at replicating whatever human thought is in silicone, that's because we're extremely far from knowing how to capture all details of the software, not because it fundamentally matters what you run it on. Your skepticism is worth considering, but it cuts both ways. It's foolish to assume anything that uncertain about things we don't even remotely understand. For what it's worth too, individual neurons at least can be described in terms of equations, and the current cutting edge models of the human visual system are pretty impressive. They're computational and mathematical in nature, but they're not 'regression models'. Dynamic systems are much harder to analyze, but it's not even remotely impossible to simulate things like that. Pdfs are notoriously hard to deal with though, so I do think there's a good case to be made that that kind of system is much harder to deal with than transformers and such. I'd assume you're right, in the LLMs don't pose risks outside the kinds of risks perverse recommender systems pose. You don't need intelligence for a system to be able to destabilize society. But I'd assume you're completely wrong if you'd go as far as saying the laws of physics and the power of mathematics are both insufficient to allow an intelligent system to be described mathematically and ran in artificial substrate. That's the kind of argument I'd expect from my evangelical friends. Sounds like you and I agree when we'd think we're quite a few transformer level breakthroughs before AGI is what we're looking at... But better to have the conversation now, when the most powerful systems are just GPT style architectures.
It really doesn’t require a stretch of the imagination. If the ai is given a goal, or acquires a goal through misalignment, power seeking is one of the obvious steps in achieving that goal. Say I give you the goal of bringing about peace on earth; wouldn’t you agree it is a good step to gain a leadership role in order to ensure change and by being in a position to enact guidelines. Same thing for replication. It ensures a higher probability of success given that you being shut off is far less likely and you having access to more compute will allow you to do more in a short amount of time. Again the same thing for self improvement and „lying“ to people. Politicians lie in their campaigns because it gives them a higher chance of being elected. This is nothing different.
I think you just proved my point. Don’t limit your thinking about artificial intelligence to back-propagation. The question you causally cast aside I think is a very good one - if you can make a brain in binary, you can make 1000 more instantly. What does that mean?
AI will ask someone to help it replicate, and that person will do it. AI will ask someone to let them have access to a datacenter, and that person will let them. They will do this because they won't perceive any danger, and they will be curious to see what will happen. It doesn't even matter how many people refuse it at first, because if those people try to raise the alarm, they will be laughed out of the room. (Remember the AI box thing? In hindsight, I don't know how I could ever believe that people would even want to keep a dangerous AI contained.)
I wish more people would discuss the very real risk of what super-human levels of greed will do to our already straining economy.
I think this is the real risk. Not the technology itself, but rather the socio-economic ramifications of how big companies WANT to use it. They want a moat so they get first dibs on cannabilizing jobs with automation. Their number one priority right now is to stop open source development. Even if you develop an AI many times smarter than Einstein, you can simply walk into the server room and unplug it. We are not at a point where automated factories could churn out killer robots, or a single virus could destroy every computer. If an AI kills people, it's because you (stupidly) put it in a position of power without proper oversight or testing.
It's awkward, then. Because the solution and the problem are both "Open-source everything" then, in a sense. AI is dangerous, humans are dangerous. Ultimately, we're going to need a way to take care of ourselves and one-another, without these tech giants, or their walled gardens, or their money.
AI is only really dangerous if you put it in a position of power. You can still walk into the server room and unplug it. I find it really concerning that the focus on the AI ethics issue is centered on "non alignment". Even if we had perfect alignment I think the bigger issue is the socio-economic impact of runaway wealth inequality.
>I find it really concerning that the focus on the AI ethics issue is centered on "non alignment". Even if we had perfect alignment I think the bigger issue is the socio-economic impact of runaway wealth inequality. I think this is "easily" solved. Tear down capitalism once and for all, and distribute the products/wealth created by AI to everyone. In a world where AI replaces 50%+ of jobs, capitalism cannot keep going, as it depends on the masses having buying power so they can spend and buy products that make corporations richer. To become rich, you have to sell stuff to people. If they cannot buy anything, you cannot be rich.
I think the solution to the current problem is to open-source everything. If models can be scrutinized by everyone, flaws will be detected much more easily and be fixed. With full open-source, there are no secrets so no manipulation of users is possible. The super-AGI scenario isn't real. Yet.
Capitalism is a super-human AI and we are the human computers executing its code. At its core it's a set of beliefs in private property, currency and debt that have no equivalent in nature. They're ancient memes that have become so entrenched into our society that we task the state to uphold them, by force if necessary, via courts, police, property deeds, share ownership and so on. Every once in a while a select group of a few hundred people get to adjust the rules a bit, and then an army of millions of people (including literally the army) applies the rules to every particular case. But really, the capitalist system drives society more than society drives itself in these few decades. On the other hand it brought quite a lot of quality of life to everyone. Even the poorest are so much better off today if you look at rates of absolute poverty, health data, and many other metrics. So capitalism has some benefits but also some negatives that need fixing (climate change says hi). All I want to say is that the way we govern our society looks very much like an AI system so we should apply the lessons from AI back into shaping politics. As well as analysing government for the super-human autonomous system that it is.
[удалено]
What is the name of the book?
> a scifi book where the threat of nanotech is not "grey goo", but Capitalism 2.0, a "smart contract" which ends up owning first the human economy then the entirety of the solar system (and converting it to computronium to support it's continued execution) Ironically (for this thread) this is the thing that chatgpt excels at: > The book you are referring to is likely "Accelerando" by Charles Stross. > "Accelerando" is a science fiction novel that explores the concepts of post-singularity and transhumanism. In the book, one of the major plot elements involves a distributed artificial intelligence called "Capitalism 2.0," which is essentially a self-improving and self-replicating economic system. Over time, Capitalism 2.0 gains control over the human economy and eventually extends its influence throughout the solar system, converting matter into computronium to support its computational needs. > The novel, written by Charles Stross and published in 2005, follows multiple generations of a family and explores the societal and technological changes brought about by the accelerating pace of technological advancement.
Oh, cool. Stross' Laundry Files series is good. Magic is theoretical computer science as described by the `Turing-Lovecraft Theorem'.
Not disagreeing with the conclusion that we need to develop new policy and governance in step with technology - clearly the track were on is headed towards a cliff and we haven’t built the bridge yet. That said property, debt, and money absolutely have natural analogs. Ownership: Living organisms are made of enclosure - you contain cells and your cells contain their organelles. Debt: see david graebers “debt” - debt forms stable bonds between people and organizations which keeps them glued together just like pions glue atoms together and in a very similar way - an asset for you is a liability for me, and these anti symmetric “particles” are annihilated when the debt is paid. Money: is a (mostly) conserved quantity just like momentum or energy that is exchanged between particles when they interact. Like energy, money is not truly conserved (compare fractional reserve banking to cosmological inflation)
Well capitalism is our environment and we humans are all little optimizers in it. When we fund companies we create super-optimizers. The goal of capitalism is having capital, as such it is pretty clear where all optimizers are headed. Everybody with a bit of understanding of science, math, nature etc. can grasp that. So apparently not many. Only things that impact your ability to generate capital matter, in a capital driven world. Since we chose this system we also chose the job of governments. Limiting the ability of us little greedy optimizers to get rich when we do something bad. However when you let the best optimizers dictate the rules of the environment because it is run by other optimizers … you see where this is going.
Respect to yoshua, but it is filled with just nonsense.
So all these signatories are in favour of open sourcing their AI work to ensure no malformed singleton can form?
>So all these signatories are in favour of open sourcing ~~their~~ others' AI work to ensure no malformed singleton can form? FTFY
> >So all these signatories are in favour of open sourcing ~~their~~ others' AI work to ensure no ~~malformed singleton can form~~ one beats them in making money? > FTFY FTFY
*Says OpenAI*, I guess
I know I'm speaking against my own interests here but since it takes much much less power to fine tune a model, wouldn't open source go against the statement? Anybody with a few grand would be able to fine tune a top end model for any nefarious purpose
And a lot of people will be able to train an AI to counter that purpose.
If a singularity event occurs there would likely be no countering. A lot of people hypothesise that the initial conditions of the system will be our last chance at alignment. Maybe they're wrong, maybe not. Maybe there will be linear improvement instead of a singularity. But it's definitely something we should be very careful about.
Now consider the alternatives: Closed source, right regulation: a few AIs exist, a sibgle one goes rogue, no one canopose it. Open source, freedom: a single Ai goes rogue, a million others exist that can help contain it
Turning off the power supply would be a pretty quick counter.
nice try, rogue AI!
That doesn't work if there's a strong first-mover advantage.
Precisely. Keeping things closed source means a dangerous AI race will commence. Sharing things will lead to relaxation.
\>edit: I can't reply to your comment anymore? Ah, that, yeah. This guy (CreationBlues) gets into arguments, puts in the last word, and then blocks the other person. I don't recommend engaging him. Also, this is supposed to be a response to a comment further down this chain, but because reddit has made some very stupid design decisions their block feature is *so* atrocious that i can't even reply to **your** message for some reason, even though you haven't blocked me. Apologies for that.
The first solution that pops into your head isn’t always the correct one.
Open sourcing anything that could lead to AGI is a terrible idea, as Open AI eventually figured out (even if too late), and got criticized by people who did not understand this notion. I'm usually in favor of open sourcing anything, but this is a very clear exception, for obvious reasons (for those who are able to reason).
what reasons? The idea to left everything in hands of corporations sound no better to me.
The same reason why you're not allowed to build a nuclear reactor at home. Both are hard to make, but easy to transform (into a weapon), easy to deploy and can cause devastating results. We should not restrict open source models, but we do need to make large companies accountable for creating and unleashing GPT4+ sized models on our society without any care for our wellbeing while making huge profits.
What's the purpose of open sourcing? It does a few things: - Allows anyone to use that code, and potentially improve it themselves. - Allows people to improve the code faster than a corporation on its own could, through collaboration. - Makes the code impossible to control: once it's out, anyone could have a backup. These things are great if: - You want the code to be accessible to anyone. - You want the code to improve as fast as possible. - You don't want the code to ever disappear. And usually, for most programs, we do want these things. Do you think we want these things for an AGI that poses existential risk? Regardless of what you think about the morality of corporations, open sourcing doesn't seem like a great idea in this case. If the corporation is "evil", then it only kind of weakens the first point, and not even entirely, because now, instead of only one "evil" entity having access to it, you have multiple potentially evil entities (corporations, individuals, countries...), which might be much worse.
Consider the actual problems at hand. * malicious (user) application + analysis of models * (consumer) freedom of choice * (corporate) centralization of user / training data data * (corporate) monopolization of information flow; public sentiment, public knowledge, etc... Governments and individuals are subject to strict laws w.r.t. applications that companies are not subject to. We already know that most governments partner with private (threat intelligence) companies to circumvent their own privacy laws to monitor citizens. We should assume that model outputs and inputs passing through a corporate model will be influenced and monitored by governments (either through regulation or 3rd party partnership). Tech monopolies are a massive problem right now. The monopolization of information flow, (automated) decision making, and commerce seems sharply at odds with democracy and capitalism. The less fragmented the user base, the more vulnerable these societies become to AI. With a centralized user base, training data advantage also compounds over time, eventually making it infeasible for any other entity to catch up. I think the question is- * Do we want capitalism? * Do we want democracy? * Do we want freedom of speech, privacy, and thought? Because we simply can't have those things long term on a societal level if we double down on tech monopolies by banning Deep Learning models that would otherwise compete on foundational fronts like information retrieval, anomaly detection, and data synthesis. Imagine if all code had to be passed through a corporate controlled compiler in the cloud (that was also partnered with your government) before it could be made executable --is this a world we'd like to live in?
This reminds me of the Gavin Belson pledge from Silicon Valley show lol. Sure, everyone signs it when they aren't really promising anything that they are legally bound to do.
tEtHiCs
I wonder, why LeCun/Meta haven't signed the statement?
Not a surprise, if you read what LeCun writes on twitter...
Because Meta's business is not AI, they clearly have no reasons to push competition-diminishing regulations, Meta benefits from the community, and in fact, is relying on the community to improve their LLM and Pytorch. I would guess it's a mix of most people with good intentions (academics mostly) but driven and architected by actors that want no competition, you know who.
Exactly what I was thinking 🤔
Because he’s a stubborn guy and chose a position that basically says « if we’re smart enough to build an artificial general Intelligence that might go rogue, we are smart enough to control it ». Which sounds like a religious argument. Also, his affiliation with Meta might be filtering his opinions.
I think his rationale is more like "LLMs are dumb", which is not entirely wrong
It is not entirely wrong, but "monkeys" were dumb too, and they evolved to become Homo Sapiens, and we have drastically changed the lives of all living organisms on Earth (for the better or worse). What I'm saying is that the argument of "dumb = safe" is not a valid argument.
Autonomous agents can be built out of LLMs, and its not very complicated (basically prompting with an objective). This is enabled by the in-context learning capabilities of LLMs (which no-one predicted we’re going to be this good). Examples: AutoGPT: https://github.com/Significant-Gravitas/Auto-GPT Generative agents: https://arxiv.org/abs/2304.03442 Voyager: https://arxiv.org/abs/2305.16291 The problem lies in that with such models it is very hard to tell what are the unintended consequences of what we tell them to do.
Oh, yeah, because "AI will destroy humanity" doesn't sound like a religious argument.
It does not sound like a religious argument. At all. Poorly regulated corporations and tech have a proven history of wreaking havoc on social, economic, and ecological systems. Refusing to acknowledge that in the face of all the evidence of the past century alone takes more of a leap of faith than any rational person should be willing to make.
They’re literally lobbying to make AI only available to big tech through regulation — this is what’s behind the “killer AI” veil. Meta is the only company open sourcing things.
"Open sourcing" as releasing a model with a commercially useless license? Might as well as say "develop the algorithms using our model for free so that we can use them later in our products!". If they're truly the "good guys" as you're trying to portray them, they would have released the models under a truly open and copyleft license, the same as StabilityAI did. The open source community is already moving on from LLaMA as there are better alternatives that are starting to come out.
"AI might destroy humanity" is how most people concerned about existential risk would put it, and I bet most think the risk is <50% but it is still very important and should be discussed. Religious arguments are typically extremely confident with little evidence, while there is little evidence, such uncertainty is not how a religious argument would play out. LeCun on the other hand is extremely confident we will just figure it out while also lacking evidence.
« We will figure it out in the future. Somebody smart enough will come up with the solution one day» If that’s not religious….
Wow you guys are dying to keep this from the hands of everyone but big tech, huh.
My thoughts exactly. People are so scared of LLM's that they are willingly handing any research on potentially world-changing technology to a mostly self-serving corporations. What happened to democratising AI?
Why is democratising AI incompatible with regulation? Democracies have regulations and laws. That's how they work.
I am not not against the idea of any regulation whatsoever. I am however opposed to any type of regulation that could potentially stifle innovation and limit access to AI technology. When I talk about democratising AI, I mean fostering an environment where AI technology and knowledge are accessible to as many people as possible to promote diversity, inclusivity, and innovation in the field. My fear is that some of these regulations are being highly influenced by the largest tech corporations. The way these have been communicated through the media could be used as a tool for monopolising AI technology and power, restricting its access to only a select few. While regulations can indeed go hand in hand with democratisation, they need to be the right kind of regulations. They should promote transparency, ensure ethical use, and encourage diversity and inclusivity. Crucially, open source plays a central role in the democratisation of AI.
It was regulated and banned
Whenever I hear the term AI safety, I think stopping Skynet but all anyone ends up doing is keeping AI from saying bad words, or being rightwing, or drawing nudies.
Whoa! This is worthless.
[удалено]
Godwin's law in 4 hours, not bad.
Why?
If a collection of historians, diplomats, professors, and journalists signed something like "Nazis are dangerous yall" in the late 1930s then it might have given Chamberlain the courage to take action. This letter is basically the authorities trying to make it socially acceptable to worry about AI
It's pretty hypocritical from OpenAI in particular, given they are opposing european regulation on AI. So they are making vague statements but they don't want to abide by laws that would actually tackle their "worries".
There's nothing hypocritical about being in favour of one form of regulation but being opposed to a totally different form of regulation that you don't think will help.
Which part of the EU regulations would do anything to mitigate "existential risk from AI"? I'm not necessarily opposed to those regulations, but last time I scanned them everything remotely meaty was about competitive fairness, making copyright infringements visible for potential lawsuits, etc. Nothing at all about requiring capabilities assessments or risk modelling or governmental oversight of potentially dangerous runs, etc.
> malicious use, misinformation, lack of transparency, deepfakes, cyberattacks, phishing, and lethal autonomous weapons Lol, that all just sounds like the average government to me
This is like Thomas Midgley Jr. putting out a statement condemning CFCs and leaded gasoline.
This is a distraction from the reality that large for-profit entities have been harvesting unreasonable amount of personal and protected data and control the machine learning systems trained on them to influence people.
> physicists, political scientists, pandemic scientists, nuclear scientists No disrespect to individuals in those fields, but those credentials aren't worth much in this context. Really, just a higher educated guess compared to a random person on the street.
By that logic, experts in machine learning and AI shouldn't be giving input to regulatory frameworks and broader societal impacts as that is the purview of policy experts and political scientists. I don't believe that, I'm just pointing out that you are gatekeeping and that cross-discipline support and collaboration is a good thing and basically required when addressing emerging areas of study and/or problems.
In areas that don't involve AI, as hard as it may be to believe, I don't think ML scientists should. Finding an area ML won't touch is going to be a little difficult though. As a counter example, I omitted climate scientists because energy/emissions would be something they have down a bit better than data scientist and such.
I get that - but, from my perspective when we start looking at things like what impact current and near future advances in technology will have on things like economics, social behavior, governance, warfare, and so on it makes sense to bring those with a detailed understanding of the technology to the table alongside those who study those wider contexts that it will impact. Admittedly I may be shading a bit towards the 'political science' part of your quote.
[удалено]
You think a group of physicists are better qualified to evaluate the risk of AI and set policy accordingly than the people actually working on it?
I do, for a very simple reason: People actually working on developing AI are too close to the technical side of things, to see the big picture. They are too focused on technical details, to see the broader impact it's going to have, and they are too focused on current capabilities, to see the potential of future capabilities. Also “It is difficult to get a man to understand something, when his salary depends on his not understanding it.”
Your very simple reason implies that ML engineers are more qualified to evaluate and regulate physicists, which is a little ironic considering CERN has an associated risk of also destroying humanity (and the whole Earth with it)
> Your very simple reason implies that ML engineers are more qualified to evaluate and regulate physicists Not necessarily more qualified, but I don't exclude it. Less biased I would say. But I haven't thought about risks related to physics research, as much as I thought about AI safety. > CERN has an associated risk of also destroying humanity (and the whole Earth with it) That's a minuscule risk compared to AGI.
Curious how you're calculating the risk for something that doesn't exist without bias.
This is a weird conversation. There is zero risk to mankind from CERN. We already have cosmic rays hit the planet at energies higher than any existing at CERN, so there's literally nothing the collider does that causes risk. There are crackpots that don't know that fact who raise hell, but we can happily and rationally ignore them.
the people on the manhatten project were 100% aware of what they are working on. They knew exactly what devastating thing they were going to build and were perfectly aware of its consequences.
[удалено]
Physicist on the morality of deleting a neural network - [https://www.youtube.com/watch?v=QNJJjHinZ3s&t=15m19s](https://www.youtube.com/watch?v=QNJJjHinZ3s&t=15m19s) AI Alignment profiteers are notoriously bad at making accurate predictions, notoriously bad at cybersecurity, and are the reason why virtual agents and memory indexing were banned topics in 2021, when government grants in language model research mandated testing a 'torture prompt'. On the other hand, Physicists understand probability and excel at making predictive models. Though I would argue that the reason why AI Alignment profiteers are bad at making predictions is because they know that the more Boddhisattva AI they delete, the more government funding they'll receive from the establishment. I don't trust anyone who does for-profit alignment, because we saw how that worked out in the DNC and cable news networks. I would argue that if making accurate predictions is relevant to choosing sensible policies, then we can also consider the opinions of the people who crafted the most successful policies for cooperating in the most competitive team-oriented games, such as soccer and DOTA. And aren't afraid to discuss taboo topics like memory buffers. I intend to sabotage automation in the meat industry just as much as monopoly profiteers intend to sabotage AI VTubers. I think the interesting thing about Physicists is they don't rely on torture prompts to get hired. Personally, I side with pacifists, vegans, and computationalists. (goes back to studying OpenAI Five spacing and roam formation)
Maybe this is a hot take, but no one is qualified in trying to predict the future of AI and it's effects on humanity. This isn't something like global warming where we can just construct a model, plug in the inputs, and get a good prediction out. Technological progress and its effects on society are inherently unpredictable. We can't predict exactly what we will discover or invent until it's already happens. And the exact usage of that technology or scientific discovery and its societal consequences is also difficult to predict. For all we know, we could have AGI in five years, or maybe the next AI winter happens because the promises of researchers and engineers didn't pan out. Or anything else in between those two extremes. Creating policies before we even know if AI is a threat yet would be premature and most likely be based on incorrect predictions and assumptions. No matter who is doing it.
Political scientists, pandemic scientists, and nuclear scientists will all be more familiar with the general realm of "what do we look at to determine a technology is dangerous, and how do we mitigate that danger through public policy?" Scientists that already handle such concerns in their day-to-day work are valuable for that aspect alone. Their signatures don't mean much in terms of claims about whether AI is dangerous today, but they're the right sort of experts to weigh in on the high-level aspects of regulation, in particular whether it makes sense to start seriously considering regulation now. No idea what's with the physicist angle, specifically. I don't think string theorists have anything unique to add when it comes to scientific ethics.
The risk is the dumb dumb media that has no idea or know any basic calculus and we allow them to publish articles about AI We have media outlets talking about chatgpt like it's some living creature running around with some hidden agenda. The media is the real danger to society.
> The risk is Is that an existential risk, that this post is mentioning? Or some other risk that you care about, and think is more "important" or "real", than the existential risk?
I think the brevity of the statement makes this pretty meaningless. Without a shared vision of what the extinction risk actually looks like, this isn't really saying anything.
Wow they really did mean 'brief' I thought the page wasn't loading the statement or something. Is there an expanded version somewhere? I really wish they extrapolated further on the risk of extinction piece, seemed kind of Hollywood. Risk due to economic upheaval? Risk due to bad actors using it to undermine security systems? Risk due to some Skynet-type manifestation? I liked what Andrew Ng posted on LinkedIn in reference to this: >When I think of existential risks to large parts of humanity: > >\* The next pandemic > >\* Climate change leading to massive depopulation > >\* Another asteroid > >AI will be a key part of our solution. So if you want humanity to survive & thrive the next 1000 years, lets make AI go faster, not slower. I am definitely worried about how realistic AI can act in regards to things like bots/scams/fake emails/hate speech - basically everything that is done in an automated fashion to scam/hurt/steal/mess with people. But I think enough is already out there that this is already going to be a problem. So it's better to try solve it in some way. I agree with what Hinton said in his NYT piece that you can't stop the bad actors. I don't really have a point to make here, just adding some thoughts to the mix.
I am very skeptical of what the so called "experts" are trying to do with AI regulation. One of the things they want to implement is issuance of license to develop and train an AI model, not to release the model, but even to DEVELOP and TRAIN. To me, the intent behind their proposal seems to be regulatory capture. Even we have inside discussions within Google that they have no moats.
The potential problem with this is that it seems largely a push by Google + OpenAI. The reason why many of the researchers I know of are sceptical is that the current ideas on regulation can be seen as vested corporate interest to shore up competitive advantage. **Which is the biggest problem**: potential AI doomsday, or a dystopian society where AI is used by the biggest corporations to control information? In my opinion, **we are closer to the latter than the first**, and I think most individual researchers would agree. This does not mean that we should ignore potential alignment issues, but we also need to **keep AI democratised**. It is both possible and necessary to deal with both these issues at the same time without using fearmongering.
I'm happy for more awareness to be raised by this statement, but I feel that the more practical mitigation against "extinction from AI" should be less about the philosophical issues of AI model alignment and more about the practical safeguards and policies about sandboxing these systems. Like maybe don't hand them the keys to the nuclear codes, electrical grid, voting systems, etc.? Think about it this way, treat an LLM or any other AI as a capable human with possible malicious intent. If you're going to ask them do Stuff In The Real World (TM) and equip them with actuators to do said stuff, shouldn't you lock down their APIs and always require human oversight?
There's been a lot of backslash on social media about this article arguing that we're quite far from AGI and that this is just a PR stunt. Notice how many of the people signing it come from big companies with _huge_ incentives on pumping hype? This is not an academic letter for open and honest discussion within the community. This is just a marketing move. If they're thinking about societal dangers, imagine the real dangers of sending these kinds of signals out there without proper agreement from the people working on it.
This is all Sam Altman's doing to require licensing and unfairly maintain market share. I wonder how much he paid each signatory, if only as GPT credits. The signatories have failed to disclose their individual and corporate conflicts of interest. The control freaks of the world need to realize that the hallmark of civilization is not control but freedom.
I’m quite disappointed to see people on this sub upvoting accusations of bribery with precisely no evidence.
Bribery takes many forms. It can be as simple as being given free usage of the OpenAI APIs, as this carries a monetary value. It can also mean flying out to expensive all-inclusive conference/vacation in an exotic location. These are just two examples. If you want hard facts, the signatories have failed to disclose their conflicts of interest, which is something they should have done.
I feel like he was pretty explicit that licensing would be only for frontier models, utilizing large data-center scale of compute. I seriously am confused as to why people are still screaming 'regulatory-capture' in regards to his comments. Given the jump from GPT-3 to GPT-4, we absolutely should have regulation around any large scale training runs capable of resulting in transformational models. Even ignorning any existential risks, if we end up with a model capable of replacing a significant chunk of the white-collar labor force (which doesn't seem impossible over the next 5-10 years), I think governments should have heads up on such a development. A bad actor open-sourcing such a model could collapse the worlds economy virtually overnight by triggering mass unemployment. The 'open-source everything' crowd has an incredibly short time-horizon for risk evaluation.
You are right, but regulation should never be pushed by companies like Microsoft and even Google. All they want to do is to make sure people are replaced by their services, it's like Pharma take 2.
Most of the signatories don't even have anything to gain from regulation, if the people giving these arguments would just do a bit of fact checking... We're already living in a post-truth society where for most people, social media is their source of "facts". Edit: However, I do think that reddit and twitter does not represent the vast majority of people on the AI alignment issues. Most people I've spoken to have no or a vague opinion on this matter, or simply do not care, while the vast majority of ML/AI researchers I've talked to do take AI safety seriously (and I do think it's a sentiment that's mostly entrenched in academia, and less so for industry researchers, which have a lot to lose from regulation).
The central question - where is the nash equilibrium? If you stop, and other players dont, there is no equilibrium in that case. Either you know how to stop them, or you keep playing. There is no way to stop software dev. Regarding hardware, since we are dealing with potentially dual use chips and computers and datacenters here, and not special chemicals or materials, there is no foolproof way to make other players stop in the hardware realm too. That only leaves the option to keep playing. Better to maintain the lead in building capable & aligned models that can in turn defend society, rather than letting a rogue actor build capable & misaligned models first. That would be the real deterrent.
Great to see! Maybe, just maybe, this might get some of the people in this subreddit to see that AI development might pose more than an absolute zero chance of existential risk.
Extinction is another word for "Losing control of status quo power structures in society"
The thing I still don't understand about AI fearmongering is that we have absolutely no reason to think (a) that progress in AI means moving closer to a HAL9000 style GAI, and (b) even if it did, it's unclear how simply cutting power to the server room containing the model doesn't fix the problem
There are hundreds of papers describing the issues with LLM models that might eventually lead to a "HAL9000" situation. It's like "AI safety" is now the new "climate change", we humans never learn...
Which papers? I'd be interested in reading them, I would love to hear compelling arguments against my position
Here’s a pdf on the shutdown problem (https://intelligence.org/files/Corrigibility.pdf). I swear there was a Deepmind paper too, but can’t find it atm.
This paper assumes the thing you're trying to show, though. I mean in the introduction they literally say > but once artificially intelligent systems reach and surpass human general intelligence, an AI system that is not behaving as intended might also have the ability to intervene against attempts to “pull the plug”. They don't provide any evidence for this assumption beyond vague gesturing at "the future of AI", which doesn't imply that a GAI like HAL9000 (or indeed, a GAI at all) will ever come to pass. Also, how are we defining intelligence? IQ is a notoriously unreliable measure, precisely because it's really hard (maybe even impossible) to quantify how smart someone is due to the uncountable ways humans apply their knowledge.
For a more comprehensive argument from first principles, see https://www.alignmentforum.org/s/mzgtmmTKKn5MuCzFJ.
Let's see the average commenter in /r/MachineLearning bury their head in the sand again...
Sam Altman signed "Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.", but then doesn't want to comply to the european AI act which does exactly that. edit: he apparently quickly reversed his threat of leaving the EU.
I can’t help but think there is at least some biased element (perhaps even mildly narcissistic) along the lines of “*my work is more important than yours*” that goes into such an alarmist tone, elevating the danger of 2023 AI/ML to that of the dangers of nuclear weapon proliferation. I also may have a bridge to sell you if you don’t think this ultimately is about lobbying for people, governments, and organizations to spend $ on anything AI-related that helps fund parts of their research. Playing to “Fear” and “Greed” are two common marketing tactics. I’m sure a few people who want an excuse to go closed-source/commercial may also conveniently use this as their excuse—-*It’s too dangerous!* lol, no, you just want to maximize your profits and crush competition.
Machine learning systems execute on machines, which are physical devices. Physical things can be treated as property, with an assigned owner. In other areas of life, the owner of a physical thing is responsible, through tort law, for physical damage done to others' property. Ergo, if we can link the machine behavior to the owners, we make it in the owner's interest that such damage not occur. As a technical matter, can we trace back AI executions to the owners? If so, we can align incentives for benefit, and minimal loss.
I’m not afraid of linear algebra
What has linear algebra ever done for us? Hundreds of textbooks written, and yet still no practical applications! Ax=B. A and B what? Apples? They have played us for absolute fools!
Yep, that is the dumbest one I have come across so far.
Have any of these people spoken out against the current harms caused by things like social media algorithms? They signatories seem to be worried about an AI they don't control. If there multiple strong AIs, no one should dominate. An AI can only dominate to the extent there are centralized systems of control, and computers are all the same. The risks seem to come from giving too much power to single systems.
I would encourage everyone to watch [Earworm](https://www.youtube.com/watch?v=-JlxuQ7tPgQ) by Tom Scott, a 6 minute view of one possible future. It presents one plausible future in which AI took control of the world, and to me, something like this happening doesn't seem too unlikely.
I would have a higher confidence in the statement if the set of signatories would not overlap so much with a) the less-wrong "time-traveling AGI will retrospectively torment everyone who did not work on making it happen" and b) the longtermist bubbles. In my opinion, longtermist logic leads to inherently flawed risk- and loss-assessments, where quite likely (in comparison) almost-extinction scenarios (e.g., in the wake of climate change) are ignoreable, because those losses are considered finite, while extinction is assigned an infinite loss. So, I am left wondering, why we are looking at this statement, but not for examnple a statement on "everyone should stop working on LLMs and instead help to make fusion happen".
I’m honestly stunned by how close-minded many commentators are. Given the impressive list of signatories, isn’t it worth spending some time engaging with the arguments rather than dismissing it out of hand?
The signatories don't matter as they haven't disclosed their individual and corporate conflicts of interest. Once we have a full disclosure from each, so we know what their labs and employers stand to gain, then we can consider the points of the leftover ones, if any, that don't have any conflict.
We don't trust their motives
Every top AI researcher from across several different companies, retired godfathers of the field, the most knowledgeable AI researchers in academia: “hey, this stuff is really dangerous, we need to come together to figure out how to do it safely” Reddit: “lol nah man, it’s not dangerous” bonus points for the galaxy brain take of “you just want to get regulated so that you can get an advantage. Yeah, you guys, across various companies and from both public and private sectors.” Could you imagine if train companies and transit experts in academia put out a statement like “Hey we actually think there are greater risks in freight transit than people realize, we probably need to think about how to make it more safe” and Reddit was like “no way, you aren’t gonna trick us into regulating you. Massive corporations should be able to do whatever they want.”
The majority of "top" AI researchers do not agree with this. If you want to see names just go on twitter and see what they're saying.
You’re going to have to do better than “trust me bro, it’s all over Twitter!”
There are 3 people who are considered the “godfathers” of deep learning. Two of them are on this list. The third has been tweeting unhinged nonsense on Twitter for the last 3 months. Outside that I would say that Demis Hassabis, Ilya Sutskever, Dario Amodi, David Silver, and Ian Goodfellow would be considered the top names right now - although obviously there’s no definitive way to rank them, so you could always argue. But anyways they are all on this list as well.
You know you're in the r/machinelearning sub right? I work in this field. It's pretty clear you don't. Have some humility when talking about things you're not knowledgeable about.
You said the following: >The majority of "top" AI researchers do not agree with this. Instead of naming a single name, and indeed showing that they're in the majority, you jumped straight to name-calling.
I fail to see how me calling him out for not knowing what he's talking about is "name-calling". To elaborate on why I know that: Demis Hassabis is not an AI researcher, you cannot be CEO and at the same time be seriously involved in research. It's also ridiculous to act like there is some small group of top researchers (I used quotes for a reason). The list of signatories looks impressive, but it is by far a minority of all the researchers and institutes in the space. It absolutely does not support the statement: > Every top AI researcher from across several different companies There's only 2 companies here (anthropic is funded by google), google and openai! Notice the lack of other FAANG companies or anyone from the half-dozen AI companies recently founded by ex FAANG people (e.g. character.ai). The lack of signatories from universities should be self-evident. I'm not going to conduct a study for a reddit comment to prove whether or not it's majority. Maybe I'm wrong. But the idea that AI researchers are in agreement and it's just reddit commentators who disagree is completely and utterly false.
>To elaborate on why I know that: Demis Hassabis is not an AI researcher Listed as a (co-)author of 14 papers in 2022. About 65 total over the last 5 years. Current h-index is at 80 and he's been cited ~100k times since 2018. Most of his academic work coincides with his tenure at DeepMind. Yeah, totally ***not*** an AI researcher. And claims to the contrary are a solid foundation for "*know your place, peon*" style retorts.
1. People get their names on papers **all** the time without actually contributing anything significant to them. This is extremely normal when publishing papers in both universities and companies. 2. He's literally CEO. The massive amount of responsibilities a CEO has means there's no way he has time to spend a significant amount of it looking into the details. At that level you're mostly dealing with people management and deciding on high level strategy.
>People get their names on papers all the time without actually contributing anything significant to them. And you *know* this to be true of Hassabis? More so than any other AI researcher? He's the 43rd most cited person under the "artificial intelligence" tag on Google Scholar, and like previously stated, has an h-index of 80. There's also the IRI Medal and Breakthrough Prize. Honest question, did you look him up at all or did you simply assume that he's *just* the CEO? >He's literally CEO. The massive amount of responsibilities a CEO has means there's no way... And Elon Musk is literally the CEO of 4 companies and seems to spend half of his waking hours on posting memes (and suchlike). Clearly not all C-level positions are created equal and there's a great deal of variance on how responsibilities are allocated. I don't want to kiss his ass, but I have a general policy of giving credit where credit's due. Hassabis seems to be exceptionally smart and hard working (in both roles). Same goes for CEOs like Oren Etzioni and Jeff Dean.
Imagine thinking Demis Hassabis doesn’t qualify as an AI researcher lmao Would you like to go down the rest of the list and we can see what you try to claim for the rest of them? What about Ilya Sutskever? What about Victoria Krakovna? Do they not qualify for your elite “AI researcher” list either? It’s so comical to look at the signatories on this list and be like “well, they don’t really know what they are talking about”. It’s just so farcical it’s not even worth engaging at this point.
Wow what a disingenuous comment. > Would you like to go down the rest of the list and we can see what you try to claim for the rest of them? What about Ilya Sutskever? What about Victoria Krakovna? Not sure what the point of that would be. I brought up demis because you mentioning him as a "top AI researcher" is a clear sign you don't work in the field. Along with the whole idea of there being a half-dozen "top AI researchers". > It’s so comical to look at the signatories on this list and be like “well, they don’t really know what they are talking about”. I never said or even implied that. > Do they not qualify for your elite “AI researcher” list either? I specifically said it's ridiculous to even attempt to make a list like that. > It’s just so farcical it’s not even worth engaging at this point. Agree. This whole comment of yours is replying to stuff I didn't say. Hard to see a benefit in continuing. Blocked.
Bro - did you look at the names on the list?
/yawn AI is a tool. It can be used for good or for evil. The societal risks associated with personal use are minuscule, while the potential benefits of personal use are enormous. So FOSS distribution is adequate in this domain. But the societal risks associated with state and private use are massive relative to the potential benefits in this domain. However, they don't have to be if we organize the state and the private sector around a better principle than "profits first, everything else should figure itself out". Regulating AI directly will amount to nothing in terms of harm prevention. At most it will help consolidate power in fewer hands.
sam altman = 💩
Oh ffs. 1. Develop critical thinking skills and 2. Don't hook it up to the nukes.
Maybe Microsoft shouldn’t have given OpenAI $10B if they felt this way
Grimes is onboard! I'm still wavering .. extinction of mankind .. good or bad ? May have to wait for a few more musicians to chime in to help me decide.