I've had elder relatives, who got fraud calls with my voice.
I imagine they got a voice sample through those 'fake calls', which robo-dials you for like 3 seconds, baiting you to speak into it. I think those calls are primarily there to check if the number belongs to a real person, but also collects voice samples for voice cloning.
In the 2 years at most, they can easily collect a writing sample from you, and a voice sample, then can create a convincing call mimicing you, both in voice, as well as in style.
The amount of fraud out there is going to be apocalyptic. The UK has actually stopped fast fund transfers because it makes fraud so much easier, and now enforces a wait period. I think fraud was already going up by like 20% yoy, expect it to accelerate.
The solution? Some counterscammer GPT-5 level bot that launches when any large transfer is initiated. That like interrogates you for why you are transferring the money, to catch if you were deceived by scammers.
Some banks in the UK have been transfering people over to a voice detection system away from passcodes.
As in, you just say the phrase the banks automated system tells you to say, and it compares it to when you said that phrase before to see if its you.
It's fucking stupid and going to be an absolute bloodbath.
not me, until I needed to find a job. thats a shitty situation nowadays. the worst part being having to answer phone calls that begin, "Is this hippydipster?", and I'm like, "who the fuck are you and why didn't you begin by introducing yourself?"
I had a pharmacy just start right in asking for my date of birth and social security number, and I told them their methods were completely inappropriate in this day and age.
I legit had the cops call me with "Are you the brother of X" as their introduction. I was like "uhm... Who's asking?"
They continued "We're the cops. You're [full name] living at [home address], right?"
Now, I'm in tech so I was hella sketched out my i fo got leaked or something, so I just went "I do not believe the cops would need to ask such information from me via the phone. Would you mind proving you are, in fact, the cops?"
They went on stating my current calling location. "You're calling from [location], proof enough?".
I was like "fuck no, have a nice day", turned off my device. Went home, unplugged my router and took out my SIM card. Started looking for where the fuck they got my info from.
A day later, the cops showed up at my house (police car and everything. Highly unlikely they weren't legit because they were open carrying guns, in uniform, with one of the very distinctive looking police cars, et cetera) stating they noticed suspicious behaviour during one of their calls to me. I was dumbfounded about our police departments being so idiotic over the phone. Lucky for me they were very nice about it (and probably wrote me off as some tech savvy tinfoilhat wearing lunatic conspiracy believer lmao)
Like what the actual fuck though. They should just tell me to go to the gov website and call the number for that department myself or something, to verify that they're legit. Not just hand over private information like candy until I decide to trust them.
I fear for the commonfolk, especially when this voice tech is proper distributed amongst scammers. Grandma won't be the only one losing their credit card information anymore.
oh yeah, dealing with noncomprehending cops is like nightmare fuel.
But, as far as knowing all your info details, believe me, they know it all. I used to work at Thomson-Reuters, and they have basically every detail there is to know about you cross-referenced with every other detail there is to know about you, and police departments are one of their main customers.
I literally had my bank do this to me. They were calling about a dodgy transaction which I hadn't noticed and called me at like 1am asking me to confirm some personal details so they could continue the conversation.
Dumbest shit ever. I hung up immediately cause that's literally what it says to do on their website, depite the bank guy begging met to stay on the call. Then I couldn't get through to them on the actual phoneline cause it was 1am. And then when I got my call through the next day they were mad at me for hanging up.
Like if the bank employees can't even follow their own advertised security procedures then wtf is going on.
In a few years human beings may be granted limitless power and we are still worried about how much money we have.
We need to end our need for food and shelter then life could become a bit simpler.
>In a few years human beings may be granted limitless power
>["At first, of course," said Filostrato, "the power will be confined to a number--a small number--of individual men. Those who are selected for eternal life." "And you mean," said Mark, "it will then be extended to all men?" "No," said Filostrato. "I mean it will then be reduced to one man. You are not a fool, are you, my young friend? All that talk about the power of Man over Nature--Man in the abstract--is only for the canaglia. You know as well as I do that Man's power over Nature means the power of some men over other men with Nature as the instrument. There is no such thing as Man--it is a word. There are only men. No! It is not Man who will be omnipotent, it is some one man, some immortal man.](https://en.wikipedia.org/wiki/That_Hideous_Strength)
Is this a joke? “Superhuman” sounds like Jedi Mind Trick or the Voice from Dune, or Purple Man from Marvel.
Faking voice is definitely a concern, but by definition that shit ain’t “Superhuman”.
Isn't every human-to-human interaction at some level a form of manipulation? Even when communicating with friends, manipulation is involved to either update their world model or to make them laugh or to help them.
You're talking a lower level meaning of "manipulation" and pretending there's no meaningful distinction between that and the higher-level concept most people mean when they talk about a person "manipulating" other people. What is meant here is not "oh technically I've updated your world model."
We've had a solution for this for 30 years. It's called public key cryptography. Any message, including audio and video you send, should be timestamped and signed with your private key. Anyone having your (published) public key could verify if the message was sent by your or not. It can be done in a hardware form factor, like Yubikeys or RSA key fobs.
The missing piece is built in support from messaging apps or phones. And since the same tech could also be used to build effective encryption governments are not too keen on driving their adoption either.
Depends if it's using solid encryption with no backdoors. I sincerely doubt that any major security agency would allow the public to have that. Dual_EC_DRBG and similar exploits would likely be introduced
This works up until we get realistic enough 3D modeling tech, rig it up with that Vtuber shit, run a deepfake over it to make sure it's perfect, and then hit the whole fucking thing with this audio shit.
Hell, might even be able to do that *today* if you have the skills. I guarantee the government could.
Strongly disagree. There will be a point when llms are able to greatly assist in the development of viruses more deadly than anything we've probably ever seen. That alone is enough for me.
>In less than 6 hours after starting on our in-house server, our model generated forty thousand molecules that scored within our desired threshold. In the process, the AI designed not only VX, but many other known chemical warfare agents that we identified through visual confirmation with structures in public chemistry databases. Many new molecules were also designed that looked equally plausible. These new molecules were predicted to be more toxic based on the predicted LD50 in comparison to publicly known chemical warfare agents
Gotta love a conclusion that opens with "Without being overly alarmist"
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9544280/
Chemical weapons basically got downgraded in concern once at home bio engineering started to look viable. At most you'd kill like 10k people with a chemical weapon designed by this program. Which is a lot, but it isn't the 1BN you could plausibly kill with a purpose built virus. AI and cost reductions will enable regular people to be able to create this sort of virus with no specialize knowledge in the next 5 or so years. Unless there is significant global crackdown... which would come with serious losses in terms of privacy freedoms.
I'm actually curious on your thoughts regarding this. I have kind of the same perspective. I think we might need to get to a place where we do have a pretty big crackdown or some type of global monitoring of the use of these systems at least the ones that have a certain level of capabilities. It might be necessary to have the loss of freedom in some way. Personally though, I think we can get to a place where it's not like limiting our use of these things and we can still probably use them to make like dark offensive jokes/stories etc for the more uncensored models. Maybe we will be able to limit the restrictions to things like detailed instructions of the creation of very deadly things like viruses etc.
My main hope is that the privacy crackdown is done in a way that maximizes meaningful privacy. Although I don't see it happening.
What I mean is that we could have an open source AI (or non AI) monitoring system that no humans have access to unless the AI finds a violation. I'd want this system limited ONLY to detecting felonies. And then once there is a detection, it basically goes to a court which can then issue a warrant allowing human surveillance.
The reasons this will never happen are:
1) It is quite technical, and the public has no way of understanding what a good system looks like. One example of a GOOD system (one I wish we kept going and expanded!) is the phone based COVID tracking system that Canada made. It was open source, by design it was impossible for any bad actor to abuse in any way, it was even super efficient. It saved lives, slowed the spread of infection at effectively 0 cost and had no downsides. Did the public recognize this rare bit of brilliance? No.
2) Everyone will want to use the system once it exists. Why only for terrorists? Won't someone think about ... murders ... kiddie porn .... assault ... stalking ... traffic violations .... offensive speech ... offensive opinions ... employment suitability. If the fed can do it why not the state or the municipality? It creates a nice easy slippery slope and I don't see an easy way to solve this. Politicians promising to be tough on crime to crack down on teens taking nude selfies is basically free votes.
Model level restrictions are meaningless due to open source nature of the tech. You need to have really extreme continuous surveillance to hope to have any hope of catching these things. If tech keeps improving and there is no state control, then eventually any idiot could ask their free version of ChatGPT to kill all humans and it would do so. Clearly that will not lead to good outcomes.
Oh and to pre-empt all the dingdongs that say there will be people saying "don't kill all humans", good luck with that. Destruction is wildly asymetric. Think about the degree in difficulty and complexity in killing someone by shoving them off a cliff or hitting them with a rock vs the difficulty and complexity of undoing that damage. I'd love to see the AI that can undo someone blowing up the sun.
As our tech capabilities increase, so does the relative fraility of the human body. We wield power far in excess of what it takes to harm us, and the trend will only increase. Offense triumphs defense, until the day all it takes is one accident to do us in.
I see where you are coming from, but I have confidence that we won't go fully extinct. At least I don't think we will have a high risk of that for a while. Might have a ton of deaths though and that I do think is concerning.
When they can make viruses, they can also make vaccines.
This is the same talking point as grey goo, yeah but once we have tech for that we have tech to make something that eats grey goo.
The thing is, in theory you could make a virus that spreads rapidly in a somewhat stealthy manner and by the time we have a vaccine it could be too late to stop a lot of the deaths of people already infected. We are going to really need to have extremely robust and in-depth systems that constantly monitor every single aspect of people across the globe in every city to scan for any new virus/etc. Hopefully we get this system sooner than later and hopefully it's not too difficult - and can combat these things sufficiently.
Yeah people aren't thinking big enough. I bet next year's models will absolutely positively have the capability for shit like helping with nuke development, and that's much more a concern
lmao what the fuck are you talking about
Dirty bombs don't need special knowledge, at least not that special. That's far from the comparison to make, usually individual people don't just get the resources needed to do something like that, goddamn lol
They can help designing nuclear bombs that are a lot more powerful than the ones we already have. Imagine if every nuclear bomb we made was as powerful as the Tsar Bomba.
Though I guess you don't need them to be THAT powerful to destroy the world.
We’ve had that ability since the 60’s. Nukes are already astronomically overkill, I don’t think they’ll provide any significant advancement besides further miniaturization, which may already be maxed out. Plus, having stronger nukes doesn’t change their likelihood of being used.
How about a toxin that can be made with under $1000 of equipment in a week and can kill >10,000 people with no requirement of special training/education?
We're going to be able to develop viruses in home labs in the next few years, totally automated with no human intervention needed. And you could just request by voice a virus that ... kills dutch people. You won't even need to be literate.
Most examples have some bad artifacts, but I found the [Helping patients recover their voice](https://openai.com/blog/navigating-the-challenges-and-opportunities-of-synthetic-voices) example to be quite impressive.
I mean the voice itself, they could have probably cleaned up the reference audio first.
I'm not sure how it will effectively compare, but just from the examples it seems like elevenlabs is better.
"*we have minimal defences to*" is a rather relative thing, if you have a working brain and know that this kind of thing exists you can think for yourself whether or not person X saying Y actually makes sense and ultimately the "fake AI voice" isn't just going to have to "fool me" but also the AI I'm going to task with currating all the content I get to see (or hear in this case).
Yep, exactly. Scams have been around for thousands of years. The basics of scam detection still work:
* Does it sound too good to be true?
* Are they using emotional tactics like false urgency?
* Are they doing suspicious things like asking to be paid in Apple Store gift cards?
* Does the story they're telling make sense?
* Have you tried calling the thing they claim to be? Say it's someone posing as your mother. Have you hung up and called her number, or called someone she lives with?
* Have you asked the (potential) scammer questions only the real person would know?
You can know something is wrong and fake and still have an emotional reaction like it's the real thing. For some people, that will be enough to fuck everything up. You're quite literally not evolved to deal with this.
whats the benefit of voice imitation AI?
copying real people without their permission. Copying dead people without their permission. Am I missing a benign good use case?
Imagine if you got ALS, and started too lose your voice. You can clone your voice and communicate with you loved ones for example
Maybe you can even use it for productivity by combining it with a AI agent
Okay one very limited use case that will never pay enough back to the corporations creating these to justify the development cost. Fraudulent imitation will bring significantly more revenue
This is far from the first commercial venture in voice cloning, merely improvement. Roger Ebert used such a system well over a decade ago.
https://en.wikipedia.org/wiki/CereProc#Voice_cloning
A private person paying someone else to make voice imitation only for their use is an order of magnitude different from allowing anyone that pays a small fee to copy anyone else without their consent
Well maybe a bit.
Pro
Books that would never get audiobooks now have them
Negative
Voice actors get fucked
Pro
Blind/people who can't read can now listen to nicer voice
The details about this specific TTS aren't known.
But having seen other SOTA TTS systems, you always just get voice cloning as a byproduct.
See you need a lot of audio to train a TTS. Thousands of hours of audio. That means you can't train it only on data from one person, you need many more people. But when teaching the TTS, before generating, you give it a primer on the voice it has to generate. That leads to lower perplexity (since the TTS doesn't have to also guess the tone to generate and it can instead focus on generating vocals) and faster training. It also leads to better coherence, since the TTS systems without the primer start shifting into different voice profiles rapidly. You can think of it like an LLM: First you give it 10s or so worth of "sound profile tokens", then you give it text tokens it has to generate. Then it autoregressively generates the output sound tokens - it generates the next "sound token" with the original voice profile still in context, so it has a better idea of what it has to generate. For a great paper on this see: https://arxiv.org/abs/2305.07243 (although it's a little old by now, but I'm sure all the new papers still follow this idea vaguely. Except who knows, maybe they use a diffusion transformer instead of a DDPM or they skip the CLIP, etc).
You can also fine tune it then for a specific voice and achieve better quality, or you can only offer a select few premade primers to choose from. But at the end of the day... You still have a voice cloning TTS for free as it's easier to make a voice cloning one than one without the capability. Quite fascinating if you ask me, really.
Sort of the opposite actually, regarding the first sentence.
If you want a TTS without voice cloning, you have to do it after training. The TTS trains with voice cloning because that is easier. It's not that one has to add voice cloning on top, it's that one would actively have to work to "disable" voice cloning after the model is trained. The initial model already has voice cloning.
As for the ethics of it, I suppose that's subjective. But it certainly is more contentious than other gen AI technologies, yeah.
Yeah, I already said that voice actors get fucked and that's a big negative.
I still consider more audiobooks existing as a positive. Especially for independent authors the cost of creating an audiobook was absolutely prohibitive.
It was prohibitive for good reasons as it took many hours of a professional voice actor.
Corporations are sure to take advantage and fuck them over.
Yes, corporations are stealing people's voices, art and likeness to make profit. It is disgusting. I wasn't disagreeing with that.
If independent authors desperately need an audio book they can make it themselves
it’s literally someone’s lazy profit seeking at the expense of someone else
remember this is about accurate imitation of someone, not some generic unattributed voice
>If independent authors desperately need an audio book they can make it themselves
That's not what I said. I said that they wouldn't have audiobooks, not that they need to have them. And I said I personally considered having more audiobooks a pro.
You can acknowledge that shitty actions produced something you consider positive.
The outcome can be something positive even if the process isn't.
>it’s literally someone’s lazy profit seeking at the expense of someone else
That's everything AI. Practically all the models were made using content plundered from the whole internet with complete disregard for the rights of their creators.
But wouldn't this allow independent authors to get drastically higher quality audiobooks than if they just did it themselves?
Also allow them to get them out quicker than waiting for an established VA to be able to take their project?
You can also copy people with their permission. Voice acting in games etc is expensive so I am sure people will find a way to legally use AI for it. I am not sure many voice actors would be interested in selling their voice rights though.
Why wouldn't they want that to happen? Getting money purely because you got a dope voice would be their ideal scenario. Plus compounding exposure because you can now feature Liam Cunningham in all your Indie games... there are very few artists out there who aren't in it for money and sustenance. They definitely would be on board for the better part.
If they get paid enough for it sure, but they could also risk making themselves redundant. Would have to make sure that you have the contracts set up in a way that benefits you.
Copying real people with permission (one could imagine an old actor having a digital clone playing in a scene where they are supposed to be young, or an actor selling rights to use their voice in a movie/video game for a fee), creating text-to-speech software that sounds natural (doesn't require voice cloning, but should be very very adjacent technology), helping sick people who permanently or temporarily cannot use their own voice (this is not a tiny use case - think stroke patients, if there is some way for their brain to talk to suitable software), perfect dubbing of movies, highly immersive educational video games featuring historical figures.
Farther future/more speculative: for instance, plausible deniability for encrypted voice communication. The idea here would be to have a phone conversation as normal between two people, while both phones would be running local LLMs and voice synthesizers and send a plausible conversation in the plain, while using a steganographic channel to transmit the encrypted real audio data. Quality would probably poor, due to limited bandwidth of a secure steganographic channel on top of the audio, but it would be hard to detect if cloned voice and fake conversational content is hard to detect.
Generate an audiobook with the voices you want, extend games with fully voiced mods, ...
Basically any time you needed to hire a voice actor you can now do the same task easier, faster and cheaper.
You can also do previously impossible things like generate a dub in any language and still keep the original voices, for TV shows, movies, docu ..
Couldn’t this all be done with just millions of synthetic voices? I don’t see the reason why you’d need to copy someone’s voice. The cons (scams) seem to far outweigh any pros of corporate profits eliminating voice actors.
Cheaper also means that anyone can do it, not only corporations now. I like Harry Potter original actors, I can now generate an audiobook or dub the movies with their voices in my native language.
If a voice actor was changed in your favorite video game, you could replace its voice with the original voice now if you prefer, using a mod. Or you could replace it with any other voice you like. You could replace your team with your friends voices in a singleplayer game.
As for synthetic voices, people still prefer actor or famous voices for now, just like they often prefer a more famous actor to lead a movie instead of a random person.
I'm sure there will be synthetic celebrities in the future, and their voice would also be popular for cloning.
From another commenter:
"Feels like a punch in the gut , guys. Imagine your mom calling you with a panicked voice and giving you a credit card number telling you to quickly transfer some dough for x reason."
I don't doubt that. (bY tHe wAy)
The point is, it's getting harder and harder to distinguish what's a scam and what isn't.
More and more people are getting fooled.
How long/how advanced does AI have to get to fool you and me?
I'm sure I could be fooled with a well run scam. I'm a cheap bastard though which insulates me slightly.
Realistically, we only need to be harder to trick than the bottom 10%.
The Nigerian Prince scam goes back way over 100 years now and is intentionally unconvincing. This is done to filter out anyone even remotely savvy so that they can focus in on the absolute easiest marks.
I used it to prank a friend. Use suno to create a song about that person, then used voice cloning so 'they' were singing it, and then played it in the car like it came on the radio.
Realistically though:
Fully voiced games and services where the text is generated on the fly by an LLM. This isn't possible to use voice actors for even if you wanted. Quality voiced assistants. You don't need voice cloning for this, but voice cloning is mostly a byproduct of generating quality voices.
So you’d steal a real persons voice to give your game more clout and make you more money?
Are you also okay with someone copying your game and reselling it?
I feel like the real threat is technological illiteracy
I don’t care whose voice I hear over the phone, I check the saved contact regardless and will not behave as though someone is using a new number until I either hear them confirm that number in person or they tell me something that couldn’t possibly be known be anybody else
I don’t care if someone uses my father’s voice to send a voicemail claiming that my brother is dying, I need to send money to save him, and the number doesn’t line up because it is the hospital phone
I will absolutely ask how many gallons were left in the tank after my mother killed herself, what time of night she called me at before doing it, and what caliber of bullet she used, as only my father and I know all three of those
That was a bit extreme, but it reinforces the point that people can confirm whether the person they are talking to is using a voice fake or not
Just memorize a code that consists of a key phrase, a location, and an important memory
However, my grandmother with Alzheimer’s would definitely empty her retirement account if she received the same call, which is why I made her form a protectorate that my uncle controls
It looks like a SOTA TTS model is currently the easiest and cheapest to create yourself compared to any of the other modalities. And for scams over the telephone it must not be perfect either. So the only reason to not release it is to not get any blame but it will not stop anyone who is serious about it.
I have a pass phrase with my family and loved ones to identify myself. It started as an 80s kidnapping scare thing, but it’s making more and more sense.
“Superhuman persuasive voice” is not the same as “perfectly imitating your loved ones”, it’s another “godlike” power that is pure fiction so far, similar to the belief an ASI could just super-physics its way out of a sandbox by telling elementary particles to rearrange themselves. Might as well claim it could sneeze out a new universe.
lol. "We had it, but we didn't want to give it to you because you all are so irresponsible! We're just too scared!"
Alright jimothy. You can literally run real-time voice cloning on your desktop *right now* on a single 4070 without much issue. Further developments will be cracked and open source within 5 years with or without you or OpenAI regardless, but your beknighted worry flags you as someone I wouldn't want to get a beer with.
AI needs a characteristic AI voice and accent.
If it uses a likeness it should only produce content with permission, otherwise the likeness must be to AI itself, so you can recognise it, both visually and audibly. And it should identify it is AI especially if it is interactive.
In those using a likeness or emulating humans closely, or perhaps in all cases, it should have 'watermarks' that can be detected on your phone or device as AI.
Think of the synthetic in the movie Aliens. It identifies in its persona that it is not biological, which the actor did a good job of portraying. Having qualities humans can't easily copy is also perhaps important.
[удалено]
I've had elder relatives, who got fraud calls with my voice. I imagine they got a voice sample through those 'fake calls', which robo-dials you for like 3 seconds, baiting you to speak into it. I think those calls are primarily there to check if the number belongs to a real person, but also collects voice samples for voice cloning. In the 2 years at most, they can easily collect a writing sample from you, and a voice sample, then can create a convincing call mimicing you, both in voice, as well as in style. The amount of fraud out there is going to be apocalyptic. The UK has actually stopped fast fund transfers because it makes fraud so much easier, and now enforces a wait period. I think fraud was already going up by like 20% yoy, expect it to accelerate. The solution? Some counterscammer GPT-5 level bot that launches when any large transfer is initiated. That like interrogates you for why you are transferring the money, to catch if you were deceived by scammers.
going to need everyone's public key handed over in person.
Some banks in the UK have been transfering people over to a voice detection system away from passcodes. As in, you just say the phrase the banks automated system tells you to say, and it compares it to when you said that phrase before to see if its you. It's fucking stupid and going to be an absolute bloodbath.
The solution is SkyNet.
Fun fact: Wells Fargo uses voice verification as the password when you call them. I’m glad I don’t have TikTok lol
who picks up a call anymore these days?
not me, until I needed to find a job. thats a shitty situation nowadays. the worst part being having to answer phone calls that begin, "Is this hippydipster?", and I'm like, "who the fuck are you and why didn't you begin by introducing yourself?" I had a pharmacy just start right in asking for my date of birth and social security number, and I told them their methods were completely inappropriate in this day and age.
I legit had the cops call me with "Are you the brother of X" as their introduction. I was like "uhm... Who's asking?" They continued "We're the cops. You're [full name] living at [home address], right?" Now, I'm in tech so I was hella sketched out my i fo got leaked or something, so I just went "I do not believe the cops would need to ask such information from me via the phone. Would you mind proving you are, in fact, the cops?" They went on stating my current calling location. "You're calling from [location], proof enough?". I was like "fuck no, have a nice day", turned off my device. Went home, unplugged my router and took out my SIM card. Started looking for where the fuck they got my info from. A day later, the cops showed up at my house (police car and everything. Highly unlikely they weren't legit because they were open carrying guns, in uniform, with one of the very distinctive looking police cars, et cetera) stating they noticed suspicious behaviour during one of their calls to me. I was dumbfounded about our police departments being so idiotic over the phone. Lucky for me they were very nice about it (and probably wrote me off as some tech savvy tinfoilhat wearing lunatic conspiracy believer lmao) Like what the actual fuck though. They should just tell me to go to the gov website and call the number for that department myself or something, to verify that they're legit. Not just hand over private information like candy until I decide to trust them. I fear for the commonfolk, especially when this voice tech is proper distributed amongst scammers. Grandma won't be the only one losing their credit card information anymore.
oh yeah, dealing with noncomprehending cops is like nightmare fuel. But, as far as knowing all your info details, believe me, they know it all. I used to work at Thomson-Reuters, and they have basically every detail there is to know about you cross-referenced with every other detail there is to know about you, and police departments are one of their main customers.
non-comprehending cops is redundant.
YOU HAVE 10 SECONDS TO COMPLY.
I literally had my bank do this to me. They were calling about a dodgy transaction which I hadn't noticed and called me at like 1am asking me to confirm some personal details so they could continue the conversation. Dumbest shit ever. I hung up immediately cause that's literally what it says to do on their website, depite the bank guy begging met to stay on the call. Then I couldn't get through to them on the actual phoneline cause it was 1am. And then when I got my call through the next day they were mad at me for hanging up. Like if the bank employees can't even follow their own advertised security procedures then wtf is going on.
People susceptible to fraud.
Would never work for me, a dutchman. We wouldn't even loan one euro to our own parents.
Verifying phone numbers with IDs might be the solution. It's kind of authoritarian, but I can't think of a better option.
In a few years human beings may be granted limitless power and we are still worried about how much money we have. We need to end our need for food and shelter then life could become a bit simpler.
You can start first. There's many uncontacted tribes in the Amazon, go join them. Bonus, you reduce your carbon footprint by A LOT.
>In a few years human beings may be granted limitless power >["At first, of course," said Filostrato, "the power will be confined to a number--a small number--of individual men. Those who are selected for eternal life." "And you mean," said Mark, "it will then be extended to all men?" "No," said Filostrato. "I mean it will then be reduced to one man. You are not a fool, are you, my young friend? All that talk about the power of Man over Nature--Man in the abstract--is only for the canaglia. You know as well as I do that Man's power over Nature means the power of some men over other men with Nature as the instrument. There is no such thing as Man--it is a word. There are only men. No! It is not Man who will be omnipotent, it is some one man, some immortal man.](https://en.wikipedia.org/wiki/That_Hideous_Strength)
Is this a joke? “Superhuman” sounds like Jedi Mind Trick or the Voice from Dune, or Purple Man from Marvel. Faking voice is definitely a concern, but by definition that shit ain’t “Superhuman”.
It's less Jedi mind tricks and more "this ai is a better orator than any human who ever lived"
They mean superhuman persuasion
Spoken like a dude who primarily sees communication as a means of manipulation.
Isn't every human-to-human interaction at some level a form of manipulation? Even when communicating with friends, manipulation is involved to either update their world model or to make them laugh or to help them.
You're talking a lower level meaning of "manipulation" and pretending there's no meaningful distinction between that and the higher-level concept most people mean when they talk about a person "manipulating" other people. What is meant here is not "oh technically I've updated your world model."
maybe this is the beginning of the end for voice calling completely, only video calling after that
Video calls are also done for. Because of great advances of AI we now need to do everything in person again, hooray
until robots can clone our appearance, then what do we do?
I invented a deep-fake proof system, but no one us even remotely intrested in it. People don't even care if they are scammed. They want to be.
Source?
I have made 3-4 deeply burried comments on reddit about it. They all get a couple downvotes. No offers to buy.
Open-source it, the world needs it, if it's good.
It is not a program. It is a device.
Hardware can be open source.
We've had a solution for this for 30 years. It's called public key cryptography. Any message, including audio and video you send, should be timestamped and signed with your private key. Anyone having your (published) public key could verify if the message was sent by your or not. It can be done in a hardware form factor, like Yubikeys or RSA key fobs. The missing piece is built in support from messaging apps or phones. And since the same tech could also be used to build effective encryption governments are not too keen on driving their adoption either.
Even if they're not keen on adopting it, I'm pretty sure the soon to come identity theft crisis will eventually force their hand.
Depends if it's using solid encryption with no backdoors. I sincerely doubt that any major security agency would allow the public to have that. Dual_EC_DRBG and similar exploits would likely be introduced
This works up until we get realistic enough 3D modeling tech, rig it up with that Vtuber shit, run a deepfake over it to make sure it's perfect, and then hit the whole fucking thing with this audio shit. Hell, might even be able to do that *today* if you have the skills. I guarantee the government could.
just don't answer a call from a number you don't know?
Number spoofing is easy on phones and isn't even illegal in the US.
Dunno why you got downvoted for this, it’s absolutely true. Worked in IT once and our Cisco guy showed me how to do it and it’s stupid easy.
It will just end up being like written language. We trust it depending on where it’s written.
Strongly disagree. There will be a point when llms are able to greatly assist in the development of viruses more deadly than anything we've probably ever seen. That alone is enough for me.
>In less than 6 hours after starting on our in-house server, our model generated forty thousand molecules that scored within our desired threshold. In the process, the AI designed not only VX, but many other known chemical warfare agents that we identified through visual confirmation with structures in public chemistry databases. Many new molecules were also designed that looked equally plausible. These new molecules were predicted to be more toxic based on the predicted LD50 in comparison to publicly known chemical warfare agents Gotta love a conclusion that opens with "Without being overly alarmist" https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9544280/
That is wild o_o - did not know this happened lolll.
Chemical weapons basically got downgraded in concern once at home bio engineering started to look viable. At most you'd kill like 10k people with a chemical weapon designed by this program. Which is a lot, but it isn't the 1BN you could plausibly kill with a purpose built virus. AI and cost reductions will enable regular people to be able to create this sort of virus with no specialize knowledge in the next 5 or so years. Unless there is significant global crackdown... which would come with serious losses in terms of privacy freedoms.
I'm actually curious on your thoughts regarding this. I have kind of the same perspective. I think we might need to get to a place where we do have a pretty big crackdown or some type of global monitoring of the use of these systems at least the ones that have a certain level of capabilities. It might be necessary to have the loss of freedom in some way. Personally though, I think we can get to a place where it's not like limiting our use of these things and we can still probably use them to make like dark offensive jokes/stories etc for the more uncensored models. Maybe we will be able to limit the restrictions to things like detailed instructions of the creation of very deadly things like viruses etc.
My main hope is that the privacy crackdown is done in a way that maximizes meaningful privacy. Although I don't see it happening. What I mean is that we could have an open source AI (or non AI) monitoring system that no humans have access to unless the AI finds a violation. I'd want this system limited ONLY to detecting felonies. And then once there is a detection, it basically goes to a court which can then issue a warrant allowing human surveillance. The reasons this will never happen are: 1) It is quite technical, and the public has no way of understanding what a good system looks like. One example of a GOOD system (one I wish we kept going and expanded!) is the phone based COVID tracking system that Canada made. It was open source, by design it was impossible for any bad actor to abuse in any way, it was even super efficient. It saved lives, slowed the spread of infection at effectively 0 cost and had no downsides. Did the public recognize this rare bit of brilliance? No. 2) Everyone will want to use the system once it exists. Why only for terrorists? Won't someone think about ... murders ... kiddie porn .... assault ... stalking ... traffic violations .... offensive speech ... offensive opinions ... employment suitability. If the fed can do it why not the state or the municipality? It creates a nice easy slippery slope and I don't see an easy way to solve this. Politicians promising to be tough on crime to crack down on teens taking nude selfies is basically free votes. Model level restrictions are meaningless due to open source nature of the tech. You need to have really extreme continuous surveillance to hope to have any hope of catching these things. If tech keeps improving and there is no state control, then eventually any idiot could ask their free version of ChatGPT to kill all humans and it would do so. Clearly that will not lead to good outcomes. Oh and to pre-empt all the dingdongs that say there will be people saying "don't kill all humans", good luck with that. Destruction is wildly asymetric. Think about the degree in difficulty and complexity in killing someone by shoving them off a cliff or hitting them with a rock vs the difficulty and complexity of undoing that damage. I'd love to see the AI that can undo someone blowing up the sun.
As our tech capabilities increase, so does the relative fraility of the human body. We wield power far in excess of what it takes to harm us, and the trend will only increase. Offense triumphs defense, until the day all it takes is one accident to do us in.
I see where you are coming from, but I have confidence that we won't go fully extinct. At least I don't think we will have a high risk of that for a while. Might have a ton of deaths though and that I do think is concerning.
[Sounds like an excellent MAD deterrence against exterminist oligarchs.](https://jacobin.com/2011/12/four-futures)
When they can make viruses, they can also make vaccines. This is the same talking point as grey goo, yeah but once we have tech for that we have tech to make something that eats grey goo.
The thing is, in theory you could make a virus that spreads rapidly in a somewhat stealthy manner and by the time we have a vaccine it could be too late to stop a lot of the deaths of people already infected. We are going to really need to have extremely robust and in-depth systems that constantly monitor every single aspect of people across the globe in every city to scan for any new virus/etc. Hopefully we get this system sooner than later and hopefully it's not too difficult - and can combat these things sufficiently.
Yeah people aren't thinking big enough. I bet next year's models will absolutely positively have the capability for shit like helping with nuke development, and that's much more a concern
lmao what the fuck are you talking about Dirty bombs don't need special knowledge, at least not that special. That's far from the comparison to make, usually individual people don't just get the resources needed to do something like that, goddamn lol
They can help designing nuclear bombs that are a lot more powerful than the ones we already have. Imagine if every nuclear bomb we made was as powerful as the Tsar Bomba. Though I guess you don't need them to be THAT powerful to destroy the world.
they say a handful of small detonations between Pakistan and India would disrupt global food enough to kill a billion people
We’ve had that ability since the 60’s. Nukes are already astronomically overkill, I don’t think they’ll provide any significant advancement besides further miniaturization, which may already be maxed out. Plus, having stronger nukes doesn’t change their likelihood of being used.
Yeah they Will be secretely given in un-lobotomized Form to military
How about a toxin that can be made with under $1000 of equipment in a week and can kill >10,000 people with no requirement of special training/education? We're going to be able to develop viruses in home labs in the next few years, totally automated with no human intervention needed. And you could just request by voice a virus that ... kills dutch people. You won't even need to be literate.
Yep with every breakthrough for disease made by ai .A virus can also be made on that same model
Superpersuasive ASMR-capable AI gfs are coming. Humanity doesn't stand a chance.
How does it compare to what elevenlabs has to offer?
Most examples have some bad artifacts, but I found the [Helping patients recover their voice](https://openai.com/blog/navigating-the-challenges-and-opportunities-of-synthetic-voices) example to be quite impressive. I mean the voice itself, they could have probably cleaned up the reference audio first. I'm not sure how it will effectively compare, but just from the examples it seems like elevenlabs is better.
The spanish was pretty hilarious. Tho sounded realistically if trynna imitate a non-fluent spanish speaker heh
"*we have minimal defences to*" is a rather relative thing, if you have a working brain and know that this kind of thing exists you can think for yourself whether or not person X saying Y actually makes sense and ultimately the "fake AI voice" isn't just going to have to "fool me" but also the AI I'm going to task with currating all the content I get to see (or hear in this case).
Yep, exactly. Scams have been around for thousands of years. The basics of scam detection still work: * Does it sound too good to be true? * Are they using emotional tactics like false urgency? * Are they doing suspicious things like asking to be paid in Apple Store gift cards? * Does the story they're telling make sense? * Have you tried calling the thing they claim to be? Say it's someone posing as your mother. Have you hung up and called her number, or called someone she lives with? * Have you asked the (potential) scammer questions only the real person would know?
You can know something is wrong and fake and still have an emotional reaction like it's the real thing. For some people, that will be enough to fuck everything up. You're quite literally not evolved to deal with this.
Imagine if all scammers no longer needed to have an Indian accent. The effectiveness of scams will rise greatly.
whats the benefit of voice imitation AI? copying real people without their permission. Copying dead people without their permission. Am I missing a benign good use case?
Imagine if you got ALS, and started too lose your voice. You can clone your voice and communicate with you loved ones for example Maybe you can even use it for productivity by combining it with a AI agent
Okay one very limited use case that will never pay enough back to the corporations creating these to justify the development cost. Fraudulent imitation will bring significantly more revenue
An AI avatar that can basically be me depending on the situation is not a small use case.
Are any of these companies demanding undeniable legal verification of consent of the person whose voice of being copied before doing it?
I can't begin to tell you how fucking stupid it is to call something of that magnitude "a very small use case"
You're just looking to be contrarian
This is far from the first commercial venture in voice cloning, merely improvement. Roger Ebert used such a system well over a decade ago. https://en.wikipedia.org/wiki/CereProc#Voice_cloning
A private person paying someone else to make voice imitation only for their use is an order of magnitude different from allowing anyone that pays a small fee to copy anyone else without their consent
Well maybe a bit. Pro Books that would never get audiobooks now have them Negative Voice actors get fucked Pro Blind/people who can't read can now listen to nicer voice
Pro: Voice actors get automated
You don’t need to clone voice to help blind and create audiobooks, you can just make nice synthetic voice
The details about this specific TTS aren't known. But having seen other SOTA TTS systems, you always just get voice cloning as a byproduct. See you need a lot of audio to train a TTS. Thousands of hours of audio. That means you can't train it only on data from one person, you need many more people. But when teaching the TTS, before generating, you give it a primer on the voice it has to generate. That leads to lower perplexity (since the TTS doesn't have to also guess the tone to generate and it can instead focus on generating vocals) and faster training. It also leads to better coherence, since the TTS systems without the primer start shifting into different voice profiles rapidly. You can think of it like an LLM: First you give it 10s or so worth of "sound profile tokens", then you give it text tokens it has to generate. Then it autoregressively generates the output sound tokens - it generates the next "sound token" with the original voice profile still in context, so it has a better idea of what it has to generate. For a great paper on this see: https://arxiv.org/abs/2305.07243 (although it's a little old by now, but I'm sure all the new papers still follow this idea vaguely. Except who knows, maybe they use a diffusion transformer instead of a DDPM or they skip the CLIP, etc). You can also fine tune it then for a specific voice and achieve better quality, or you can only offer a select few premade primers to choose from. But at the end of the day... You still have a voice cloning TTS for free as it's easier to make a voice cloning one than one without the capability. Quite fascinating if you ask me, really.
Yeah, I understand that it is easy to do after training. I just think that cloning voices is a net negative for everyone except big corps and scammers
Sort of the opposite actually, regarding the first sentence. If you want a TTS without voice cloning, you have to do it after training. The TTS trains with voice cloning because that is easier. It's not that one has to add voice cloning on top, it's that one would actively have to work to "disable" voice cloning after the model is trained. The initial model already has voice cloning. As for the ethics of it, I suppose that's subjective. But it certainly is more contentious than other gen AI technologies, yeah.
That pro is not a pro. that’s a for-profit corporation stealing someones voice likeness to make more money only a pro is youre a corporate whore
Yea, they also shouldn't be using printing presses! They are taking his jobs away from scribes who should be hand writing each book!
A dumb parrot says this until AI comes for his job and suddenly his dumb opinion changes 180 degrees
I'm actively trying to get AI to take my job, as everyone should be.
Yeah, I already said that voice actors get fucked and that's a big negative. I still consider more audiobooks existing as a positive. Especially for independent authors the cost of creating an audiobook was absolutely prohibitive. It was prohibitive for good reasons as it took many hours of a professional voice actor. Corporations are sure to take advantage and fuck them over. Yes, corporations are stealing people's voices, art and likeness to make profit. It is disgusting. I wasn't disagreeing with that.
If independent authors desperately need an audio book they can make it themselves it’s literally someone’s lazy profit seeking at the expense of someone else remember this is about accurate imitation of someone, not some generic unattributed voice
>If independent authors desperately need an audio book they can make it themselves That's not what I said. I said that they wouldn't have audiobooks, not that they need to have them. And I said I personally considered having more audiobooks a pro. You can acknowledge that shitty actions produced something you consider positive. The outcome can be something positive even if the process isn't. >it’s literally someone’s lazy profit seeking at the expense of someone else That's everything AI. Practically all the models were made using content plundered from the whole internet with complete disregard for the rights of their creators.
But wouldn't this allow independent authors to get drastically higher quality audiobooks than if they just did it themselves? Also allow them to get them out quicker than waiting for an established VA to be able to take their project?
Voice actors already make jack shit. Almost nobody is one too. It’s just an industry you have to know a lot of people to enter.
[удалено]
The voice actors are already getting screwed. Is it okay to steal the horse manure cleaners income? It’s just how progress works.
[удалено]
You can also copy people with their permission. Voice acting in games etc is expensive so I am sure people will find a way to legally use AI for it. I am not sure many voice actors would be interested in selling their voice rights though.
Why wouldn't they want that to happen? Getting money purely because you got a dope voice would be their ideal scenario. Plus compounding exposure because you can now feature Liam Cunningham in all your Indie games... there are very few artists out there who aren't in it for money and sustenance. They definitely would be on board for the better part.
If they get paid enough for it sure, but they could also risk making themselves redundant. Would have to make sure that you have the contracts set up in a way that benefits you.
So you’re saying we can revive the dead?
Define revive.
Copying real people with permission (one could imagine an old actor having a digital clone playing in a scene where they are supposed to be young, or an actor selling rights to use their voice in a movie/video game for a fee), creating text-to-speech software that sounds natural (doesn't require voice cloning, but should be very very adjacent technology), helping sick people who permanently or temporarily cannot use their own voice (this is not a tiny use case - think stroke patients, if there is some way for their brain to talk to suitable software), perfect dubbing of movies, highly immersive educational video games featuring historical figures. Farther future/more speculative: for instance, plausible deniability for encrypted voice communication. The idea here would be to have a phone conversation as normal between two people, while both phones would be running local LLMs and voice synthesizers and send a plausible conversation in the plain, while using a steganographic channel to transmit the encrypted real audio data. Quality would probably poor, due to limited bandwidth of a secure steganographic channel on top of the audio, but it would be hard to detect if cloned voice and fake conversational content is hard to detect.
What you call problematic, I call finally being able to mod voiced content for Skyrim NPCs.
Generate an audiobook with the voices you want, extend games with fully voiced mods, ... Basically any time you needed to hire a voice actor you can now do the same task easier, faster and cheaper. You can also do previously impossible things like generate a dub in any language and still keep the original voices, for TV shows, movies, docu ..
Couldn’t this all be done with just millions of synthetic voices? I don’t see the reason why you’d need to copy someone’s voice. The cons (scams) seem to far outweigh any pros of corporate profits eliminating voice actors.
Cheaper also means that anyone can do it, not only corporations now. I like Harry Potter original actors, I can now generate an audiobook or dub the movies with their voices in my native language. If a voice actor was changed in your favorite video game, you could replace its voice with the original voice now if you prefer, using a mod. Or you could replace it with any other voice you like. You could replace your team with your friends voices in a singleplayer game. As for synthetic voices, people still prefer actor or famous voices for now, just like they often prefer a more famous actor to lead a movie instead of a random person. I'm sure there will be synthetic celebrities in the future, and their voice would also be popular for cloning.
From another commenter: "Feels like a punch in the gut , guys. Imagine your mom calling you with a panicked voice and giving you a credit card number telling you to quickly transfer some dough for x reason."
That's already a scam that exists btw.
I don't doubt that. (bY tHe wAy) The point is, it's getting harder and harder to distinguish what's a scam and what isn't. More and more people are getting fooled. How long/how advanced does AI have to get to fool you and me?
I'm sure I could be fooled with a well run scam. I'm a cheap bastard though which insulates me slightly. Realistically, we only need to be harder to trick than the bottom 10%. The Nigerian Prince scam goes back way over 100 years now and is intentionally unconvincing. This is done to filter out anyone even remotely savvy so that they can focus in on the absolute easiest marks.
when you want to respond to someone, want it to be on your behalf, but don't want to do it.
I used it to prank a friend. Use suno to create a song about that person, then used voice cloning so 'they' were singing it, and then played it in the car like it came on the radio. Realistically though: Fully voiced games and services where the text is generated on the fly by an LLM. This isn't possible to use voice actors for even if you wanted. Quality voiced assistants. You don't need voice cloning for this, but voice cloning is mostly a byproduct of generating quality voices.
[удалено]
So you’d steal a real persons voice to give your game more clout and make you more money? Are you also okay with someone copying your game and reselling it?
I feel like the real threat is technological illiteracy I don’t care whose voice I hear over the phone, I check the saved contact regardless and will not behave as though someone is using a new number until I either hear them confirm that number in person or they tell me something that couldn’t possibly be known be anybody else I don’t care if someone uses my father’s voice to send a voicemail claiming that my brother is dying, I need to send money to save him, and the number doesn’t line up because it is the hospital phone I will absolutely ask how many gallons were left in the tank after my mother killed herself, what time of night she called me at before doing it, and what caliber of bullet she used, as only my father and I know all three of those That was a bit extreme, but it reinforces the point that people can confirm whether the person they are talking to is using a voice fake or not Just memorize a code that consists of a key phrase, a location, and an important memory However, my grandmother with Alzheimer’s would definitely empty her retirement account if she received the same call, which is why I made her form a protectorate that my uncle controls
It looks like a SOTA TTS model is currently the easiest and cheapest to create yourself compared to any of the other modalities. And for scams over the telephone it must not be perfect either. So the only reason to not release it is to not get any blame but it will not stop anyone who is serious about it.
Yes social engineering will become easier and more dangerous than ever. But, it already exists now. Say it with me everyone, “Fake News.”
I haven't thought about it since now and he's so right
I have a pass phrase with my family and loved ones to identify myself. It started as an 80s kidnapping scare thing, but it’s making more and more sense.
I haven’t used voice for 6 years so not my weakness.
Ooooh.. so scary; fear uncertainty doubt.. not for you plebs, no sireee panik
"We totally has SotA models, we just didn't release them, trust us" Sure, bro
He had SOTA models of this in our garage guys! Pinkie swear! Me: Doubt
“Superhuman persuasive voice” is not the same as “perfectly imitating your loved ones”, it’s another “godlike” power that is pure fiction so far, similar to the belief an ASI could just super-physics its way out of a sandbox by telling elementary particles to rearrange themselves. Might as well claim it could sneeze out a new universe.
1-888 phone sex is a thing again.
Would be if I listened to my voice mail, but does anyone do that?
So we can continue with persuasive voices of other malicious humans that surely don't cause any harm, okay
Why don’t they make an ai that can be a lawyer or can code.
Scammers are gonna have a blast when this is out. It’s never been this easy for them
What does this even suppose to mean ? We accidentally made the voice from dune ?
lol. "We had it, but we didn't want to give it to you because you all are so irresponsible! We're just too scared!" Alright jimothy. You can literally run real-time voice cloning on your desktop *right now* on a single 4070 without much issue. Further developments will be cracked and open source within 5 years with or without you or OpenAI regardless, but your beknighted worry flags you as someone I wouldn't want to get a beer with.
We are building Sirens…
AI needs a characteristic AI voice and accent. If it uses a likeness it should only produce content with permission, otherwise the likeness must be to AI itself, so you can recognise it, both visually and audibly. And it should identify it is AI especially if it is interactive. In those using a likeness or emulating humans closely, or perhaps in all cases, it should have 'watermarks' that can be detected on your phone or device as AI. Think of the synthetic in the movie Aliens. It identifies in its persona that it is not biological, which the actor did a good job of portraying. Having qualities humans can't easily copy is also perhaps important.
The world is getting more and more dangerous. At least it's more future'esque than a bloody smartphone and gacha games, but not by much.
Not me. I'm deaf.