Weird A.I. Yankovic: a cursed deep dive into the world of voice cloning

#1

Post by waxpancake » Mon, Oct 02, 2023, 3:15 PM UTC

Weird A.I. Yankovic: a cursed deep dive into the world of voice cloning
waxy.org

Re: Weird A.I. Yankovic: a cursed deep dive into the world of voice cloning

#2

Post by simonw » Mon, Oct 02, 2023, 3:17 PM UTC

I did not know about this: "The center of the A.I. cover songs community is a massive 500,000+ member Discord called A.I. Hub, where members trade new tips, tools, techniques, and links to their original and cover songs."

Re: Weird A.I. Yankovic: a cursed deep dive into the world of voice cloning

#3

Post by mecredis » Mon, Oct 02, 2023, 3:21 PM UTC

It's kind of wild that these tools just transfer a copy of these models every time they're spun up (whether it's to a Google Colab notebook or a local machine.)

This must mean Hugging Face's bandwidth bill must be crazy, or am I missing something (maybe they have a peering agreement? heavily caching things?)

Re: Weird A.I. Yankovic: a cursed deep dive into the world of voice cloning

#4

Post by mckirk » Mon, Oct 02, 2023, 4:39 PM UTC

My absolute favorite application of this tech so far is The Beach Boys singing 'Hurt'. It's the first time I seriously didn't notice any artifacts, and it just works so well even though it really shouldn't.

Enjoy: https://youtu.be/gmNSFqyg_Z8

Re: Weird A.I. Yankovic: a cursed deep dive into the world of voice cloning

#5

Post by causi » Mon, Oct 02, 2023, 4:43 PM UTC

AI song covers are incredible, from Goku singing "Don't Stop Me Now" to the cast of Spongebob singing "Ocean Man".

Re: Weird A.I. Yankovic: a cursed deep dive into the world of voice cloning

#6

Post by satertek » Mon, Oct 02, 2023, 4:43 PM UTC

mecredis wrote: post #3

It's kind of wild that these tools just transfer a copy of these models every time they're spun up (whether it's to a Google Colab notebook or a local machine.) This must mean Hugging Face's bandwidth bill must be crazy, or am I missing something (maybe they have a peering agreement? heavily caching things?)

Their Python module caches the downloads, which is checked before downloading them again...but you're probably not wrong on the crazy bandwidth bill. Looks like they have crazy VC money though, considering the current climate.

Re: Weird A.I. Yankovic: a cursed deep dive into the world of voice cloning

#7

Post by civilitty » Mon, Oct 02, 2023, 4:46 PM UTC

mecredis wrote: post #3

It's kind of wild that these tools just transfer a copy of these models every time they're spun up (whether it's to a Google Colab notebook or a local machine.) This must mean Hugging Face's bandwidth bill must be crazy, or am I missing something (maybe they have a peering agreement? heavily caching things?)

Unmetered 10+ gigabit connections were on the order of $1/mbit/mo wholesale over a decade ago when I priced out a custom CDN so for the cost of 100 TB of data transfer out of AWS you could get a 24/7 sustained 10gbit/s (>3 PB per month at 100% utilization).

Bandwidth has always been crazy cheap.

Re: Weird A.I. Yankovic: a cursed deep dive into the world of voice cloning

#8

Post by dwringer » Mon, Oct 02, 2023, 4:52 PM UTC

mckirk wrote: post #4

My absolute favorite application of this tech so far is The Beach Boys singing 'Hurt'. It's the first time I seriously didn't notice any artifacts, and it just works so well even though it really shouldn't. Enjoy: https://youtu.be/gmNSFqyg_Z8

I don't know what I was expecting but that isn't Hurt, it's Surfin' USA with Hurt's lyrics that sound extremely jittery and grainy.

I'm curious though if some AI soon could in fact synthesize the Beach Boys' style with the actual chords and melody from the NIN song, possibly with some of the pathos of Johnny Cash as well.

Re: Weird A.I. Yankovic: a cursed deep dive into the world of voice cloning

#9

Post by satvikpendem » Mon, Oct 02, 2023, 4:53 PM UTC

What's the best open source text to speech? Eleven Labs and others are interesting but closed source. I want to use them mainly for audiobooks as I have a lot of ePubs and I'm just using the basic Google text to speech voices on my Android, via Moon+ Reader. It works fine but it's still more robotic than state of the art.

Re: Weird A.I. Yankovic: a cursed deep dive into the world of voice cloning

#10

Post by minimaxir » Mon, Oct 02, 2023, 4:56 PM UTC

This article only covers the musical aspects of AI voice cloning, but there's another dynamic to AI voice cloning that's more complicated: replacing general voice actors in movies/video games/anime (example: https://www.axios.com/2023/07/24/ai-voice-actors-victoria-at... )

Unlike musicians who can't be replaced without significant postprocessing, have enough money to not be impacted by competition, and have legal muscle, voice over artists:

- Can be reproduced with good-enough results from out-of-the-box voice cloning settings on ElevenLabs or an open source equivalent (Bark, VALL-E X)

- Are already underpaid for their work as-is

- Have no legal ownership of their voice since they are contractors, and their voicework is owned by their clients who may not be as incentivised in protecting the VO.

I want to write a blog post about it but I suspect most people on Hacker News won't be interested in a treatise on the cultural impacts of the voicework in Persona 5 and Genshin Impact.