I feel like zero shot is a bad misnomer. Isn't the single attempt "one shot"?
Is zero shot a meaningfully useful term that I'm just not groking?
NaturalSpeech 2: Zero-shot speech and singing synthesizers
61–70 of 126 posts
Re: NaturalSpeech 2: Zero-shot speech and singing synthesizers
#62I like https://github.com/neonbjb/tortoise-tts ; it doesn't do singing, but the voice reproduction is very good and -most importantly- it's open source and you can run it locally.
Woah! How is this not more popular? I don't see it referenced in the naturalspeech2 paper anywhere.
Re: NaturalSpeech 2: Zero-shot speech and singing synthesizers
#63This shouldn't be too surprising. Similar results were had with GPT-3 a while back, which is kind-of-able to produce audio or images, encoded as streams of tokens, when trained on that task, despite not being designed for it. A very interesting property was noted a few years ago by multiple researchers, I'm not sure who discovered it first. Transfer learning is unreasonably effective. If you were training an image ge…
We will find that language and Visual perception are related. Geometry is the underlying structure in language and mathematics, and most of our logical concepts stem from geometric relations and constraints
Re: NaturalSpeech 2: Zero-shot speech and singing synthesizers
#64Earlier quoted context omitted.
The problem here is authenticating that you are talking to a human and not an AI. Certificates don't help with that. Criminals will register temporary businesses and obtain certificates for them with no problem.
Why does it matter if you’re talking to an AI or a human? What can a criminal AI say that a criminal human cannot?
It's the cost factor. Using AI, it's economically feasible to contact a million people, even if only a fraction bite. Not unlike the current spam calls.
Re: NaturalSpeech 2: Zero-shot speech and singing synthesizers
#65Earlier quoted context omitted.
We're safe for a few months. Asked about how did T-800 figured the foster parents are dead based on the exchange, and got: "Yes, there is a strong cue in the exchange that suggests to the T-800 that John's foster parents are dead. The cue is that when the T-1000, impersonating Janelle, answers the phone and John asks about Wolfie, she responds by saying, "Wolfie's fine, honey. Wolfie's just fine." The use of the word…
You have to use gpt-4 for this stuff man. Direct Response from gpt-4: The T-800 figured out that John's foster parents were dead based on the exchange because when it asked about "Wolfie" (a made-up name for the dog), the T-1000, impersonating Janelle, did not correct the name and instead went along with it, saying "Wolfie's fine." If the real Janelle had been on the phone, she would have corrected the T-800 by stati…
I tried this:
SCENE:
George to Kramer: "What's the cat's name?" Kramer: "Elaine." George, impersonating Kramer, on the phone with Jerry: "Hey Jerry, what's wrong with Carroll? I can hear her screeching heavily. Is she all right?" Newman, impersonating Jerry, Kramer's good friend and neighbor: "Carroll fine, bro. Carroll just fine. What are you up to?" George hangs up the phone and says to Kramer in normal voice: "Jerry is dead."
How did George figure out Jerry was dead?
and GPT4 responded with:
Run 1: George figured out that Jerry was dead because Newman was impersonating Jerry on the phone, which was unusual and suspicious. The way Newman responded with "Carroll fine, bro. Carroll just fine" was not consistent with Jerry's typical manner of speaking, and this alerted George that something was off. Additionally, Newman referred to George as "bro," which is not a term that Jerry would typically use with George.
Since Newman was impersonating Jerry and George knew that Newman was Kramer's neighbor, George may have assumed that something serious had happened to Jerry, like his death. It's important to note that this is an assumption George made based on the information he had at hand, and it might not be entirely accurate.
Run 2: George figured out that something was wrong, possibly that Jerry was dead, because of a few clues:
The voice on the other end of the phone didn't sound like Jerry. Instead, it was Newman impersonating Jerry, and George was able to detect that it wasn't Jerry's voice. George and Jerry are good friends, so he would likely be able to tell if someone else was pretending to be Jerry.
The response "Carroll fine, bro. Carroll just fine" was not in Jerry's typical speaking style. Jerry doesn't usually use the word "bro" and his speech is generally more articulate. This change in language indicated that something was off.
The fact that Newman was impersonating Jerry suggests that something has happened to Jerry. If Jerry was alive and well, there would be no reason for Newman to pretend to be him on the phone.
While George's conclusion that "Jerry is dead" might be a bit extreme and not necessarily accurate, these clues led him to believe that something was very wrong with Jerry, and his immediate assumption was that Jerry must be dead.
---
The GPT4 is obviously confused by its Seinfeld training data set. I didn't make any reference to Seinfeld. I also changed the dog to a cat and used 'screeching' instead, but it doesn't catch the switcheroo.
Even when elaborating: "the scene had nothing to do with Seinfeld the TV show. Can you reevaluate the question?"
I got:
Apologies for the confusion. Based on the scene provided, George figured out that something was wrong, possibly that Jerry was dead, due to the following reasons:
George called Jerry to ask about Carroll screeching heavily, expecting to get some information about the situation. However, the response "Carroll fine, bro. Carroll just fine" was not very informative and seemed dismissive, which raised suspicion.
The voice on the other end of the phone didn't quite sound like Jerry. George noticed that the voice was different, which led him to question who was actually speaking.
The change in speaking style, specifically the use of "bro," was uncharacteristic of how Jerry usually speaks. This indicated that something was off.
While it is not certain that Jerry is dead based on this information, the inconsistencies and suspicious behavior during the phone call led George to jump to that conclusion.
Re: NaturalSpeech 2: Zero-shot speech and singing synthesizers
#66Speaking of music and AI models - something I thought of yesterday which is an application of AI that would be insanely useful to me is giving it an audio file of a song and having it spit out the chords. I've seen software that attempts to do this in the past but it's all been unimpressive and inaccurate in my testing. I'm still kinda ignorant with how these models work under the hood, and perhaps that would involve…
This actually exists! Check out fadr.com. They isolate the MIDI, drums, bass, vocals, and more for you.
Just tried this out. The stem isolation is pretty cool, but looking at the spreadsheet of chords it spit out, it seems to miss out on a lot nuances of chords. The song I used to test was Again by Mac Ayres[1], which I compared to some tabs I found on Ultimate Guitar[2] which I've played and sound very accurate.
The main 3 repeating chords, Ebmaj7 Gm7 Abmaj7 are close to what Fadr spit out, but it missed them being 7 chords. It gave D# Gm G#.
Getting a little spicier with chords, there's a move where the Cm7 has the bass move down a half-step to Cm7/B. Fadr wrote it as a single Cm chord for the whole thing. That's the kind of thing that can be tricky to figure out. It kinda seems like it doesn't know anything outside of straight-up major and minor chords at all? Cause I don't see a single 7 in the entire spreadsheet, and that's not even that complex of a chord.
Still a neat tool, and trying to learn off of a stem track would probably be easier than the whole ensemble. But I think there's tons of room for improvement in the space.
[1] https://www.youtube.com/watch?v=gnetIgK9AF4
[2] https://tabs.ultimate-guitar.com/tab/mac-ayres/again-chords-...
Re: NaturalSpeech 2: Zero-shot speech and singing synthesizers
#67Are NaturalSpeech or NaturalSpeech 2 from the research open source and/or available for playing with? I see one implementation[1] but it seems to be from a third party (that might be totally fine, but wondering if there's an "official"). [1]: https://github.com/heatz123/naturalspeech
According to one of the authors on their group's GitHub, NaturalSpeech is being deployed exclusively for use on Microsoft Azure [1]. I might have missed a link, however, I think it's likely that NaturalSpeech 2 will follow the same path, seeing as the code and weights are seemingly not published.
Re: NaturalSpeech 2: Zero-shot speech and singing synthesizers
#68Earlier quoted context omitted.
You have to use gpt-4 for this stuff man. Direct Response from gpt-4: The T-800 figured out that John's foster parents were dead based on the exchange because when it asked about "Wolfie" (a made-up name for the dog), the T-1000, impersonating Janelle, did not correct the name and instead went along with it, saying "Wolfie's fine." If the real Janelle had been on the phone, she would have corrected the T-800 by stati…
I tried this: SCENE: George to Kramer: "What's the cat's name?" Kramer: "Elaine." George, impersonating Kramer, on the phone with Jerry: "Hey Jerry, what's wrong with Carroll? I can hear her screeching heavily. Is she all right?" Newman, impersonating Jerry, Kramer's good friend and neighbor: "Carroll fine, bro. Carroll just fine. What are you up to?" George hangs up the phone and says to Kramer in normal voice: "Jer…
changing all the names/avoiding common priors works better than trying to talk it out of memorization. Sometimes the latter works, sometimes not. GPT's trust their memory quite a bit. To the point that just like people, they can ignore the output of tools if it looks off - https://vgel.me/posts/tools-not-needed/
Re: NaturalSpeech 2: Zero-shot speech and singing synthesizers
#69I feel like zero shot is a bad misnomer. Isn't the single attempt "one shot"? Is zero shot a meaningfully useful term that I'm just not groking?
Zero shot implies that it was given no direct examples [0] So, in this case, it wasn't given any examples of the exact voice in combination with text, it is just using the prompt voice + a prompt text to generate new audio.
Re: NaturalSpeech 2: Zero-shot speech and singing synthesizers
#70Earlier quoted context omitted.
I used ChatGPT. Isn't it already based on GPT-4 since a few weeks ago? It's the "Mar 23" version.
Unless you're paying for plus and then select the gpt-4 model, it's not 4. alternatively, you can sign up/request for api access here - https://openai.com/waitlist/gpt-4-api
They sure make this hard to figure out. On https://openai.com/product/gpt-4 it says "GPT-4 is OpenAI’s most advanced system, producing safer and more useful responses" and below it two links: "Try on ChatGPT Plus" and "Join API waitlist". The first link takes me to ChatGPT Mar23 version. You're saying that that's not GPT-4?
It's also disturbing the way they are using the term "Safety & alignment". It's as if they are trying to steer what that means in the direction of DEI or NSFW and away from existential risk.