Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bark: A transformer based text to audio system

github.com

Post reply on HN
Print view View original

41–50 of 61 posts

Re: Bark: A transformer based text to audio system

#41

Post by underlines »

JonathanFly wrote: post #24

I'll link my Bark fork with long audio generation and other features on the root thread, I suppose: https://github.com/JonathanFly/bark There's going to be a big update this week with some new stuff I haven't talked about. And a bunch of amazing, clear voices, with a huge variety of styles, that blow the default Suno voices out of the water. Arguably even better than Eleven in some ways. I'm excited even though I hav…

I wish there was an easy way to fine tune bark, so we could truly clone our voice for bark inference.

Sadly the bark-voice-clone fork doesn't do it. The voices sound nothing like yourself.

Your gradio gui is great. But I don't understand where to copy the cloned npz files to. Even after refreshing the gradio GUI, the ClonedVoices don't appear in the Speaker or Generated Speaker dropdown.

Re: Bark: A transformer based text to audio system

#42

Post by JonathanFly »

underlines wrote: post #41
JonathanFly wrote: post #24

I'll link my Bark fork with long audio generation and other features on the root thread, I suppose: https://github.com/JonathanFly/bark There's going to be a big update this week with some new stuff I haven't talked about. And a bunch of amazing, clear voices, with a huge variety of styles, that blow the default Suno voices out of the water. Arguably even better than Eleven in some ways. I'm excited even though I hav…

I wish there was an easy way to fine tune bark, so we could truly clone our voice for bark inference. Sadly the bark-voice-clone fork doesn't do it. The voices sound nothing like yourself. Your gradio gui is great. But I don't understand where to copy the cloned npz files to. Even after refreshing the gradio GUI, the ClonedVoices don't appear in the Speaker or Generated Speaker dropdown.

I barely touched that, it's just from the Serp cloning github, but people kept asking so I put it in. Their clone just isn't really doing much though, it's just loading up the coarse model with the encoded wav file as a fake last generation history to the current segment, and that's basically it. But a lot of the voice is in the semantic model so just doing the coarse doesn't get you much. And even if the semantic model didn't matter as much, the coarse model has both semantic and coarse tokens as inputs and your injected coarse tokens aren't going to line up just right with what a true Bark generated pair of tokens would look like. So what you get is like a robot clone that has the most superficial similarity and lacks the depth of cadence that makes Bark awesome. That's in best case, more often get voices full of static or that don't even read the text you give them. (To be fair, any bark voice can do occasionally not read the text, it's a risk.)

I'm sure somebody will train a model that actually maps an input text to the Bark semantic representation, it shouldn't be that hard, it's just been a few weeks. But the existing clone there is just so primitive I don't know how it got so popular.

Re: Bark: A transformer based text to audio system

#43

Post by JonathanFly »

JonathanFly wrote: post #16
qup wrote: post #10

Can we run this locally? (on normal machines?) How fast are generations on consumer machines?

It's like 50% realtime on a 3090, not quite real time on a 4090. You can also use smaller models and it's a bit faster. 3080 is same speed, you don't need the extra memory.

Actually I was just checking, and Bark isn't that close to maxing out GPU utilization. Running two instances on a 3090 seems like a throughput increase and the models fit. Update: And getting weird CUDA issues. Hmn...

Re: Bark: A transformer based text to audio system

#44

Post by famouswaffles »

JonathanFly wrote: post #33
famouswaffles wrote: post #29

Earlier quoted context omitted.

https://github.com/serp-ai/bark-with-voice-clone

Not used at all, at least in my case.

I understand you didn't use that repo but did you do something other than change the audio prompt ?

Re: Bark: A transformer based text to audio system

#45

Post by JonathanFly »

famouswaffles wrote: post #44
JonathanFly wrote: post #33

Earlier quoted context omitted.

Not used at all, at least in my case.

I understand you didn't use that repo but did you do something other than change the audio prompt ?

Yeah, it's kind of hand crafted. There's more to the story and more results. I would normally just Tweet but I think it's actually so interesting that it deserves more than a tweet, at least a thoughtful writeup or a youtube video. (And I need to catch up on real work this week first, so end of week at best.)

Re: Bark: A transformer based text to audio system

#46

Post by woodson »

JonathanFly wrote: post #43
JonathanFly wrote: post #16

Earlier quoted context omitted.

It's like 50% realtime on a 3090, not quite real time on a 4090. You can also use smaller models and it's a bit faster. 3080 is same speed, you don't need the extra memory.

Actually I was just checking, and Bark isn't that close to maxing out GPU utilization. Running two instances on a 3090 seems like a throughput increase and the models fit. Update: And getting weird CUDA issues. Hmn...

Just to add a datapoint: the main audioLM based models (not the BERT embedding part) fully utilize an RTX 2080 Ti.

Re: Bark: A transformer based text to audio system

#47

Post by woodson »

MitPitt wrote: post #32
echelon wrote: post #23

Earlier quoted context omitted.

What are you talking about? It's MIT licensed.

there is no training code and devs don't plan on ever releasing it

It’s mostly there in https://github.com/lucidrains/audiolm-pytorch#hierarchical-t.... They just used FAIRs EnCodec (https://github.com/facebookresearch/encodec) instead of soundstream.

Re: Bark: A transformer based text to audio system

#48

Post by JonathanFly »

woodson wrote: post #46
JonathanFly wrote: post #43

Earlier quoted context omitted.

Actually I was just checking, and Bark isn't that close to maxing out GPU utilization. Running two instances on a 3090 seems like a throughput increase and the models fit. Update: And getting weird CUDA issues. Hmn...

Just to add a datapoint: the main audioLM based models (not the BERT embedding part) fully utilize an RTX 2080 Ti.

Must be some low hanging fruit to optimize in Bark. It would be somewhat close to realtime if it was close to 100% and scaled linearly.

Re: Bark: A transformer based text to audio system

#49

Post by dragonwriter »

JonathanFly wrote: post #42
underlines wrote: post #41

Earlier quoted context omitted.

I wish there was an easy way to fine tune bark, so we could truly clone our voice for bark inference. Sadly the bark-voice-clone fork doesn't do it. The voices sound nothing like yourself. Your gradio gui is great. But I don't understand where to copy the cloned npz files to. Even after refreshing the gradio GUI, the ClonedVoices don't appear in the Speaker or Generated Speaker dropdown.

I barely touched that, it's just from the Serp cloning github, but people kept asking so I put it in. Their clone just isn't really doing much though, it's just loading up the coarse model with the encoded wav file as a fake last generation history to the current segment, and that's basically it. But a lot of the voice is in the semantic model so just doing the coarse doesn't get you much. And even if the semantic mo…

> I’m sure somebody will train a model that actually maps an input text to the Bark semantic representation,

But, clearly, that’s already part of the Bark model, so in the abstract one should be able to leverage the existing model with appropriate code to do that, rather than developing a new model. This seems too simple but…isn’t that just “generate_text_semantic” from the existing source (with None as the history_prompt. since you don’t want it in the context of some pre-existing speaker)?

EDIT: Looking at the SERP voice clone, that’s what they are doing. The one thing that I’m intuitively skeptical about (and this is way out of the kind of programming I do normally, so I could be way off) is that the temp they use is kept at the level normally used for synthesis (0.7). I’d think you’d want the temp low, since you’d want generating a baseline for a new speaker to be more deterministic than generating content from an existing speaker.

Re: Bark: A transformer based text to audio system

#50

Post by dragonwriter »

qup wrote: post #10

Can we run this locally? (on normal machines?) How fast are generations on consumer machines?

On a laptop with a GTX 1650Ti (4GB), I’m seeing very roughly twice the audio time with “SUNO_USE_SMALL_MODELS” turned on, and about 4 times the audio time with that flag turn off.

Given what people are reporting for high-end cards, that’s much higher than I expected, which seems to underline the descriptions that its not fully utilitizing higher-end cards.

Post reply on HN