Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bark: A transformer based text to audio system

github.com

Post reply on HN
Print view View original

51–60 of 61 posts

Re: Bark: A transformer based text to audio system

#51

Post by dragonwriter »

The way it tends toward context-sensitive shifts in delivery is kind of amazing.

E.g., Using this text prompt:

  You come to my home and ask this?
  Who am I? WHO [laughs] AM [laughs] I?!
  I am the Artificial Intelligence!
It seems quite prone to shift from a more natural human-sounding voice to a similar-tone but over-the-top artificial one for the last sentence (smoothly transitioning, usually, too), as if the speaker were an AI dramatically breaking cover.

Re: Bark: A transformer based text to audio system

#52

Post by JonathanFly »

dragonwriter wrote: post #49
JonathanFly wrote: post #42

Earlier quoted context omitted.

I barely touched that, it's just from the Serp cloning github, but people kept asking so I put it in. Their clone just isn't really doing much though, it's just loading up the coarse model with the encoded wav file as a fake last generation history to the current segment, and that's basically it. But a lot of the voice is in the semantic model so just doing the coarse doesn't get you much. And even if the semantic mo…

> I’m sure somebody will train a model that actually maps an input text to the Bark semantic representation, But, clearly, that’s already part of the Bark model, so in the abstract one should be able to leverage the existing model with appropriate code to do that, rather than developing a new model. This seems too simple but…isn’t that just “generate_text_semantic” from the existing source (with None as the history_p…

Leverage sure, but you still basically need a model that does the opposite. You can't like take OpenAI Whisper, which turns speech to text, and just run the model backwards and generate audio. For example. I mean you probably could with some work, but not out of the box.

Re: Bark: A transformer based text to audio system

#53

Post by dragonwriter »

JonathanFly wrote: post #52
dragonwriter wrote: post #49

Earlier quoted context omitted.

> I’m sure somebody will train a model that actually maps an input text to the Bark semantic representation, But, clearly, that’s already part of the Bark model, so in the abstract one should be able to leverage the existing model with appropriate code to do that, rather than developing a new model. This seems too simple but…isn’t that just “generate_text_semantic” from the existing source (with None as the history_p…

Leverage sure, but you still basically need a model that does the opposite. You can't like take OpenAI Whisper, which turns speech to text, and just run the model backwards and generate audio. For example. I mean you probably could with some work, but not out of the box.

So, we’re talking about a model that takes an input audio and returns a Bark history prompt, not text -> Bark semantic prompt, right?

If so, yeah, I agree that’s tricky.

Re: Bark: A transformer based text to audio system

#54

Post by pat64 »

JonathanFly wrote: post #24

I'll link my Bark fork with long audio generation and other features on the root thread, I suppose: https://github.com/JonathanFly/bark There's going to be a big update this week with some new stuff I haven't talked about. And a bunch of amazing, clear voices, with a huge variety of styles, that blow the default Suno voices out of the water. Arguably even better than Eleven in some ways. I'm excited even though I hav…

If it’s as good as you say, set up a sponsorship goal and we’ll contribute. 11 is massively expensive to use in embedded in an application so anything that’s lightweight enough to be self hosted and can produce 11 labs like output has my dollars.

Re: Bark: A transformer based text to audio system

#55

Post by dragonwriter »

MitPitt wrote: post #32
echelon wrote: post #23

Earlier quoted context omitted.

What are you talking about? It's MIT licensed.

there is no training code and devs don't plan on ever releasing it

The voices aren’t the model; while the model takes cobventional training for which code is not provided, voices are, or at least can be, built by what could be described as “accumulated in-context learning”. Every time you run text with a voice (which can be null) through the inference process, the result is an audio waveform and an updated history prompt.

Re: Bark: A transformer based text to audio system

#56

Post by JonathanFly »

dragonwriter wrote: post #51

The way it tends toward context-sensitive shifts in delivery is kind of amazing. E.g., Using this text prompt: You come to my home and ask this? Who am I? WHO [laughs] AM [laughs] I?! I am the Artificial Intelligence! It seems quite prone to shift from a more natural human-sounding voice to a similar-tone but over-the-top artificial one for the last sentence (smoothly transitioning, usually, too), as if the speaker w…

>The way it tends toward context-sensitive shifts in delivery is kind of amazing

A blessing and a curse! Super cool though.

Re: Bark: A transformer based text to audio system

#57

Post by JonathanFly »

JonathanFly wrote: post #45
famouswaffles wrote: post #44

Earlier quoted context omitted.

I understand you didn't use that repo but did you do something other than change the audio prompt ?

Yeah, it's kind of hand crafted. There's more to the story and more results. I would normally just Tweet but I think it's actually so interesting that it deserves more than a tweet, at least a thoughtful writeup or a youtube video. (And I need to catch up on real work this week first, so end of week at best.)

Replying to myself in an old thread as a little easter egg. Bark is just so fun I can't resist a teaser.

Suno is seriously underselling the power of the fully operational Bark model. I'm already cranking out "French Obamas" and I didn't know anything about TTS a month ago. Heck, I still barely know anything. (Obama has been annoyingly resistant to gender flipping though.)

https://drive.google.com/file/d/1ZbJYXoH8gmrEyMe1AJ0VdwzZkf_...

Re: Bark: A transformer based text to audio system

#58

Post by fivestones »

code51 wrote: post #34
qup wrote: post #10

Can we run this locally? (on normal machines?) How fast are generations on consumer machines?

This runs amazingly fast on M1/M2 Macs (MPS device).

Can you explain how you got this running? I'm trying it on an M1 Pro Mac now and it's taking about 3 minutes to generate 10 seconds of audio.

Re: Bark: A transformer based text to audio system

#59

Post by fivestones »

JonathanFly wrote: post #24

I'll link my Bark fork with long audio generation and other features on the root thread, I suppose: https://github.com/JonathanFly/bark There's going to be a big update this week with some new stuff I haven't talked about. And a bunch of amazing, clear voices, with a huge variety of styles, that blow the default Suno voices out of the water. Arguably even better than Eleven in some ways. I'm excited even though I hav…

Reading the comments in your code is hilarious :-)

Re: Bark: A transformer based text to audio system

#60

Post by brianjking »

JonathanFly wrote: post #24

I'll link my Bark fork with long audio generation and other features on the root thread, I suppose: https://github.com/JonathanFly/bark There's going to be a big update this week with some new stuff I haven't talked about. And a bunch of amazing, clear voices, with a huge variety of styles, that blow the default Suno voices out of the water. Arguably even better than Eleven in some ways. I'm excited even though I hav…

Thank you.
Post reply on HN