Llasa: Llama-Based Speech Synthesis • Hacker News Board

Re: Llasa: Llama-Based Speech Synthesis

#21

Post by woodson » Fri, May 02, 2025, 1:50 PM UTC

oezi wrote: post #17

gapeleon wrote: post #13

Earlier quoted context omitted.

> Interesting that there isn't a mention of Orpheus as prior art either Llasa-3b ( https://huggingface.co/HKUSTAudio/Llasa-3B ) came out before Orpheus ( https://huggingface.co/canopylabs/orpheus-3b-0.1-ft ). > it's the exact same thing. They're very similar, but they're not the exact same thing. Llasa uses xcodec2, a much simpler, lossless 16khz wav codec. This makes it superior for one-shot voice cloning. Orpheus'…

Isn't xcodec2 also lossy? I thought it is also just another neural codec (50 tok/s, single codebook). What are people using to upsampling back to 44,1 or 48 khz? Anything fancy?

They’re both lossy. They use a VAE-VQ type architecture trained with a combination of losses/discriminators. The differences are mainly the encoder/decoder architecture, the type of bottleneck quantization (RVQ, FSQ, etc.) and of course the training data.

Re: Llasa: Llama-Based Speech Synthesis

#22

Post by thot_experiment » Fri, May 02, 2025, 8:12 PM UTC

oezi wrote: post #16

gapeleon wrote: post #13

Earlier quoted context omitted.

> Interesting that there isn't a mention of Orpheus as prior art either Llasa-3b ( https://huggingface.co/HKUSTAudio/Llasa-3B ) came out before Orpheus ( https://huggingface.co/canopylabs/orpheus-3b-0.1-ft ). > it's the exact same thing. They're very similar, but they're not the exact same thing. Llasa uses xcodec2, a much simpler, lossless 16khz wav codec. This makes it superior for one-shot voice cloning. Orpheus'…

Do you happen to know why Orpheus and Llasa use Finetuning for voice cloning? Zonos uses 128-float embeddings for voices and it seems so much nicer. Because you can just mix and match voices without changing the model.

No, you just condition it with text-voice token pairs and then when conditioning further inference w/ text the voice tokens tend to match the pairs further up in the context.

Re: Llasa: Llama-Based Speech Synthesis

#23

Post by dheera » Sat, May 03, 2025, 12:03 AM UTC

imtringued wrote: post #18

dheera wrote: post #7

> employs a single-layer vector quantizer (VQ) codec and a single Transformer architecture to fully align I really wish when new models were released that they would draw a diagram of all the layers and the tensor input and output sizes at each layer, with zoom in/out capabilities if needed using D3.js or whatever visualization framework if needed. Every single layer should be on there with its input and output sizes…

This already exists in Transformer Lab and ONNX (not recommended for transformers). You can also build a custom version of llama.cpp that writes out the ggml compute graph. What's irritating is that hugging face didn't add it to their GGUF file viewer.

Oh, sure, for the well-known models that are already on there.

I just wish that new research would always spell it out in full instead of these silly block diagrams labelled with just e.g. "Cross Attention" and not the exact parameters, number of heads, layer sizes, etc.

Also some of these diagrams use a + for concatenation and some use it for addition, that's another headache to figure out, having layer sizes would make it clear.