Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

200ms Voice LLM

github.com

Post reply on HN
Print view View original

11–20 of 24 posts

Re: 200ms Voice LLM

#11

Post by mosselman »

ofrzeta wrote: post #5
Gys wrote: post #4

Cannot wait for an instant translator. Something that can translate synchronously (!) one language it hears into a language that I understand. Getting closer!

I don't understand how that could possibly work. You will always need some time window, don't you? Also sometimes the translation of the first word can only be inferred when you process the last word in a sentence (simplified example).

There are people who do this and that is synchronous enough. So indeed maybe no Star Trek level mouth-moves-and-translation-comes-out, but with 200ms we are getting close to being able to speak perfectly fine with someone who’s language we don’t speak.

Re: 200ms Voice LLM

#12

Post by akreal »

Cool!

HF Transformers is great for prototyping and research, but should not an interactive tool like this be based on something more speed-focused, like llama.cpp?

Any plans for languages beyond English?

Re: 200ms Voice LLM

#13

Post by ofrzeta »

mosselman wrote: post #11
ofrzeta wrote: post #5

Earlier quoted context omitted.

I don't understand how that could possibly work. You will always need some time window, don't you? Also sometimes the translation of the first word can only be inferred when you process the last word in a sentence (simplified example).

There are people who do this and that is synchronous enough. So indeed maybe no Star Trek level mouth-moves-and-translation-comes-out, but with 200ms we are getting close to being able to speak perfectly fine with someone who’s language we don’t speak.

I've experienced human live translators and more often than not they paraphrase a sentence as a whole after it is uttered. I couldn't imagine otherwise because you can't start with translating until the sentence is finished by the original speaker. With human translators there's the additional challenge to translate/speak while listing.

Re: 200ms Voice LLM

#14

Post by ein0p »

Gys wrote: post #7
ofrzeta wrote: post #5

Earlier quoted context omitted.

I don't understand how that could possibly work. You will always need some time window, don't you? Also sometimes the translation of the first word can only be inferred when you process the last word in a sentence (simplified example).

You mean the job of a translator is by definition impossible? Edit: Yes, for sure there will be a delay of a few words or even one or two short sentences. Just like human translators. Not a problem I think. Edit 2: Very curious why my first comment was downvoted?

You can’t translate word by word. You need entire phrases, sometimes pretty long ones, to understand the meaning.

Re: 200ms Voice LLM

#15

Post by mosselman »

ofrzeta wrote: post #13
mosselman wrote: post #11

Earlier quoted context omitted.

There are people who do this and that is synchronous enough. So indeed maybe no Star Trek level mouth-moves-and-translation-comes-out, but with 200ms we are getting close to being able to speak perfectly fine with someone who’s language we don’t speak.

I've experienced human live translators and more often than not they paraphrase a sentence as a whole after it is uttered. I couldn't imagine otherwise because you can't start with translating until the sentence is finished by the original speaker. With human translators there's the additional challenge to translate/speak while listing.

Yes exactly and the EU has meetings this way, so it seems good enough already. I don't think in practice, let's say on holiday, you'd need anything faster than 200ms for the first token. Of course, more speed would probably mean better experience, but it is already well within the realm of fast enough. Accuracy matters a lot more.

Re: 200ms Voice LLM

#16

Post by woleium »

ein0p wrote: post #14
Gys wrote: post #7

Earlier quoted context omitted.

You mean the job of a translator is by definition impossible? Edit: Yes, for sure there will be a delay of a few words or even one or two short sentences. Just like human translators. Not a problem I think. Edit 2: Very curious why my first comment was downvoted?

You can’t translate word by word. You need entire phrases, sometimes pretty long ones, to understand the meaning.

sometimes, but in a constrained situation, e.g. at a hotel reception desk, maybe not.

Re: 200ms Voice LLM

#17

Post by nmstoker »

woleium wrote: post #16
ein0p wrote: post #14

Earlier quoted context omitted.

You can’t translate word by word. You need entire phrases, sometimes pretty long ones, to understand the meaning.

sometimes, but in a constrained situation, e.g. at a hotel reception desk, maybe not.

So what would you do if translating from a language that puts the main verb at the end of a sentence, into a language that doesn't...

Feels like there will be plenty of cases you can't just get around.

Re: 200ms Voice LLM

#18

Post by juberti »

akreal wrote: post #12

Cool! HF Transformers is great for prototyping and research, but should not an interactive tool like this be based on something more speed-focused, like llama.cpp? Any plans for languages beyond English?

We're running it on vLLM and are working with others in the community to bring it to other optimized inference frameworks.

Re: 200ms Voice LLM

#19

Post by ricardobeat »

unraveller wrote: post #2

I hope this catches on and can be transplanted into any LLM soon. Instant voice integration is poised to unlock many UX things like auto-pilot in cars making you a real powerful co-pilot where you can suggest fine-tune settings on the fly for the current situation or anything that changes lots of settings at once with visual feedback.

Nio’s built-in assistant uses ChatGPT with Whisper (I think) and is almost real time already. It can change settings like that and rarely misses a beat.

I expect once GPT 4o becomes available it will be awesome (even more if they “unlock” it to be asked generic questions and hold a conversation).

Re: 200ms Voice LLM

#20

Post by demarq »

That was amazing and so productive. Looking at the transcript I was able to cover so much more ground than if I was stuck typing all my questions with a keyboard. I can imagine a future in offices where people have Ai rooms where you go not for a meeting with other people but to have a convo with ai.
Post reply on HN