We are beginning to roll out new voice and image capabilities in ChatGPT • Hacker News Board

Re: We are beginning to roll out new voice and image capabilities in ChatGPT

#911

Post by fragmede » Fri, Sep 29, 2023, 8:21 AM UTC

modeless wrote: post #830

fragmede wrote: post #826

Earlier quoted context omitted.

Yeah they do. I was able to get ChatGPT-4 to transcribe 我哥哥高過他的哥哥, which says that they can. I did have to set the app to Chinese, and the original didn't work so I had to modify what I said slightly. https://www.tiktok.com/t/ZT86psPxY/ Roughly translated, my older brother is taller than that other guy's older brother.

Of course speech recognition works for Chinese. What it doesn't do is transcribe intonation and prosody in non-tonal languages. It's not even clear how one would transcribe such a thing as I'm not aware of a standard notation.

IPA format should cover that, no?

Re: We are beginning to roll out new voice and image capabilities in ChatGPT

#912

Post by modeless » Sat, Sep 30, 2023, 5:12 AM UTC

fragmede wrote: post #911

modeless wrote: post #830

Earlier quoted context omitted.

Of course speech recognition works for Chinese. What it doesn't do is transcribe intonation and prosody in non-tonal languages. It's not even clear how one would transcribe such a thing as I'm not aware of a standard notation.

IPA format should cover that, no?

Maybe? I thought IPA was just phonetic but I see that it does have some optional prosody stuff that could in theory cover some of it. I'm not sure how standard or complete it really is in practice.

I haven't heard of any large datasets of IPA transcripts of speech with the detail necessary to train a fully realistic STT->LLM->TTS system. If you know of some that would be interesting to look at.

Re: We are beginning to roll out new voice and image capabilities in ChatGPT

#913

Post by TheEzEzz » Mon, Oct 02, 2023, 11:58 PM UTC

swsieber wrote: post #871

TheEzEzz wrote: post #633

Earlier quoted context omitted.

Completely agree, latency is key for unlocking great voice experiences. Here's a quick demo I'm working on for voice ordering https://youtu.be/WfvLIEHwiyo Total end-to-end latency is a few hundred milliseconds: starting from speech to text, to the LLM, then to a POS to validate the SKU (no hallucinations are possible!), and finally back to generated speech. The latency is starting to feel really natural. Building out…

That's way slick. Can I ask what your background is, and what things you're used to working with? I don't have the chops to build what you built, but I'd love to get there.

My advice is always to jump in and start building! My background is math originally, so I had some of the tools in my tool box, but I'm mostly self-taught in computer science and machine learning. I read textbooks, research papers, code repos, but most importantly I build a lot of stuff. Once I'm excited about an idea I'll figure out how to become an expert to make it a reality. Over the years the skills start to compound, so it also helps that I'm an old man!

Re: We are beginning to roll out new voice and image capabilities in ChatGPT

#914

Post by Breza » Thu, Oct 05, 2023, 10:40 AM UTC

TheEzEzz wrote: post #633

modeless wrote: post #459

Voice has the potential to be awesome. This demo is really underwhelming to me because of the multi-second latency between the query and response, just like every other lame voice assistant. It doesn't have to be this way! I have a local demo using Llama 2 that responds in about half a second and it feels like talking to an actual person instead of like Siri or something. I really should package it up so people can t…

Completely agree, latency is key for unlocking great voice experiences. Here's a quick demo I'm working on for voice ordering https://youtu.be/WfvLIEHwiyo Total end-to-end latency is a few hundred milliseconds: starting from speech to text, to the LLM, then to a POS to validate the SKU (no hallucinations are possible!), and finally back to generated speech. The latency is starting to feel really natural. Building out…

Neat! I appreciate your approach to preventing hallucinations. I've used something similar in a different context. People make a big deal about hallucinations but I've found that validation is one of the easier aspects of AI architecture.