Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We are beginning to roll out new voice and image capabilities in ChatGPT

openai.com

Post reply on HN
Print view View original

881–890 of 914 posts

Re: We are beginning to roll out new voice and image capabilities in ChatGPT

#881

Post by pmarreck »

modeless wrote: post #839
pmarreck wrote: post #805

Earlier quoted context omitted.

Do you have a rough design outline of what you built? I feel like we're on the cusp of something like this and it sounds amazing.

I'm using Llama2-chat-13B via mlc-llm @ 4bit quantization + whisper-streaming + coqui TTS, all running simultaneously on one 4090 in real time. It didn't take long to prototype. Polishing and shipping it to non-expert users would take much longer than I've spent on it so far. I'd have to test for and solve a ton of installation problems, find better workarounds for whisper-streaming's hallucination issues, improve th…

> It didn't take long to prototype. Polishing and shipping it to non-expert users would take much longer than I've spent on it so far. I'd have to test for and solve a ton of installation problems

I've found some success at this by using Nix... but Nix is a whole 'nother ball of yarn to learn. It WILL get you to declarative/deterministic installs of any piece of the toolchain it covers, though, and it does a hell of a lot better job managing dependencies than anything in Python's ecosystem ever will (in fact, I am pretty sure that Python's being terrible at this is actually driving Nix adoption)

As an example of the power Nix might enable, check out https://nixified.ai/ (which is a project that hasn't been updated in some months and I hope is not dead... It does have some forks on Github, though). Assuming you already have Nix installed, you can get an entire ML toolchain up including a web frontend with a single command. I have dozens of projects on my work laptop, all with their own flake.nix files, all using their own versions of dependencies (which automatically get put on the PATH thanks to direnv), nothing collides with anything else, everything is independently updateable. I'm actually the director of engineering at a small startup and having our team's dev environments all controlled via Nix has been a godsend already (as in, a massive timesaver).

I do think that you could walk a live demo of this into, say, McDonald's corporate, and walk out with a very large check and a contract to hire a team to look into building it out into a product, though. (If you're going to look at chains, I'd suggest Wawa first though, as they seem to embrace new ordering tech earlier than other chains.)

Re: We are beginning to roll out new voice and image capabilities in ChatGPT

#882

Post by tasuki »

marktl wrote: post #807
plutoh28 wrote: post #134

This is the dagger that will make online schooling unviable. ChatGPT already made it so that you could easily copy & paste any full-text questions and receive an answer with 90% accuracy. The only flaw was that problems that also used diagrams or figures would be out of the domain of ChatGPT. With image support, students could just take screenshots or document scans and have ChatGPT give them a valid answer. From wha…

I've taken certification exams where an app is run on my machine verifying I have nothing else open and my camera had to be enabled, with me and my hands in view for the entirety of the test. There are ways to ensure cheating is more difficult than it's worth, however I see this tech as greatly changing what we want to learn and how we might learn it. It is transformative and not slowing down.

> I've taken certification exams where an app is run on my machine verifying I have nothing else open and my camera had to be enabled, with me and my hands in view for the entirety of the test. There are ways to ensure cheating is more difficult than it's worth

Yes sure it makes cheating inconvenient. It also makes exam taking inconvenient.

If I can at all help it, I will not be a subject to this sort of abuse and neither will my kid.

Re: We are beginning to roll out new voice and image capabilities in ChatGPT

#883

Post by whimsicalism »

DrScientist wrote: post #874
whimsicalism wrote: post #597

Earlier quoted context omitted.

Nonsense. The medical possibilities that will be unlocked by large generative deep multimodal models are on an entirely different scale from "statistical diagnoses." Imagine feeding in an MRI image, asking if this person has cancer, and then asking the model to point out why it thinks the person has cancer. That will be possible within a few years at most . The regulatory challenges will be surmounted eventually once…

But in your scenario - which part is adding the value? Your deep multimodal models or the MRI imaging? What you are essentially saying is the signal is so subtle that only a large NN can reliably extract it. While that may well be the case, it would be better to have a scan/diagnostic that doesn't need that level of signal processing to interpret. For example - you don't need a large generative deep multimodal model…

There are tons & tons of conditions that do not have easy scans/diagnostic and rely on subtle signals - especially if they are not a binary yes/no but a regression style prediction.

We've picked a lot of the low-hanging simple to extract signals, we need large models to go to the next phase for things like parkinsons, etc.

Re: We are beginning to roll out new voice and image capabilities in ChatGPT

#884

Post by TheEzEzz »

7_hours_ago wrote: post #857
TheEzEzz wrote: post #753

Earlier quoted context omitted.

Lots of work around speculative decoding, optimizing across the ASR->LLM->TTS interfaces, fine-tuning smaller models while maintaining accuracy (lots of investment here), good old fashioned engineering around managing requests to the GPU, etc. We're considering commercializing this so I can't open source just yet, but if we end up not selling it I'll definitely think about opening it up.

Can you at least share the stack that you're using in building this? What kind of business model are you considering in commercializing it?

We're design the stack to be fairly flexible. It's Python/Pytorch under the hood, with the ability to plug and play various off the shelf models. For ASR we support GCP/AssemblyAI/etc, as well as a customized self-hosted version of Whisper that is tailored for stream processing. For the LLM we support fine-tuned GPT3 models, fine-tuned Google text-bison models, or locally hosted fine-tuned Llama models (and a lot of the project goes into how to do the fine-tuning to ensure accuracy and low latency). For the TTS we support Elevenlabs/GCP/etc, and they all tie into the latency reducing approaches.

Re: We are beginning to roll out new voice and image capabilities in ChatGPT

#885

Post by barfingclouds »

modeless wrote: post #459

Voice has the potential to be awesome. This demo is really underwhelming to me because of the multi-second latency between the query and response, just like every other lame voice assistant. It doesn't have to be this way! I have a local demo using Llama 2 that responds in about half a second and it feels like talking to an actual person instead of like Siri or something. I really should package it up so people can t…

There needs to be an optional button that you hold while speaking and let go when you are done. If button is not held it should auto detect

Re: We are beginning to roll out new voice and image capabilities in ChatGPT

#886

Post by rubslopes »

bytefactory wrote: post #733
rubslopes wrote: post #84

Earlier quoted context omitted.

I've replaced my voice google assistant searches with the voice feature of the Bing app. It's a night and day difference. Bing voice is what I always expected from an AI companion of the future, it is just lacking commands -- setting tasks, home automation, etc.

Did you find a way to do this seamlessly including being able to say something like "Hey Bing", or do you just have a shortcut or widget for this?

No. At least on Android there is no system shortcut that takes you directly to the voice feature yet. For now, I'm using the widget.

Re: We are beginning to roll out new voice and image capabilities in ChatGPT

#887

Post by modeless »

pmarreck wrote: post #881
modeless wrote: post #839

Earlier quoted context omitted.

I'm using Llama2-chat-13B via mlc-llm @ 4bit quantization + whisper-streaming + coqui TTS, all running simultaneously on one 4090 in real time. It didn't take long to prototype. Polishing and shipping it to non-expert users would take much longer than I've spent on it so far. I'd have to test for and solve a ton of installation problems, find better workarounds for whisper-streaming's hallucination issues, improve th…

> It didn't take long to prototype. Polishing and shipping it to non-expert users would take much longer than I've spent on it so far. I'd have to test for and solve a ton of installation problems I've found some success at this by using Nix... but Nix is a whole 'nother ball of yarn to learn. It WILL get you to declarative/deterministic installs of any piece of the toolchain it covers, though, and it does a hell of…

I'm not the guy working on ordering, it's this guy https://news.ycombinator.com/user?id=TheEzEzz.

Nix sounds good for duplicating my setup on other machines I control. But I'd like a way to install it on user machines, users who probably don't want to install Nix just for my thing. Nix probably doesn't have a way to make self contained packages, right?

Re: We are beginning to roll out new voice and image capabilities in ChatGPT

#888

Post by TerrifiedMouse »

famouswaffles wrote: post #783
TerrifiedMouse wrote: post #727

Earlier quoted context omitted.

> I feel like using LLM today is like using search 15 years ago - you get a feel for getting results you want. I don't think it's quite the same. With search results, aka web sites, you can compare between them and get a "majority opinion" if you have doubts - it doesn't guarantee correctness but it does improve the odds. Some sites are also more reputable and reliable than others - e.g. if the information is from Re…

>although some like Bard provide alternate drafts but they are all from the same source and can all be hallucinations ... Yes and no. If the LLM is repeating the same thing on multiple drafts then it's very unlikely to be a hallucination. It's when multiple generations are all saying different things that you need to take notice. LLMs hallucinate yes but getting the same hallucination multiple times is incredibly rar…

Then why aren’t hallucinations being eliminated by comparing drafts?

Re: We are beginning to roll out new voice and image capabilities in ChatGPT

#889

Post by joshspankit »

barfingclouds wrote: post #885
modeless wrote: post #459

Voice has the potential to be awesome. This demo is really underwhelming to me because of the multi-second latency between the query and response, just like every other lame voice assistant. It doesn't have to be this way! I have a local demo using Llama 2 that responds in about half a second and it feels like talking to an actual person instead of like Siri or something. I really should package it up so people can t…

There needs to be an optional button that you hold while speaking and let go when you are done. If button is not held it should auto detect

To me this is the cleanest and most efficient solution to the problem.

Tbh, ever since voice assistants landed I’ve wanted a handheld mic with a hardware button. No wake command, no (extra) surveillance, just snappy low-latency responses.

Re: We are beginning to roll out new voice and image capabilities in ChatGPT

#890

Post by joshspankit »

modeless wrote: post #839
pmarreck wrote: post #805

Earlier quoted context omitted.

Do you have a rough design outline of what you built? I feel like we're on the cusp of something like this and it sounds amazing.

I'm using Llama2-chat-13B via mlc-llm @ 4bit quantization + whisper-streaming + coqui TTS, all running simultaneously on one 4090 in real time. It didn't take long to prototype. Polishing and shipping it to non-expert users would take much longer than I've spent on it so far. I'd have to test for and solve a ton of installation problems, find better workarounds for whisper-streaming's hallucination issues, improve th…

I wish docker could be used more easily with graphic cards and other hardware peripherals (speakers/mic in this case). It would solve a lot of these issues.
Post reply on HN