Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Self-host Whisper As a Service with GUI and queueing

github.com

Post reply on HN
Print view View original

51–60 of 60 posts

Re: Show HN: Self-host Whisper As a Service with GUI and queueing

#51

Post by nshm »

tsycho wrote: post #49

Is there an open source speech recognition model which can be restricted to a smaller domain-specific dictionary? Use case: I want to transcribe my poker hands while playing, eg: "Flop was 2 of spaces, 3 of diamonds and King of spades", "Button raised to $20" etc. When I tried using Whisper and some other model, the recognition accuracy was atrocious, and it kept finding non-poker words that sounded similar to poker…

Vosk

https://alphacephei.com/vosk/lm

You can restrict the vocabulary the way you like, for example, here is the chess app built with Vosk

https://www.chessvis.com/

Re: Show HN: Self-host Whisper As a Service with GUI and queueing

#53

Post by [deleted] »

swyx wrote: post #19
nasir wrote: post #18

Earlier quoted context omitted.

I'm badly looking for that! Is there a repo I can follow?

not GP (hoping he responds tho) but i've been collecting a couple of diarization options: https://github.com/sw-yx/ai-notes/blob/main/AUDIO.md basically whisper.cpp has some support but its not great (based on my own testing) - https://huggingface.co/spaces/vumichien/whisper-speaker-diar... - https://github.com/Majdoddin/nlp pyannote diarization - whisperX with diarization https://twitter.com/maxhbain/status/16196987…

[deleted]

Re: Show HN: Self-host Whisper As a Service with GUI and queueing

#54

Post by [deleted] »

henry_viii wrote: post #8

By the way there is also another project called Whisper.cpp: https://github.com/ggerganov/whisper.cpp Which uses x8 less memory than the Python implementation for the tiny model. It would be a good idea to keep an eye on it since there are Python bindings planned on the roadmap: https://github.com/ggerganov/whisper.cpp#bindings

[deleted]

Re: Show HN: Self-host Whisper As a Service with GUI and queueing

#55

Post by Jach »

sgt wrote: post #11

Is the Whisper model better than say Youtube's auto transcribing? I hope it is because the one on YT gets so much wrong it's almost comical.

Generally yes when it produces sane output at all, but while YT can get stuff comically wrong I've never seen it just go off the rails and start hallucinating and mindlessly repeating itself, which Whisper sometimes does especially if you're also trying to get it to translate something. Like Whisper will sometimes output a stream of things like "Please subscribe to my channel and follow me on Twitter!" or "Thank you for watching.".

On one source I tried the other day, the first 90 seconds or so is just generic opening music, no speech, but it "transcribes" it as "This is the end of the video. Thank you for watching. Please subscribe to the channel if you like. See you in the next video. Thank you for watching. Please subscribe to the channel if you like. Thank you for watching. ..." If you help it along by cutting up the source into only spoken segments you can get it to do better but just throwing it at a directory of material is probably going to leave you with some disappointment.

Then sometimes it does something surprising, on a j-pop song after hallucinating a bit during the intro it spit out a translation in the form you might find on a lyrics site, that is each line was "japanese-characters romaji-version english-translation". I haven't been able to get it to do it again (even for the same source).

Re: Show HN: Self-host Whisper As a Service with GUI and queueing

#57

Post by kamranjon »

Jach wrote: post #55
sgt wrote: post #11

Is the Whisper model better than say Youtube's auto transcribing? I hope it is because the one on YT gets so much wrong it's almost comical.

Generally yes when it produces sane output at all, but while YT can get stuff comically wrong I've never seen it just go off the rails and start hallucinating and mindlessly repeating itself, which Whisper sometimes does especially if you're also trying to get it to translate something. Like Whisper will sometimes output a stream of things like "Please subscribe to my channel and follow me on Twitter!" or "Thank you…

Have you tried setting condition_on_previous_text to False?

Re: Show HN: Self-host Whisper As a Service with GUI and queueing

#58

Post by teucris »

phodo wrote: post #56
teucris wrote: post #38

Very cool - I have a homegrown setup where a script scans my iCloud audio notes directory and generates transcriptions for any new notes. Works like a charm.

Is it running on a max or are you accessing iCloud from a non-Mac?

It’s on a Mac mini I use as a home server.

Re: Show HN: Self-host Whisper As a Service with GUI and queueing

#59

Post by Jach »

kamranjon wrote: post #57
Jach wrote: post #55

Earlier quoted context omitted.

Generally yes when it produces sane output at all, but while YT can get stuff comically wrong I've never seen it just go off the rails and start hallucinating and mindlessly repeating itself, which Whisper sometimes does especially if you're also trying to get it to translate something. Like Whisper will sometimes output a stream of things like "Please subscribe to my channel and follow me on Twitter!" or "Thank you…

Have you tried setting condition_on_previous_text to False?

Yeah, it can help a bit with looping, but introduces other problems. I recalled from earlier that a combo of tweaking no_speech_threshold and logprob_threshold settings helped somewhat, though trying again on a random video it doesn't do much. Still hallucinates a stream of captions (albeit non-repetitive, though one run had several Touhou related lines) for what should be 4 minutes of looping background music before the first sentence. If all one needs Whisper for is transcribing English though, I still think it's pretty decent. On my test video now it will 'correctly' transcribe the music as ♪ when I ask it to just transcribe it as English.

Re: Show HN: Self-host Whisper As a Service with GUI and queueing

#60

Post by 19h »

sebastianvoelkl wrote: post #10

The only thing Whisper misses is speaker diarization. I'm currently working on a model that uses Whisper + pyannote to transcribe Interviews and also detects who is speaking. It's working but damn it takes so long

https://github.com/ggerganov/whisper.cpp has diarization
Post reply on HN