Transcribe.cpp

It sounds like the only way this would make sense is if such model knew the range of sounds it expects to "hear". There's a lot of possible sounds that IPA knows about, but world languages only use a fraction of them at once. Think English dark and light "l" (ball/light) or aspirated "p" (pin/spin) - some languages contrast them, while in english the difference is not meaningful.

Or maybe linguists are actually interested in having maximally faithful IPA representation and manually normalizing it? You are clearly way more knowledgeable about that topic than I, so I'm curious what you think.

One thing I find that's missing a lot or at least I haven't come across other than commercial offerings like AquaVoice is a decent injected technical vocabulary so that the initial transcript requires minimum cleanup afterwards. Because I mostly use these tools to essentially ramble at the command line with coding agents. So there's a lot of technical terms that don't translate well. Like OpenBao comes out as open bowel sometimes,lol. That necessitates significant cleanup prompt or background text available to the cleanup llm, usually in the form of screenshot or something that gets converted to text but that in turn requires good hw for speed to be almost imperceptible. For example m5 max turns cleanup into a noticeable delay while 5090 is decent.

Only way I have found that's relatively easy to inject technical vocab is to use whisper, but limited, I think to about 220 or so tokens. Whisper has sort of like a priming prompt where one can put in a bunch of technical words and it will try to recognize those. But again, that's limited to small number tokens. And that limits one use a relatively slow, by today's standards, whisper.cpp.

I benchmarked it across a bunch of different hardware that I have available, and Whisper gives decent performance as far as speed goes only on a pretty top-end GPU, such as a 5090 or 4070, like for example on Strix Halo, it's still relatively slow for longer transcriptions because I prefer just a stream of consciousness ramblings for minutes and then that being transcribed and cleaned up versus short sentences. So in that scenario something like 5090 really is good because the cleanup prompt runs fast using usually Qwen 3.6 MOE model. Whisper on 4070 itself is about 0.7 seconds for two or three minute transcription. So the total wait time for a three-minute transcription is roughly a second, or a little bit more than a second, so totally acceptable. But it does take decent hardware, and it grows to be double that on if running totally local. Well, in my case, it's all local, but it's my own hardware all over the place, but truly running on laptop, it's much faster using Parakeet, but then the cleanup is the bottleneck.

Anyway, it's just my experience messing around with this for the last year. I did start using AquaVoice, but their speed was exceptional, and tech vocab was exceptional, but they would have some annoying delays occasionally, and I didn't like paying the money and sending sensitive topics and screenshots into the cloud, and I had hardware, so my local solution is basically almost as good as commercial one. But I think they train their own model. So what I'm doing is I collect all the samples of my transcriptions, and I am slowly building my own data set that hopefully at some point when I get energy I will find some way to fine tune something.

[dead]

The linguists I know are not necessarily a representative sample... but they're mostly interested in just that: maximally faithful IPA representation. They want to know if the speaker switches back and forth between aspirated p and unaspirated p on the same word, because that tells them something about the language — that aspirated consonants are not meaningful.

Linguists studying the sounds of a language, its phonology, often want to find "minimal pairs", words that differ by only a single sound. For example, din and tin in English. You record a native speaker saying both words, and telling you their meaning, and then you play back either recording A or recording B to other native speakers and ask them which word it is. If they can identify the word every time, then you've found two sounds that are meaningfully distinct in this language. (Some languages don't distinguish the d and t sounds, but English does). But if the native speakers go 50/50 on which word it is, or ask to hear it in a sentence for clarification because it could be two or three different words, then you've found a pair of sounds that this language does not distinguish. (Note that you're playing the words in isolation, because sentence context might make it obvious which one it is, e.g. you can't tell if an English speaker is saying their or there until you hear more words of the sentence).

So yes, the linguists that I know (who, again, are not necessarily a representative sample) are interested in as faithful an IPA representation as they can get, because that inconsistent transcription will give them many clues about the language. It still all has to be checked, because that switching back and forth between aspirated and unaspirated p (for example) could have been an artifact of a poor-quality microphone not picking up the aspiration, or a windy day causing aspiration sounds that the speaker never said, or the speech-to-text model making a mistake. But I watched my wife listen to the same two-second recording on loop over and over, trying to be certain of which sound she was hearing in the middle of the word. Double-checking the output of the model would (in most cases) only require listening to the audio once or twice, not half-a-dozen times like she typically did while researching her thesis. At the time, LLMs were not really a thing yet, but if she were doing her thesis today I bet a speech-to-IPA model would have saved her quite a lot of time — but only if it output every distinction, even the ones not meaningful in the target language. The "maximally faithful" representation, as you put it.

yes