One of the hardest parts I've found is the diarisation (who said what) side of things. Trying to tune this and have it working in a way that doesn't absolutely grind the laptop to a halt or take forever to complete has been _hard_ but also extremely rewarding.
Another part has been the fine tuning side of the Phi-4 model, I'm on version 10 now, getting that pipeline down was a journey in itself, but I've got some great results. I wrote a bit about it in a comment here - https://news.ycombinator.com/item?id=48385906#48389625
I absolutely love working on this, I still wake up and the first thing I think about is voice transcription pipelines (sad I know), but I'm excited to see how much further performance and utility I can squeeze out.