This is very interesting. I do something related: matching student oral reading transcripts back against the text they were assigned to read.Students will misread then re-read a sentence, skip sentences, skip paragraphs etc. I think its similar because we both seek to assign audio time stamps to sentences and words. But I wonder why your forced alignment algorithm is so heavy duty. (My head started to spin at CTC emi…
The current Storyteller alignment algorithm actually does do just that! We use Whisper to transcribe the audio to text, and then use error-align[1] to align on the text.
There are a few disadvantages to this approach:
1. Whisper only supports ~25 languages, and only about 10 of those very well. We want to support more languages, and Massively Multilingual Speech supports "1000+" 2. Whisper's timing outputs are not very good. We want to do word-level highlighting, like in the demo at the top of the post, but in order for that to be a good user experience, those timings need to be very precise. Much easier to do that with CTC!
CTC Viterbi is the tried and true forced alignment algorithm for good reason. It's not really that it's heavier duty than running Whisper and aligning on the output. Rather, it's like you stop Whisper early, before it does the final step of actually producing text, and step in and say: take the data you just calculated and use it to produce _this_ text, specifically. And then, since it produced _your_ text, you don't have to do anything else, you just use the timestamps directly.
The only reason Storyteller never used it in the past is because I couldn't come up with a good way to do the boundary search I describe in this post! This is super important for books in a way that it may not be for your oral reading transcript use case, because chapters can be (and often are) out of order between the ebook and audiobook. But once I worked out the n-gram RANSAC approach, it became much more tenable.