Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Aeneas – a Python audio/text aligner

github.com

Post reply on HN
Print view View original

21–30 of 37 posts

Re: Show HN: Aeneas – a Python audio/text aligner

#21

Post by alpe »

rspeer wrote: post #19
psobot wrote: post #2

This is super cool. I'm trying to think of common practical applications for this - would one use this to sync a script with a performance? Could this remove a lot of the work required to manually subtitle movies, TV shows, and YouTube videos?

When I was an undergrad freshman, I took a job with a research group as a data annotator. My job was to go through the Switchboard corpus (recordings of hour-long phone calls that people agreed to have recorded, in exchange for having the long-distance charges paid) and label features such as who was speaking, whether the pitch of the voice was rising or falling, whether the vowels were elongated, vocal fry, and stuf…

I agree on most of your observations.

However, please note that other tools are better suited than aeneas if one wants to align at phoneme level: gentle, Kaldi, SPPAS, etc.

aeneas' goals are covering as many languages as possible, fast computing, targeting (sub)sentence granularity (e.g., ebook-audiobook or closed captions). Phoneme-level annotation really requires more sophisticated techniques, like HMM/GMM/NN as implemented by the tools mentioned above. Yet, aeneas can be used to quickly bootstrap e.g. a manually-reviewed alignment.

Re: Show HN: Aeneas – a Python audio/text aligner

#22

Post by alpe »

TuringNYC wrote: post #11

Thanks for creating this. I can imagine a not-so-distant future where thousands of random video-watchers could annotate tiny parts of videos via some free-form box, and aeneas could clean up and formalize this into an official transcription. Seems like a minor feature, until one realizes how much the public just lost due to missing transcriptions: https://www.washingtonpost.com/local/education/why-uc-berkel...

Thank you.

Indeed, while aeneas was created for ebook-audiobook synchronization, several of its current users are producing closed captions --- because, in most cases, they already have a clean transcript (e.g., speakers provide transcripts to the captioner) or they clean up an automated transcript, derived from an automatic speech recognition system.

Re: Show HN: Aeneas – a Python audio/text aligner

#24

Post by x1798DE »

psobot wrote: post #2

This is super cool. I'm trying to think of common practical applications for this - would one use this to sync a script with a performance? Could this remove a lot of the work required to manually subtitle movies, TV shows, and YouTube videos?

I had a vague plan to start working on something like this recently with the idea that I could automatically take audiobook media files and their accompanying ebook representation and use it to automatically re-divide the file by chapter (or using something based on chapter). Not sure if this will work well for that (or if my use is considered "common"), but I'm certainly glad to see it.

Re: Show HN: Aeneas – a Python audio/text aligner

#25

Post by cityhall »

psobot wrote: post #2

This is super cool. I'm trying to think of common practical applications for this - would one use this to sync a script with a performance? Could this remove a lot of the work required to manually subtitle movies, TV shows, and YouTube videos?

There's nothing new about this, it's how speech recognition training data has been generated for a long time. Whether you can align a script will depend on how accurate it is and how expressive your models are for generating spoken/surface form alternatives for the ways things like dates are verbalized, which look different in text. If more than one person is speaking at the same time the results will be terrible.

Re: Show HN: Aeneas – a Python audio/text aligner

#26

Post by sargun »

echelon wrote: post #23

This is going to be beyond useful for me. I can extract far more labeled audio samples for my Donald Trump text to speech engine [1]. Thanks for sharing this! [1] http://jungle.horse

Have you looked at applying the techniques used in Google's Wavenet to your corpus? In addition, any interest in releasing your corpus?

Re: Show HN: Aeneas – a Python audio/text aligner

#27

Post by alpe »

cityhall wrote: post #25
psobot wrote: post #2

This is super cool. I'm trying to think of common practical applications for this - would one use this to sync a script with a performance? Could this remove a lot of the work required to manually subtitle movies, TV shows, and YouTube videos?

There's nothing new about this, it's how speech recognition training data has been generated for a long time. Whether you can align a script will depend on how accurate it is and how expressive your models are for generating spoken/surface form alternatives for the ways things like dates are verbalized, which look different in text. If more than one person is speaking at the same time the results will be terrible.

I would like to note once again that aeneas is not based on automatic speech recognition techniques, but on MFCC + DTW, which is an even older approach, with pro's and con's.

Interestingly, there are situations where ASR-based forced aligners seem to be tricked into error, while aeneas handles them more robustly --- for example, if the speaker repeats a word in the spoken audio, but the transcript has only one occurrence, or when the speaker mumbles (uhm's, ah's, etc.). On the other hand, it is true that if you want word- or phoneme- alignment, ASR-based aligners outperform aeneas.

Finally, let me note three major goals of aeneas are: 1. be able to process hours of audio relatively fast on a standard PC (the current real time factor is between 0.008 and 0.020); 2. easy to install and run (unlike many other open source aligners derived from academic projects, which require a PhD just to get the dependencies right); and 3. working out-of-the-box for many languages, including ones that are not covered by academia or commercial solutions because they are "minor" (say, Icelandic or ancient Greek (!)).

But yes, the core algorithmic approach of aeneas has been around since the 1970s.

Re: Show HN: Aeneas – a Python audio/text aligner

#28

Post by alpe »

x1798DE wrote: post #24
psobot wrote: post #2

This is super cool. I'm trying to think of common practical applications for this - would one use this to sync a script with a performance? Could this remove a lot of the work required to manually subtitle movies, TV shows, and YouTube videos?

I had a vague plan to start working on something like this recently with the idea that I could automatically take audiobook media files and their accompanying ebook representation and use it to automatically re-divide the file by chapter (or using something based on chapter). Not sure if this will work well for that (or if my use is considered "common"), but I'm certainly glad to see it.

I have used aeneas myself to do it, with mixed results. You will probably need to increase the DTW margin. Also note that you will need a lot of RAM --- say 16 GB if you plan to work on a single audio file with duration 10-15 hours, which is typical for an audiobook.

In theory one can perform the DTW out-of-core, saving the accumulated cost matrix and path to disk, but I have had not time to implement this yet (i.e., the accumulated, reduced DTW cost matrix should fit into RAM). I tested it can be done with PyTables, but it will probably come with the next major version of aeneas (v2).

BTW, if your goal is to split, say, chapters of an audiobook, probably there are more efficient ways of doing this. For example, finding the long silence intervals between chapters might be enough. Or, instead of aligning all the text against all the audio, just perform a "partial matching" of the first sentences of each chapter against the audio.

Re: Show HN: Aeneas – a Python audio/text aligner

#29

Post by hftf »

alpe wrote: post #17
oulipo wrote: post #13

This code is also useful " rel="nofollow">https://github.com/lowerquality/gentle>

Yes, there are several other open source aligners out there, mostly from academic research or derived from academic projects. In my personal GitHub page I have a repo with an annotated list of forced aligners. (If I add a link to it, the spam detector triggers ?! Anyway, google "github forced-alignment-tools" to find it.) Gentle, which is based on Kaldi, has a good performance, and an handy setup script. However, the…

Do you know of any existing forced alignment tools that work well with live audio (microphone) input? I would like to create a live stream in which the words of a known text are displayed as they are being spoken into a microphone.

Re: Show HN: Aeneas – a Python audio/text aligner

#30

Post by [deleted] »

TuringNYC wrote: post #11

Thanks for creating this. I can imagine a not-so-distant future where thousands of random video-watchers could annotate tiny parts of videos via some free-form box, and aeneas could clean up and formalize this into an official transcription. Seems like a minor feature, until one realizes how much the public just lost due to missing transcriptions: https://www.washingtonpost.com/local/education/why-uc-berkel...

[deleted]
Post reply on HN