Thanks for creating this. I can imagine a not-so-distant future where thousands of random video-watchers could annotate tiny parts of videos via some free-form box, and aeneas could clean up and formalize this into an official transcription. Seems like a minor feature, until one realizes how much the public just lost due to missing transcriptions: https://www.washingtonpost.com/local/education/why-uc-berkel...
Show HN: Aeneas – a Python audio/text aligner
11–20 of 37 posts
Re: Show HN: Aeneas – a Python audio/text aligner
#12This is really cool! Would you like me to make a conda package for this? I can do so for Linux and OSX so that someone who uses python for data science can do `conda install aeneas` and it will install this and it's dependencies into a virtualenv. I'd do it on windows too, but I don't know of an easy way to get my hands on a windows box. If anyone knows of a service that can give me 30 minutes of CLI access to a wind…
AppVeyor has free Windows CI for open-source projects (which probably would be a good idea to set up for later updates), and they explicitly mention that you can install remote access tools in their build VMs to work on/debug the build: https://www.appveyor.com/docs/how-to/rdp-to-build-worker/
Re: Show HN: Aeneas – a Python audio/text aligner
#13This code is also useful " rel="nofollow">https://github.com/lowerquality/gentle>
Re: Show HN: Aeneas – a Python audio/text aligner
#14Earlier quoted context omitted.
aeneas is not based on ASR (i.e., it does not try to "recognize" words and align them with the input text), but on the "older" MFCC + DTW approach. Hence, it is difficult to give you a precise answer, e.g. in terms of word-error-rate or similar metrics. For the task aeneas has been designed for --- aligning an ebook and the corresponding audiobook --- and for similar tasks (e.g., captioning videos of lectures or spok…
Thanks for the explanation. Will it work if there are gaps in the transcript? Eg, the clean verbatim transcript where the ah's and uhm's are left out.
Several users of aeneas interested in producing caption files for videos told me that it does. And considering how DTW works, it is plausible.
Unfortunately, I have not had the time to setting up a suitable corpus and performing a rigorous evaluation to comfortably answering your question with a definitive answer "yes".
Perhaps the best option to see if aeneas works for your use case, consists in trying it out.
If you do not want to install anything on your machine, you can use the aeneas Web app: https://aeneasweb.org --- basically you submit an audio file (or a YouTube URL) and a text file, and get a SRT/TTML/etc. file emailed back.
Re: Show HN: Aeneas – a Python audio/text aligner
#15This is super cool. I'm trying to think of common practical applications for this - would one use this to sync a script with a performance? Could this remove a lot of the work required to manually subtitle movies, TV shows, and YouTube videos?
Thank you. Indeed several users of aeneas adopted it for producing SRT/TTML files, i.e. captions, for videos, both online and offline --- and many of them start with an existing transcript. However, please note that there are limitations on the amount of "non speech" that aeneas can tolerate: for example, long spurious portions of audio or sung passages might affect the quality of the alignment. For details on how ae…
> there are limitations on the amount of "non speech" that aeneas can tolerate
couldn't you have as part of the input also a very simple map where users could define times that should be ignored to help with that? Might also be possible to look at the spectrum at any time to possibly identify areas of the file to skip.
And speaking about spectrum, just wondering, are you doing any pre-processing in terms of EQ (narrow band-pass on spoken frequencies), compression to not deal with volume, etc. to help with this also?
Re: Show HN: Aeneas – a Python audio/text aligner
#16"Audio is assumed to be spoken: not suitable for song captioning"
Can anyone recommend alternative approaches for music lyrics alignment?
Re: Show HN: Aeneas – a Python audio/text aligner
#17This code is also useful " rel="nofollow">https://github.com/lowerquality/gentle>
Yes, there are several other open source aligners out there, mostly from academic research or derived from academic projects. In my personal GitHub page I have a repo with an annotated list of forced aligners. (If I add a link to it, the spam detector triggers ?! Anyway, google "github forced-alignment-tools" to find it.)
Gentle, which is based on Kaldi, has a good performance, and an handy setup script.
However, these aligners, which are based on automatic speech recognition techniques, have pre-trained models only for English and maybe an handful of other "popular" languages. Some allows you to train your own language model, but very few users have the actual competence/resources for doing that.
aeneas is build using an older approach, which has the advantage of requiring weaker language models, that are already available (in the form of TTS voices): this is the reason why it "supports" so many languages. Of course the disadvantage is that aeneas works decently well at (sub)sentence granularity, but worse than ASR-based aligners at word granularity or with more noisy audio files.
Re: Show HN: Aeneas – a Python audio/text aligner
#18Earlier quoted context omitted.
Thank you. Indeed several users of aeneas adopted it for producing SRT/TTML files, i.e. captions, for videos, both online and offline --- and many of them start with an existing transcript. However, please note that there are limitations on the amount of "non speech" that aeneas can tolerate: for example, long spurious portions of audio or sung passages might affect the quality of the alignment. For details on how ae…
> there are limitations on the amount of "non speech" that aeneas can tolerate couldn't you have as part of the input also a very simple map where users could define times that should be ignored to help with that? Might also be possible to look at the spectrum at any time to possibly identify areas of the file to skip. And speaking about spectrum, just wondering, are you doing any pre-processing in terms of EQ (narro…
Definitely.
Actually, aeneas can be used as a Python library (rather than just a CLI tool), and you can definitely provide an audio file, a list of audio intervals where the spoken text is, and align "piece-wise". See the "aeneas library tutorial" in the docs.
At the moment, the CLI tool aligns only a single audio interval (possibly chopping the head or the tail of the audio file) --- which is just a special case of the above case.
I remember a user requested this feature in the past. I have not added it yet because:
1. I have not heard much interest about it, and I have not needed it myself;
2. I am not satisfied with the current CLI interface --- (historical reasons mandated it the use of big config strings and strange, long parameter names) --- and hence I think that this kind of new features should be added once aeneas 2.x is out, with a redesigned CLI.
Re: Show HN: Aeneas – a Python audio/text aligner
#19This is super cool. I'm trying to think of common practical applications for this - would one use this to sync a script with a performance? Could this remove a lot of the work required to manually subtitle movies, TV shows, and YouTube videos?
When I was an undergrad freshman, I took a job with a research group as a data annotator. My job was to go through the Switchboard corpus (recordings of hour-long phone calls that people agreed to have recorded, in exchange for having the long-distance charges paid) and label features such as who was speaking, whether the pitch of the voice was rising or falling, whether the vowels were elongated, vocal fry, and stuff like that.
But the most time-consuming and mind-numbing part of it was just annotating the words in the sound file.
The interface for all of this was a terrible GUI hacked in on top of some Solaris sound editor, and it couldn't do things for you like find the moments that words began, or say "hey the pitch is obviously falling here" because frequency tracking is a thing computers can do, or anything.
There's still a lot more voice data to annotate in the world, and maybe having a flexible Python tool like this will make the next undergrad doing the grunt work much more effective at it.
Re: Show HN: Aeneas – a Python audio/text aligner
#20Earlier quoted context omitted.
Thank you. Indeed several users of aeneas adopted it for producing SRT/TTML files, i.e. captions, for videos, both online and offline --- and many of them start with an existing transcript. However, please note that there are limitations on the amount of "non speech" that aeneas can tolerate: for example, long spurious portions of audio or sung passages might affect the quality of the alignment. For details on how ae…
> there are limitations on the amount of "non speech" that aeneas can tolerate couldn't you have as part of the input also a very simple map where users could define times that should be ignored to help with that? Might also be possible to look at the spectrum at any time to possibly identify areas of the file to skip. And speaking about spectrum, just wondering, are you doing any pre-processing in terms of EQ (narro…
> Might also be possible to look at the spectrum at any time to possibly identify areas of the file to skip.
I would say yes and no.
Currently you can add a switch that makes aeneas ignore the audio intervals that are detected as "non speech" by the built-in Voice Activity Detector (VAD), which is a very rough energy-based VAD. For sure this is a part that can use some improvement.
However, AFAIK e.g. music/singing separation is a really difficult open problem, with people in academia doing PhDs on it. So, I am not sure how far one can push this line, while staying relatively fast on a regular machine. (Which is one of the goals of aeneas.)
> And speaking about spectrum, just wondering, are you doing any pre-processing in terms of EQ (narrow band-pass on spoken frequencies), compression to not deal with volume, etc. to help with this also?
Besides converting the input audio file to mono 16 kHz 16 bit WAVE, I do not perform any other operation on the audio data before passing it to the MFCC extractor (which by default runs with "standard" settings, but the user can change them).
Unfortunately, I have had no time to perform an exhaustive search of the parameter space, nor to try other pre-processing techniques.
But for sure if you have means to "pre-clean" the audio file before feeding it into aeneas, that is probably going to improve the quality of the output alignment.
(I did play with amplitude normalization and it did not seem to improve the results. The non-speech masking mentioned above seems beneficial if you do word-level alignment.)