Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Generate audiobooks from E-books with Kokoro-82M

claudio.uk

Post reply on HN
Print view View original

51–60 of 255 posts

Re: Generate audiobooks from E-books with Kokoro-82M

#51

Post by micw »

KeplerBoy wrote: post #46
micw wrote: post #42

Earlier quoted context omitted.

With this technology, one could produce high quality audio books without having access to high quality narrators by annotating the books with the voice, speed and such things. I wonder if a standardized markup exists to do so.

Don't end to end trained models already do this to some extent? Like raising the pitch towards a question mark, like a human would. TortoiseTTS has a few examples under prompt engineering on their demo site: https://nonint.com/static/tortoise_v2_examples.html

That's a bit of basic and random. Some models have the features you describe. From the better models you get a slightly different voice for text in quotes.

But the difference to good audio books is that you have * different voices for the narrator and each character * different emotions and/or speed in certain situations.

I guess you could use a LLM to "understand" and annotate an existing book if there's a markup and then use TTS to create an audio book from it and so automate most of the the process.

Re: Generate audiobooks from E-books with Kokoro-82M

#52

Post by nottorp »

stavros wrote: post #47
nottorp wrote: post #45

Well there was some hope with ChatGPT that people will go back to being able to process text communication. Guess it was just a matter of time till someone figured out how to use "AI" to resume encouraging illiteracy.

There was some hope with the rise of equestrianism that people will go back to be able to shoe horses. Guess it was just a matter of time till someone figured out how to use "cars" to resume encouraging being unable to to a basic farrier job.

Except cars were faster than horses, while audio or video content is much slower than reading.

Re: Generate audiobooks from E-books with Kokoro-82M

#53

Post by dmazin »

laserbeam wrote: post #21

On the one hand, this is very convenient. Probably cool for some non-fiction. On the other, some of my favorite audio books all stood out because the narrator was interpreting the text really well, for example by changing the pacing during chaotic moments. Or those audiobooks with multiple narrators and different voices for each character. Not to mention that sometimes the only cue you get for who's speaking during d…

Absolutely.

Even on the non-fiction side, the narration for Gleick's The Information adds something.

While I want this tool for all the stuff with no narration, NYT/New Yorker/etc replacing human narrators with AI ones has been so shitty. The human narrators sound good, not just average. They add something. The AI narrators are simply bad.

Re: Generate audiobooks from E-books with Kokoro-82M

#54

Post by micw »

micw wrote: post #51
KeplerBoy wrote: post #46

Earlier quoted context omitted.

Don't end to end trained models already do this to some extent? Like raising the pitch towards a question mark, like a human would. TortoiseTTS has a few examples under prompt engineering on their demo site: https://nonint.com/static/tortoise_v2_examples.html

That's a bit of basic and random. Some models have the features you describe. From the better models you get a slightly different voice for text in quotes. But the difference to good audio books is that you have * different voices for the narrator and each character * different emotions and/or speed in certain situations. I guess you could use a LLM to "understand" and annotate an existing book if there's a markup an…

Edit: I actually tried this. I prompted in ChatGPT:

"Annotate the following text with speakers and emotions so that it can be turned into an audiobook via TTS", followed by a short text from "The Hobbit" (The "Good morning scene"). The result is very good.

Re: Generate audiobooks from E-books with Kokoro-82M

#55

Post by Melomomololo »

laserbeam wrote: post #21

On the one hand, this is very convenient. Probably cool for some non-fiction. On the other, some of my favorite audio books all stood out because the narrator was interpreting the text really well, for example by changing the pacing during chaotic moments. Or those audiobooks with multiple narrators and different voices for each character. Not to mention that sometimes the only cue you get for who's speaking during d…

I like one speaker in one particular book.

He also narrates another scifi book series and honestly I dislike this a lot.

He became the voice of one particular character for me.

I would love variety

Re: Generate audiobooks from E-books with Kokoro-82M

#56

Post by albert_e »

InsideOutSanta wrote: post #30
pprotas wrote: post #25

I would love to have an e-reader that allows me to switch between text and audio at the press of a button. Imagine reading your book on the couch and then switching into audio mode while doing the dishes seamlessly, by connecting bluetooth headphones.

Kindles used to provide this feature, but publishers and/or the Authors Guild stopped it, because audio rights and text rights are handled differently. In other words, when Amazon sells you a text book, it does not have the right to then also do TTS on that text and let you listen to it. There's some contemporary discussion of what happened here: https://tidbits.com/2009/03/02/why-the-kindle-2-should-speak... I think…

Yes the feature is called WhisperSync -- I used it many years ago and it was pretty good.

I tried it while on a treadmill so it allowed me to follow the book with more focus without sacrificing much else.

Re: Generate audiobooks from E-books with Kokoro-82M

#57

Post by stavros »

nottorp wrote: post #52
stavros wrote: post #47

Earlier quoted context omitted.

There was some hope with the rise of equestrianism that people will go back to be able to shoe horses. Guess it was just a matter of time till someone figured out how to use "cars" to resume encouraging being unable to to a basic farrier job.

Except cars were faster than horses, while audio or video content is much slower than reading.

Cars also have legs while audio doesn't, a point which is equally irrelevant. If people don't need to read, they don't need to read, and no matter how much a random Internet commenter wants them to need it, it won't change anything.

Skills atrophy for a reason. It's fine to let them. You may as well be lamenting the lost art of long division.

Re: Generate audiobooks from E-books with Kokoro-82M

#60

Post by micw »

albert_e wrote: post #50
micw wrote: post #42

Earlier quoted context omitted.

With this technology, one could produce high quality audio books without having access to high quality narrators by annotating the books with the voice, speed and such things. I wonder if a standardized markup exists to do so.

There is SSML for speech markup to indicate various characters of speech like whispers, pronunciation, pace, emphasis, etc. With LLMs proving to be very good at generating code, it may be reasonable to assume they can get good at generating SSML as well. Not sure if there is a more direct way to channel the interpretation of the tone/context/emotion etc from prose into generated voice qualities. If we train some mode…

Good points, thank you! I just tested it. While ChatGPT was very good in adding generic (textual) annotations, the result for generating SSML where very poor (lack of voice names, lack of distinction between narrator and character etc).

Probably the results with a model trained for this plus human audit could lead to very good results.

Post reply on HN