Live data from Hacker News

Challenges with semantic search on transcribed audio files

notjoemartinez.com

11–13 of 13 posts

Re: Challenges with semantic search on transcribed audio files

#11

Earlier quoted context omitted.

I've found that 512 characters, more or less, is a good length for embeddings. More than that, and a bunch of models will just ignore the text. Less than half that, there isn't always enough context in the text to be able to group the chunks by keyterm or other labels. I do this type of additive context using the first few results from the embedding search. If it's tweets, it makes sense to use shorter chunks. I was…

That makes a lot of sense that a fuzzy 512 character window that preserves sentences properly would be a good strategy. Do you also use nltk to extract keyterm then? I was looking at Qdrant labels and thinking about how vectors could be labeled with location metadata but also perhaps with named entity information from the chunk. I imagine it how I might flip though a book using the index and also table of contents an…

I use a dict completion prompt to extract keyterms from various LLMs. Technically, this is transfer learning and I use it in a way that allows set operations to filter the vector space to speed things up. In a version I built of this, I referred to it as a "back of book index".

One issue would be that the prompt for keyterm extraction might miss crucial keyterms, but a subsequent access of those might update them...

Re: Challenges with semantic search on transcribed audio files

#12

Rather than simply embedding the query, you may want to look into query expansion techniques. You might find that expanding the query semantically and then using more old school (read: extremely fast) tools like bm25 or tf-idf are pretty good already. Your fundamental problem is that short snippets don’t have a lot of semantic meaning, but long snippets lose specificity. So one way of dealing with that would be to in…

I contemplated the same problem and concluded the thing to try first would be probabilistic clustering of snippets, but I haven’t actually tried it yet.

Re: Challenges with semantic search on transcribed audio files

#13

Do you have access to the waveform of the video? The most natural segmentation of text here seems to me to be on significant quiet periods in the waveform. An ideal solution is likely adaptive based on the video style which is likely unique to the creator. But I think you could get far with a sound below db threshold for x millis approach.

The transcription/subtitles have punctuation, so I think looking for full stops ("periods") would provide a similar to result to audio analysis. PS. The technical search term for the audio process you have described is VAD.
Post reply on HN