Earlier quoted context omitted.
My favorite real-world example that I worked on: I created a YouTube documentary series about corruption in a small town in Illinois. This required downloading thousands of hours of government meeting videos from YouTube, transcribing + chunking + embedding, summarizing meeting segments, running a couple passes of validation, then searching for interesting storylines. I also scraped thousands of public documents whic…
Do you have a writeup of how you did it, which tools you used? I have a similar project on a much more mundane topic, where I want to build a searchable index of Youtube videos, blog posts, etc. based on their content.
My advice would be to first set up an automation for yt-dlp to pull the media, then use Whisper to build a transcription pipeline. Chunk the transcript based on desired result granularity, then store embeddings in local Qdrant. It helps to use an orchestrator to handle all of this - my current recommendation is Dagster (dagster.io).