Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: YouTube Full Text Search – Search all of a channel from the commandline

github.com

Post reply on HN
Print view View original

141–150 of 172 posts

Re: Show HN: YouTube Full Text Search – Search all of a channel from the commandline

#141

Post by smcleod »

xurukefi wrote: post #25

$ python yt_fts.py download 'https://www.youtube.com/@ycombinator/videos' [...] File "/app/yt_fts.py", line 176, in get_channel_id channel_id = re.search('channelId":"(.{24})"', html).group(1) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'group' $ python yt_fts.py download 'https://www.youtube.com/@ycombinator/videos' --channel-id UCj089h5WsDdh1q8t54K3ZCw [...] File…

Reminds me how much I dislike python error messages, not as much as java but still so much noise to signal by default.

Re: Show HN: YouTube Full Text Search – Search all of a channel from the commandline

#142

Post by mos_basik »

tiffanyg wrote: post #110
belter wrote: post #78

Earlier quoted context omitted.

I mean this: "Tell HN: YouTube's web UI just got even worse" - https://news.ycombinator.com/item?id=33371268 They used to have a drop down box, for oldest or most recent, when you clicked on the tab Videos for a YouTube channel. Now you will notice, it has now a "Latest" and "Popular", something that of course the algorithm will decide upon...

We seem to be locked into an ever more blatant "cycle of addiction". As with (other) drugs, the addict and the dealer are something like "two sides of the same coin". Briefly, new tech / markets / apps ... i.e., 'innovation' ... often 'creates' tremendous 'value'*. Money (aka capital) rushes in - there's a boom, everyone's excited: consumers are getting new products, services, conveniences, improved prices; engineers…

Sounds like you might enjoy reading Doctorow's essay on "enshittification".

"Here is how platforms die: first, they are good to their users; then they abuse their users to make things better for their business customers; finally, they abuse those business customers to claw back all the value for themselves. Then, they die."

-- Corey Doctorow, https://pluralistic.net/2023/01/21/potemkin-ai/#hey-guys

Re: Show HN: YouTube Full Text Search – Search all of a channel from the commandline

#143

Post by harshreality »

crazygringo wrote: post #135
derefr wrote: post #3

I love that a third party is stepping up here, but it's incomprehensible to me that Google doesn't do this themselves. They're a search company, and they own YouTube. The YouTube data — including the subtitle files — is already sitting there on their servers; they don't have to scrape it, they just have to index it. What are they even doing? Fun thing to try: do a Google search with "site:youtube.com" in it. You get…

Google already does this themselves. If you search for rare words (e.g. try "indubitably") it will absolutely pull up videos that have the word in the auto-generated transcript and nowhere else (not in descriptions, not in comments). Also, using "site:youtube.com" on Google works perfectly for me. If I look up "site:youtube.com david letterman" it gives me the David Letterman channel, followed by a seemingly infinite…

What I'd expect is that

"having a distribution that's both radially symmetric" site:youtube.com

would return 3b1b's "Why π is in the normal distribution" video, which has that in subtitles at 22:28.

Even without the site: term, all I get is an allreadable.com page that's scraped the subtitles for that video. Allreadable appears at first glance to be a site owned by someone in China and hosted on liquidweb.

Re: Show HN: YouTube Full Text Search – Search all of a channel from the commandline

#144

Post by ramraj07 »

roncesvalles wrote: post #119
ramraj07 wrote: post #62

Earlier quoted context omitted.

I’m pretty sure google uses the captions in search. As long as you search from within YouTube. I regularly search for keywords and find hourlong videos where it’s mentioned somewhere in the middle and nowhere in the description.

I suspect what you see is the search "learning" from the users -- someone is looking for a specific video that they've seen before but need to refine the search terms a few times before they find it. Thereafter, Google remembers the entire sequence of search terms and associates them with the video even though none of those terms might actually exist in the indexed video metadata. I've noticed that Google Search at l…

I mean Google search does it obviously using page rank which is that if someone links to the page with that word it uses it. Me I was searching for arcane words that I remembered hearing in the middle of a podcast. I doubt anyone actually linked or searched for the same with relevance to that podcast. Also the solution you're saying needs far more intuitive and intricate development than just indexing captions lol.

Re: Show HN: YouTube Full Text Search – Search all of a channel from the commandline

#145

Post by zdwolfe »

gukoff wrote: post #137
cced wrote: post #55

Earlier quoted context omitted.

This is exactly what I’ve built. Nothing fancy just ytdlp + whisper + ripgrep + fzf and I’ve got a pretty interesting way to ctrl+f my YT history.

Mind to share? I'd like to try this out

Not OP, but I too wrote something nearly identical with whisper so I could creep on old EthosLab videos. Here's the gist:

from pytube import Channel

import whisper

channel_yt = Channel(channel_url)

video_yt = channel_yt.videos[0]

video_yt_stream = video_yt.streams.filter(mime_type="video/mp4").filter(res="720p").first()

video_yt_video_file_path = video_yt_stream.download()

audio = whisper.load_audio(video_yt_video_file_path)

model = whisper.load_model("tiny")

transcript = model.transcribe(audio)

Re: Show HN: YouTube Full Text Search – Search all of a channel from the commandline

#146

Post by crazygringo »

harshreality wrote: post #143
crazygringo wrote: post #135

Earlier quoted context omitted.

Google already does this themselves. If you search for rare words (e.g. try "indubitably") it will absolutely pull up videos that have the word in the auto-generated transcript and nowhere else (not in descriptions, not in comments). Also, using "site:youtube.com" on Google works perfectly for me. If I look up "site:youtube.com david letterman" it gives me the David Letterman channel, followed by a seemingly infinite…

What I'd expect is that "having a distribution that's both radially symmetric" site:youtube.com would return 3b1b's "Why π is in the normal distribution" video, which has that in subtitles at 22:28. Even without the site: term, all I get is an allreadable.com page that's scraped the subtitles for that video. Allreadable appears at first glance to be a site owned by someone in China and hosted on liquidweb.

I just checked and it returns that video as the first result if you put your phrase into the YouTube search box in quotes, so that's good news.

I wonder why YouTube indexes the transcripts for YouTube search, but Google chooses not to include them in its index for Google search? Seems like an intentional decision since it's the same company after all.

Re: Show HN: YouTube Full Text Search – Search all of a channel from the commandline

#147

Post by lennxa »

gorgoiler wrote: post #10

Very nice! FYI: sqlite ships with a full text search engine featuring a Boolean query language, highlight(), snippet() and scoring: https://www.sqlite.org/fts5.html I’ve not used it with enough content to know how much faster it is than LIKE ‘%my query%’ but it should be a lot quicker. (Also, in most cases you don’t need to create an id column — every table has one already in the form of rowid .)

Not sure if this includes fuzzy search, but having it will make this much more usable.

Re: Show HN: YouTube Full Text Search – Search all of a channel from the commandline

#148

Post by tonto »

harlanji wrote: post #104

I hate being the poo pooer who says that subtitles are available via the API and wish the tool went that route. I'm all for stuff being archivable with tools like youtube-dl, but I much prefer to see tools like this use the API despite its quotas because it goes beyond archiving a copy for reference. Tools that (ab)use scraping only justify anti-scraping efforts that journalists and the like use and escalate that arm…

agree that using the API is likely the nicer route. you can also apply for a quota increase, I recently applied for youtube API quota increase to 100,000 units and it was approved for my app (https://cmdcolin.github.io/ytshuffle/) I was concerned they wouldn't like that the app downloads so much data but it was approved without much question, they just wanted terms of service prominently displayed to end users

Re: Show HN: YouTube Full Text Search – Search all of a channel from the commandline

#149

Post by simonw »

It looks like you're running searches using LIKE: https://github.com/NotJoeMartinez/yt-fts/blob/050981c0519a96...

SQLite has a really power full-text search mechanism built in - FTS5. It can handle things like stemming and stop words and relevance ranking.

My sqlite-utils Python library includes helper methods for setting that up: https://sqlite-utils.datasette.io/en/stable/python-api.html#...

Re: Show HN: YouTube Full Text Search – Search all of a channel from the commandline

#150

Post by ramraj07 »

gniv wrote: post #87
ramraj07 wrote: post #62

Earlier quoted context omitted.

I’m pretty sure google uses the captions in search. As long as you search from within YouTube. I regularly search for keywords and find hourlong videos where it’s mentioned somewhere in the middle and nowhere in the description.

I still have doubts. Google uses anchors heavily in search, so it's hard to prove if a result is due to matches in captions or in a website linking to the video. Would be good to find proof one way or another.

FWIW I tried to find the norm MacDonald clip where he sings about the "impossible bat" and YouTube was spot on.
Post reply on HN