Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: YouTube Full Text Search – Search all of a channel from the commandline

github.com

Post reply on HN
Print view View original

151–160 of 172 posts

Re: Show HN: YouTube Full Text Search – Search all of a channel from the commandline

#151

Post by fzliu »

cced wrote: post #59
bigzyg33k wrote: post #37

Earlier quoted context omitted.

Here's a (kinda) ELI5: you would use a language model to create "embeddings" of the text, which you can think of as a set of numbers representing the "meaning" of a set of characters. These numbers can be plotted as points in a space, and embeddings of things with similar meanings are plotted close to each other. So things like "exam preparation" would have embeddings close to things like "top study tips". Say you ha…

Do you have any resources that might guide one on doing something like this from scratch?

Here's one that may be helpful for you: https://zilliz.com/blog/using-vector-database-search-white-h...

Re: Show HN: YouTube Full Text Search – Search all of a channel from the commandline

#152

Post by vram22 »

Apropos of the post: sometime back I had wondered if it would be possible to search videos for content by text keywords, but not to match text occurring in the video title, chapter titles or comments. Instead, if an app or library could somehow search for the given keywords by matching them with the spoken words in the video. That would be a potentially cool and useful feature.

I realize this may be technically impossible [1] or very difficult, but thought of mentioning the idea here.

[1] On further thought, speech recognition (as seen in mobile phones at least) has progressed to quite a good level (speaking as a layman for this part), so maybe the idea is not wholly infeasible. If an app or lib could somehow "internally" play the video, and speech-recognize the spoken words into text, then the problem would reduce to normal text pattern matching.

Putting the idea out here to see what devs think of it.

Re: Show HN: YouTube Full Text Search – Search all of a channel from the commandline

#153

Post by vram22 »

vram22 wrote: post #152

Apropos of the post: sometime back I had wondered if it would be possible to search videos for content by text keywords, but not to match text occurring in the video title, chapter titles or comments. Instead, if an app or library could somehow search for the given keywords by matching them with the spoken words in the video. That would be a potentially cool and useful feature. I realize this may be technically impos…

>somehow "internally" play the video,

Analogous to how headless browsers are used for automated testing or scraping of web apps.

Re: Show HN: YouTube Full Text Search – Search all of a channel from the commandline

#154

Post by jimmySixDOF »

lessname wrote: post #11

What I really would like to see on youtube is a full text search on video content, at least for videos with subtitles.

I stumbled across a ShowHN that did not get to the front page but seems to fit here:

https://clipbase.xyz/

Re: Show HN: YouTube Full Text Search – Search all of a channel from the commandline

#155

Post by lfconsult »

simonw wrote: post #149

It looks like you're running searches using LIKE: https://github.com/NotJoeMartinez/yt-fts/blob/050981c0519a96... SQLite has a really power full-text search mechanism built in - FTS5. It can handle things like stemming and stop words and relevance ranking. My sqlite-utils Python library includes helper methods for setting that up: https://sqlite-utils.datasette.io/en/stable/python-api.html#...

You're right. Thanks for sharing the link to your full-text search helper, really neat.

Re: Show HN: YouTube Full Text Search – Search all of a channel from the commandline

#156

Post by bob_theslob646 »

StrangeATractor wrote: post #115
derefr wrote: post #3

I love that a third party is stepping up here, but it's incomprehensible to me that Google doesn't do this themselves. They're a search company, and they own YouTube. The YouTube data — including the subtitle files — is already sitting there on their servers; they don't have to scrape it, they just have to index it. What are they even doing? Fun thing to try: do a Google search with "site:youtube.com" in it. You get…

Have you tried searching through your call history on a Google phone? It's awful. You'd think they'd have a solid search built in but it's nearly useless. Especially considering you're usually searching for that number you only dialed once and isn't in your contacts and is for some reason excluded from your recents, so you go into your call history (strangely hidden behind a menu) and...there's no search function. WT…

I have this same exact problem. I thought I was crazy. The call history on Android/Pixel is absolutely terrible.

Re: Show HN: YouTube Full Text Search – Search all of a channel from the commandline

#157

Post by lfconsult »

lfconsult wrote: post #63
xurukefi wrote: post #25

$ python yt_fts.py download 'https://www.youtube.com/@ycombinator/videos' [...] File "/app/yt_fts.py", line 176, in get_channel_id channel_id = re.search('channelId":"(.{24})"', html).group(1) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'group' $ python yt_fts.py download 'https://www.youtube.com/@ycombinator/videos' --channel-id UCj089h5WsDdh1q8t54K3ZCw [...] File…

This tool itself does works great. This behavior is due to the YouTube cookies consent page. I opened an issue about this specific issue: https://github.com/NotJoeMartinez/yt-fts/issues/1 Glad if you want to help.

Fixed.

Re: Show HN: YouTube Full Text Search – Search all of a channel from the commandline

#158

Post by lfconsult »

lfconsult wrote: post #71
DyslexicAtheist wrote: post #48

python3 yt_fts.py download https://www.youtube.com/@PerspectiveArts/videos --channel-id UCUCN8V_pO0xOFKLL4XG1tshnw Downloading channel Saving vtt files to /tmp/user/1000/tmpm4xoskpo Traceback (most recent call last): File "/home/user/src/yt-fts/yt_fts.py", line 273, in cli() File "/home/user/src/yt-fts/.env/lib/python3.11/site-packages/click/core.py", line 1130, in __call__ return self.main(*args, \*kwargs) ^^^^^^^^^…

Probably due the YouTube cookies consent page. I opened an issue: https://github.com/NotJoeMartinez/yt-fts/issues/1 Hope it helps.

Fixed.

Re: Show HN: YouTube Full Text Search – Search all of a channel from the commandline

#159

Post by bob_theslob646 »

>yt-fts is a simple python script that uses yt-dlp to scrape all of a youtube channels subtitles and load them into an sqlite database that is searchable from the command line

"Scrape all of YouTube subtitles"

So if that data is not good...then how is this useful. The captions in YouTube have been pretty bad. Are these the same thing? I'll test it out.

Re: Show HN: YouTube Full Text Search – Search all of a channel from the commandline

#160

Post by dizhn »

gniv wrote: post #45
dizhn wrote: post #32

Earlier quoted context omitted.

Google doesn't even do a lot simpler things like searching by language or location. And the search is garbage. I am trying to learn Italian so that's what I am interested in but even when I enter a search term spelled correctly with its accents and everything I get anything from Brazilian Portuguese to French. They do a very helfpul translation of the term and return results that are unfortunately useless to me. (I w…

On web search, you can append &lr=lang_it to the URL. Maybe make it into a Chrome extension.

This does not seem to be doing anything. It's also being removed from the url bar when the page loads.
Post reply on HN