Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: YouTube Full Text Search – Search all of a channel from the commandline

github.com

Post reply on HN
Print view View original

21–30 of 172 posts

Re: Show HN: YouTube Full Text Search – Search all of a channel from the commandline

#21

Post by sharess »

derefr wrote: post #3

I love that a third party is stepping up here, but it's incomprehensible to me that Google doesn't do this themselves. They're a search company, and they own YouTube. The YouTube data — including the subtitle files — is already sitting there on their servers; they don't have to scrape it, they just have to index it. What are they even doing? Fun thing to try: do a Google search with "site:youtube.com" in it. You get…

Social media companies are in a constant tug-of-war against the end users when it comes to controlling what the users see. The ideal is that the user has absolutely no control over on what they see and the social media company can fully dictate content. That is what makes the money.

Allowing users to freely query content in their own websites is completely antithetical to what they are trying to do. YouTube is also very aggressive in preventing scraping and limiting the usage of the official API. Which is quite ironic considering the history of the company.

Re: Show HN: YouTube Full Text Search – Search all of a channel from the commandline

#22

Post by ranting-moth »

derefr wrote: post #3

I love that a third party is stepping up here, but it's incomprehensible to me that Google doesn't do this themselves. They're a search company, and they own YouTube. The YouTube data — including the subtitle files — is already sitting there on their servers; they don't have to scrape it, they just have to index it. What are they even doing? Fun thing to try: do a Google search with "site:youtube.com" in it. You get…

Conflict of interests. They want you faffing around provisioning all those valuable clicks for them to sell.

Re: Show HN: YouTube Full Text Search – Search all of a channel from the commandline

#24

Post by monkeydust »

Reflecticon wrote: post #18
monkeydust wrote: post #17

Nice work. You could encode the text, load this into a vector database and allow semantic search.

Something like this? https://dexa.ai

Yes in theory although they are pretty expensive. I am doing something like this at work as I wanted to unlock the wealth of information we have in our tutorials, webinars etc.

Re: Show HN: YouTube Full Text Search – Search all of a channel from the commandline

#25

Post by xurukefi » 

    $ python yt_fts.py download 'https://www.youtube.com/@ycombinator/videos'
    [...]
      File "/app/yt_fts.py", line 176, in get_channel_id
        channel_id = re.search('channelId":"(.{24})"', html).group(1)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    AttributeError: 'NoneType' object has no attribute 'group'

    $ python yt_fts.py download 'https://www.youtube.com/@ycombinator/videos' --channel-id UCj089h5WsDdh1q8t54K3ZCw
    [...]
      File "/app/yt_fts.py", line 191, in get_channel_name
        data = json.loads(script.string)
                          ^^^^^^^^^^^^^
    AttributeError: 'NoneType' object has no attribute 'string'
works great

Re: Show HN: YouTube Full Text Search – Search all of a channel from the commandline

#26

Post by gniv »

derefr wrote: post #3

I love that a third party is stepping up here, but it's incomprehensible to me that Google doesn't do this themselves. They're a search company, and they own YouTube. The YouTube data — including the subtitle files — is already sitting there on their servers; they don't have to scrape it, they just have to index it. What are they even doing? Fun thing to try: do a Google search with "site:youtube.com" in it. You get…

Beyond conspiracy theories it's interesting to speculate why Google is not providing native search-in-subtitles and search-in-comments. The easiest explanation is that they don't trust the quality. They probably tried it and reached the conclusion that it doesn't improve search in any meaningful direction.

I know from experience that search in user reviews is very hard. Unless you really understand the review (which was tried via sentiment analysis) you cannot rank results well. But now with the new LLM models I think it would work better.

Re: Show HN: YouTube Full Text Search – Search all of a channel from the commandline

#28

Post by pknerd »

monkeydust wrote: post #17

Nice work. You could encode the text, load this into a vector database and allow semantic search.

Pardon my ignorance as I have not worked on Vector DBs yet, could you come up with an example how it'd be different than a full text search?

Re: Show HN: YouTube Full Text Search – Search all of a channel from the commandline

#30

Post by nomilk »

> yt-fts is a simple python script that uses yt-dlp to scrape all of a youtube channels subtitles and load them into an sqlite database that is searchable from the command line.

Critically, this is per channel. I wonder if we can optionally configure this to share the downloaded transcripts to a central repository so eventually a good proportion of youtube's transcripts could be downloaded as one big text file.

Post reply on HN