Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: YouTube Full Text Search – Search all of a channel from the commandline

github.com

Post reply on HN
Print view View original

41–50 of 172 posts

Re: Show HN: YouTube Full Text Search – Search all of a channel from the commandline

#41

Post by derefr »

hysan wrote: post #12
derefr wrote: post #3

I love that a third party is stepping up here, but it's incomprehensible to me that Google doesn't do this themselves. They're a search company, and they own YouTube. The YouTube data — including the subtitle files — is already sitting there on their servers; they don't have to scrape it, they just have to index it. What are they even doing? Fun thing to try: do a Google search with "site:youtube.com" in it. You get…

I would posit that google has determined that it’s more profitable to keep viewers on YouTube via controlling the viewing experience vs whatever additional ad revenue they’d gain by making videos more easily indexable. I’m basing this hypothesis on YouTube’s consistent trend of removing features related to controlling your own viewing experience. For example, removing subscription collections.

How would making videos more indexible move people off of YouTube? Once you're there, you'd stay there, with all the current recommender algorithms still in effect; all the external indexability would do is give Google (and Bing, and everyone else) more reasons to lead you into that labyrinth.

Re: Show HN: YouTube Full Text Search – Search all of a channel from the commandline

#42

Post by dexterdog »

_aaed wrote: post #39

Perhaps I'm wrong but how is this full text search? It's just using the LIKE operator

That is full text search. It just doesn't maintain an index because it's a small enough dataset per channel that it can brute force it in memory.

Re: Show HN: YouTube Full Text Search – Search all of a channel from the commandline

#43

Post by derefr »

Pazzaz wrote: post #16
derefr wrote: post #3

I love that a third party is stepping up here, but it's incomprehensible to me that Google doesn't do this themselves. They're a search company, and they own YouTube. The YouTube data — including the subtitle files — is already sitting there on their servers; they don't have to scrape it, they just have to index it. What are they even doing? Fun thing to try: do a Google search with "site:youtube.com" in it. You get…

I think you're exaggerating a little when you say "site:youtube.com" doesn't work. If I search 'site:youtube.com apple watch' I get 143 000 000 results, and if I search something more specific like 'site:youtube.com "Featuring Dr James Grime"' I'll find exactly what I'm looking for. But you're correct that it doesn't seem to search video comments, only titles and descriptions.

The problem I ran into is that, like I said, YouTube doesn't seem to get indexed by hypertext inward-edges from other sites like a regular website does — and so you can't search by how you recall a video being described in pages that link to it; instead, you have to remember how the video describes itself. Which it may not always do well.

Re: Show HN: YouTube Full Text Search – Search all of a channel from the commandline

#45

Post by gniv »

dizhn wrote: post #32
derefr wrote: post #3

I love that a third party is stepping up here, but it's incomprehensible to me that Google doesn't do this themselves. They're a search company, and they own YouTube. The YouTube data — including the subtitle files — is already sitting there on their servers; they don't have to scrape it, they just have to index it. What are they even doing? Fun thing to try: do a Google search with "site:youtube.com" in it. You get…

Google doesn't even do a lot simpler things like searching by language or location. And the search is garbage. I am trying to learn Italian so that's what I am interested in but even when I enter a search term spelled correctly with its accents and everything I get anything from Brazilian Portuguese to French. They do a very helfpul translation of the term and return results that are unfortunately useless to me. (I w…

On web search, you can append &lr=lang_it to the URL. Maybe make it into a Chrome extension.

Re: Show HN: YouTube Full Text Search – Search all of a channel from the commandline

#48

Post by DyslexicAtheist » 

   python3 yt_fts.py download https://www.youtube.com/@PerspectiveArts/videos --channel-id UCUCN8V_pO0xOFKLL4XG1tshnw
   Downloading channel
   Saving vtt files to /tmp/user/1000/tmpm4xoskpo
   Traceback (most recent call last):
   File "/home/user/src/yt-fts/yt_fts.py", line 273, in 
     cli()
   File "/home/user/src/yt-fts/.env/lib/python3.11/site-packages/click/core.py", line 1130, in __call__
     return self.main(*args, \*kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/home/user/src/yt-fts/.env/lib/python3.11/site-packages/click/core.py", line 1055, in main
     rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
   File "/home/user/src/yt-fts/.env/lib/python3.11/site-packages/click/core.py", line 1657, in invoke
     return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/home/user/src/yt-fts/.env/lib/python3.11/site-packages/click/core.py", line 1404, in invoke
     return ctx.invoke(self.callback, \*ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/home/user/src/yt-fts/.env/lib/python3.11/site-packages/click/core.py", line 760, in invoke
     return __callback(*args, \*kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/home/user/src/yt-fts/yt_fts.py", line 31, in download
     download_channel(channel_id)
   File "/home/user/src/yt-fts/yt_fts.py", line 82, in download_channel
     channel_name = get_channel_name(channel_id)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/home/user/src/yt-fts/yt_fts.py", line 191, in get_channel_name
     data = json.loads(script.string)
                      ^^^^^^^^^^^^^
   AttributeError: 'NoneType' object has no attribute 'string'

Re: Show HN: YouTube Full Text Search – Search all of a channel from the commandline

#49

Post by dmbche »

derefr wrote: post #41
hysan wrote: post #12

Earlier quoted context omitted.

I would posit that google has determined that it’s more profitable to keep viewers on YouTube via controlling the viewing experience vs whatever additional ad revenue they’d gain by making videos more easily indexable. I’m basing this hypothesis on YouTube’s consistent trend of removing features related to controlling your own viewing experience. For example, removing subscription collections.

How would making videos more indexible move people off of YouTube? Once you're there, you'd stay there, with all the current recommender algorithms still in effect; all the external indexability would do is give Google (and Bing, and everyone else) more reasons to lead you into that labyrinth.

If you get information by reading text on your terminal you didn't see the ads - I think the thinking stops there.

Re: Show HN: YouTube Full Text Search – Search all of a channel from the commandline

#50

Post by pyinstallwoes »

Reflecticon wrote: post #18
monkeydust wrote: post #17

Nice work. You could encode the text, load this into a vector database and allow semantic search.

Something like this? https://dexa.ai

https://weaviate.io/ Looks interesting. I was just reading about it.
Post reply on HN