Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: YouTube Full Text Search – Search all of a channel from the commandline

github.com

Post reply on HN
Print view View original

1–10 of 172 posts

Show HN: YouTube Full Text Search – Search all of a channel from the commandline

#1

Post by notjoemartinez »

yt-fts is a simple python script that uses yt-dlp to scrape all of a youtube channels subtitles and load them into an sqlite database that is searchable from the command line. It allows you to query a channel for specific key word or phrase and will generate time stamped youtube urls to the video containing the keyword.

Show HN: YouTube Full Text Search – Search all of a channel from the commandline
github.com

Re: Show HN: YouTube Full Text Search – Search all of a channel from the commandline

#3

Post by derefr »

I love that a third party is stepping up here, but it's incomprehensible to me that Google doesn't do this themselves. They're a search company, and they own YouTube. The YouTube data — including the subtitle files — is already sitting there on their servers; they don't have to scrape it, they just have to index it. What are they even doing?

Fun thing to try: do a Google search with "site:youtube.com" in it. You get basically nothing, no matter what keywords you use. It seems that Google actually entirely ignores/excludes YouTube from their regular HTML indexing, and instead only relies on the YouTube backend to actively push content into (a special, separate part of) the search index. Which gets you "results from YouTube" and "video search" — but doesn't get you the ability to search youtube videos pages qua web pages. (Consider: you can find a post in a Reddit comment thread on Google. Can you find a post in a YouTube video comments section on Google?)

Heck, when I first heard about YouTube's autogenerated captions, my first thought was "oh, so this is Google building deep indexing of video through audio transcription, because they can't trust externally-provided subtitles, right?" But it's been 10 years, and I couldn't have been more wrong.

Re: Show HN: YouTube Full Text Search – Search all of a channel from the commandline

#4

Post by stingraycharles »

derefr wrote: post #3

I love that a third party is stepping up here, but it's incomprehensible to me that Google doesn't do this themselves. They're a search company, and they own YouTube. The YouTube data — including the subtitle files — is already sitting there on their servers; they don't have to scrape it, they just have to index it. What are they even doing? Fun thing to try: do a Google search with "site:youtube.com" in it. You get…

There’s a company founded by a friend of mine called MediaDistillery, who are doing awesome stuff in this area. Real-time searchable massive video archives, with contextual understanding (e.g. “a WW2 fragment with a Jewish mother holding her baby”). Super useful for so many purposes.

And then there’s YouTube where you can’t even search subtitles. It makes me shake my head. Google seems to be at the forefront of AI, but doesn’t seem to be able to turn all that expertise into relevant products. Maybe the recent disruptions will shake them awake?

Re: Show HN: YouTube Full Text Search – Search all of a channel from the commandline

#5

Post by mrazomor »

derefr wrote: post #3

I love that a third party is stepping up here, but it's incomprehensible to me that Google doesn't do this themselves. They're a search company, and they own YouTube. The YouTube data — including the subtitle files — is already sitting there on their servers; they don't have to scrape it, they just have to index it. What are they even doing? Fun thing to try: do a Google search with "site:youtube.com" in it. You get…

Probably because YouTube =! Google Search, while YouTube is still a subset of Google. So, going an extra mile for YouTube and not for others might put Google Search in anti competition issues.

Then again, I also find it absurd. YouTube is one of the most valuable parts of the Internet. And its lack of searchability is criminal. At least the YT search itself should make up for it. It's shame it doesn't.

Re: Show HN: YouTube Full Text Search – Search all of a channel from the commandline

#7

Post by choppaface »

derefr wrote: post #3

I love that a third party is stepping up here, but it's incomprehensible to me that Google doesn't do this themselves. They're a search company, and they own YouTube. The YouTube data — including the subtitle files — is already sitting there on their servers; they don't have to scrape it, they just have to index it. What are they even doing? Fun thing to try: do a Google search with "site:youtube.com" in it. You get…

They do have search in video but the launch was kinda miffed

https://www.socialmediatoday.com/news/google-tests-text-sear...

Re: Show HN: YouTube Full Text Search – Search all of a channel from the commandline

#9

Post by derefr »

mrazomor wrote: post #5
derefr wrote: post #3

I love that a third party is stepping up here, but it's incomprehensible to me that Google doesn't do this themselves. They're a search company, and they own YouTube. The YouTube data — including the subtitle files — is already sitting there on their servers; they don't have to scrape it, they just have to index it. What are they even doing? Fun thing to try: do a Google search with "site:youtube.com" in it. You get…

Probably because YouTube =! Google Search, while YouTube is still a subset of Google. So, going an extra mile for YouTube and not for others might put Google Search in anti competition issues. Then again, I also find it absurd. YouTube is one of the most valuable parts of the Internet. And its lack of searchability is criminal. At least the YT search itself should make up for it. It's shame it doesn't.

Google doesn't necessarily have to do anything special for YouTube, though. Google could "just" index YouTube videos as if they were any other web pages, in a standard way. It would then be YouTube's job, to make the data inside those video pages legible to Google's indexer. Where Google could enable this, by pushing for web standards to increase machine-legibility of video in HTML — e.g. standardized ARIA-accessible captions sources for the element, etc.

If they got it set up such that in theory any web spider could come along and index a YouTube video — then there would be no anti-trust reason that Google couldn't just directly ingest the subtitle files off their own servers; it'd just be a bandwidth-saving optimization over the scraping process that they could otherwise do.

Re: Show HN: YouTube Full Text Search – Search all of a channel from the commandline

#10

Post by gorgoiler »

Very nice! FYI: sqlite ships with a full text search engine featuring a Boolean query language, highlight(), snippet() and scoring:

https://www.sqlite.org/fts5.html

I’ve not used it with enough content to know how much faster it is than LIKE ‘%my query%’ but it should be a lot quicker.

(Also, in most cases you don’t need to create an id column — every table has one already in the form of rowid.)

Post reply on HN