Earlier quoted context omitted.
Here's a (kinda) ELI5: you would use a language model to create "embeddings" of the text, which you can think of as a set of numbers representing the "meaning" of a set of characters. These numbers can be plotted as points in a space, and embeddings of things with similar meanings are plotted close to each other. So things like "exam preparation" would have embeddings close to things like "top study tips". Say you ha…
Do you have any resources that might guide one on doing something like this from scratch?
Show HN: YouTube Full Text Search – Search all of a channel from the commandline
101–110 of 172 posts
Re: Show HN: YouTube Full Text Search – Search all of a channel from the commandline
#102Earlier quoted context omitted.
Something like this? https://dexa.ai
https://weaviate.io/ Looks interesting. I was just reading about it.
If you're getting started with Weaviate, these two are probably what you need:
1. Wizard to create a docker-compose file: https://weaviate.io/developers/weaviate/installation/docker-... (e.g. choose the embedding model)
2. Sample notebook showing how to index items using the python library: https://github.com/weaviate-tutorials/vector-provision-optio...
Re: Show HN: YouTube Full Text Search – Search all of a channel from the commandline
#103Earlier quoted context omitted.
Here's a (kinda) ELI5: you would use a language model to create "embeddings" of the text, which you can think of as a set of numbers representing the "meaning" of a set of characters. These numbers can be plotted as points in a space, and embeddings of things with similar meanings are plotted close to each other. So things like "exam preparation" would have embeddings close to things like "top study tips". Say you ha…
Do you have any resources that might guide one on doing something like this from scratch?
Re: Show HN: YouTube Full Text Search – Search all of a channel from the commandline
#104I hate being the poo pooer who says that subtitles are available via the API and wish the tool went that route.
I'm all for stuff being archivable with tools like youtube-dl, but I much prefer to see tools like this use the API despite its quotas because it goes beyond archiving a copy for reference. Tools that (ab)use scraping only justify anti-scraping efforts that journalists and the like use and escalate that arms race. I think one could still scrape a channel or two per day within API usage limits [1]--50 units per list, 200 units per download; quota 10,000 units per day.
[1]: https://developers.google.com/youtube/v3/docs/captions/downl...
Re: Show HN: YouTube Full Text Search – Search all of a channel from the commandline
#105I love that a third party is stepping up here, but it's incomprehensible to me that Google doesn't do this themselves. They're a search company, and they own YouTube. The YouTube data — including the subtitle files — is already sitting there on their servers; they don't have to scrape it, they just have to index it. What are they even doing? Fun thing to try: do a Google search with "site:youtube.com" in it. You get…
As far as I can tell, Google (but not YouTube?) does search YouTube transcripts.
I have successfully Googled text in a video's transcript and found that video.
The transcripts themselves are pretty bad though (Google's using old tech).
They're usually good enough for auto-summarization though.
Re: Show HN: YouTube Full Text Search – Search all of a channel from the commandline
#106I love that a third party is stepping up here, but it's incomprehensible to me that Google doesn't do this themselves. They're a search company, and they own YouTube. The YouTube data — including the subtitle files — is already sitting there on their servers; they don't have to scrape it, they just have to index it. What are they even doing? Fun thing to try: do a Google search with "site:youtube.com" in it. You get…
Google does it, Youtube does not.
Google will find a video when you search for a phrase that was said in it (as long as its bad speech recognition got it right), it will find a video with a text that shows up on an object with enough clarity to OCR (for example electronic component name on a pcb in the foreground). There is one plot twist - Google will not always do it when you search for VIDEO specifically :) but will gladly give you videos when searching text/images :)
Youtube search on the other hand will try
- suggest something you liked that has nothing to do with the search term entered
- popular videos at this moment
- videos mildly related to proper results. One of them had a horse in it? clearly you want more horses!
- videos with title mildly related to search term
- to ignore upload date filter when they panic (Christchurch mosque massacre).
For example YT search for "Si5351A" limited to this month will give 11 somewhat proper results mostly with "Si5351" (no A postfix) in title/description AND some dude DXing in Indonesian "Menerima Modif Radio Yaesu FTC 1540A Ke DDS System" because "Si5351A" is a "DDS" so its the same thing right? Its like when Im looking for "NSR Ro80" you should show me plenty of other cars because Ro80 is a car :). Searching for Si5351A without quote marks will show one additional video with Si5351B in the title.
Gets better, searching Google Video for Si5351A last month also gives ~11 results, but only 4 of those are direct YT links :]
Re: Show HN: YouTube Full Text Search – Search all of a channel from the commandline
#107I love that a third party is stepping up here, but it's incomprehensible to me that Google doesn't do this themselves. They're a search company, and they own YouTube. The YouTube data — including the subtitle files — is already sitting there on their servers; they don't have to scrape it, they just have to index it. What are they even doing? Fun thing to try: do a Google search with "site:youtube.com" in it. You get…
It's quite intriguing that Google doesn't offer full-text search capabilities for YouTube, considering its position as a leading search company. However, I think there are several reasons for this, some of which may not be immediately apparent.
Firstly, if Google did offer this feature, it would likely be targeted by Search Engine Optimization (SEO) exploits. In essence, any time a new search parameter is introduced, there's a risk of it being manipulated to prioritize certain content—especially by those interested in gaming the system for increased visibility or monetary gain. If YouTube's search feature were to be plagued by such spamming, it could severely degrade the user experience and lead to Google having to strip it away. While not a guarantee, it's a probable outcome given the history of SEO misuse.
Secondly, YouTube's primary focus is on its recommendation algorithm rather than search functionality. With billions of videos hosted, the key goal is to keep users engaged by serving up content they're likely to enjoy, thereby increasing view times and ad revenue. The search feature, while useful, is not as integral to this objective. Further, offering full-text search could provide yet another avenue to manipulate the algorithm, which YouTube surely wants to avoid.
Finally, implementing and maintaining such a feature would require substantial resources. It would necessitate hiring teams of high-salaried employees to moderate and ensure fair use of the feature, adding considerable operational costs. Considering these factors, it seems that Google has made a strategic decision to avoid this feature for now.
That said, the fact that third-party solutions are emerging, such as the one shared here, shows that there's a demand for full-text search capabilities. It also underscores the potential that these solutions have when unencumbered by the constraints faced by a tech giant like Google. This provides a fascinating insight into the dynamic relationship between third-party developers and tech corporations and the way they can complement each other.
Re: Show HN: YouTube Full Text Search – Search all of a channel from the commandline
#108Earlier quoted context omitted.
Actually, letting people search video text would enable less watching and that is probably the reason they aren't interested in it.
Exactly! Google is in the attention business, the more you stay the more they make
So they say, attention is all you need.
Re: Show HN: YouTube Full Text Search – Search all of a channel from the commandline
#109I love that a third party is stepping up here, but it's incomprehensible to me that Google doesn't do this themselves. They're a search company, and they own YouTube. The YouTube data — including the subtitle files — is already sitting there on their servers; they don't have to scrape it, they just have to index it. What are they even doing? Fun thing to try: do a Google search with "site:youtube.com" in it. You get…
I would posit that google has determined that it’s more profitable to keep viewers on YouTube via controlling the viewing experience vs whatever additional ad revenue they’d gain by making videos more easily indexable. I’m basing this hypothesis on YouTube’s consistent trend of removing features related to controlling your own viewing experience. For example, removing subscription collections.
I’m willing to bet that the features you’ve seen removed are taken out because their utilization is low, and there’s some associated maintenance cost. I’d also like to know what other features were removed. I had never heard of subscription collections, for example, and a search returned [this video](https://youtu.be/qGSHPhR8k8g) (I wish markdown worked on here) that says collections were a test feature (I’m guessing it didn’t do well enough to make it to prod).
Re: Show HN: YouTube Full Text Search – Search all of a channel from the commandline
#110Earlier quoted context omitted.
I don’t understand, can you explain more?
I mean this: "Tell HN: YouTube's web UI just got even worse" - https://news.ycombinator.com/item?id=33371268 They used to have a drop down box, for oldest or most recent, when you clicked on the tab Videos for a YouTube channel. Now you will notice, it has now a "Latest" and "Popular", something that of course the algorithm will decide upon...
We seem to be locked into an ever more blatant "cycle of addiction". As with (other) drugs, the addict and the dealer are something like "two sides of the same coin".
Briefly, new tech / markets / apps ... i.e., 'innovation' ... often 'creates' tremendous 'value'*. Money (aka capital) rushes in - there's a boom, everyone's excited: consumers are getting new products, services, conveniences, improved prices; engineers, technologists, scientists, machinists, etc. are excited - building the new tech, exploring capabilities, using it to advance research, improving precision / throughout / etc. Dopamine hits all around, vibrant 'ecosystem', everything's lively, feelings of 'changing the world', etc.
But, then, the space reaches a comparative plateau. The low-hanging fruit is exhausted. The lucky/skilled'VCs' have made their 50x+ returns (that far more than cover the various failed ventures), the technologist innovators are moving into some other space, etc.
What's left are basically the 'landlords' and their crew. There's nothing inherently wrong with that. Those are fine 'occupations' with essential (enough) functions etc. The problem is, there are pressures, with our current culture, towards unlimited growth. There's this need ... this addiction to / normalization of 'neverending exponential curves'. So, you get rent-seeking, forms of 'maximum tolerable extraction' - a sort of exploitative feudalism... Locking up of all the resources now developed in the hands of a comparative few.
There's something of a spectrum, when it comes to economic sectors / businesses, as to how naturally monopolization forms. In general, unchecked, just about any sector / business area can fairly easily trend towards one 'company' controlling everything ... there's more that could be written here, but I'm out of time right now ... [this part should probably be edited / made into a more suitable transition]
Ultimately, every time a given market reaches a certain degree of concentration, the businesses left, particularly these days, increasingly act like "Polyergus" ants, or possibly Cuckoos, or other parasites of various types - organisms that use some degree of parasitic strategies. They 'farm' the rest of us for ever-increasing amounts of 'value'. But, not like responsible farmers, more like the human species as described by A. Smith in The Matrix - "Every mammal on this planet instinctively develops a natural equilibrium with the surrounding environment but you humans do not. You move to an area and you multiply and multiply until every natural resource is consumed and the only way you can survive is to spread to another area."
That pattern is woven throughout this way overly long comment (that also desperately needs editing) ... sadly, gotta run now.
* Some of these words / phrases are so clichéd / co-opted, it's hard not to put quotes of some type around them - partly calling attention to that aspect, partly calling attention to more fundamental meaning