Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Search inside YouTube videos using natural language queries

github.com

Post reply on HN
Print view View original

31–40 of 45 posts

Re: Show HN: Search inside YouTube videos using natural language queries

#31

Post by vladoh »

Abishek_Muthian wrote: post #28
vladoh wrote: post #2

This is a demonstration of the abilities of OpenAI's CLIP neural network. The tool will download a YouTube video, extract frames at regular intervals and precompute the feature vectors of each frame using CLIP. You can then use natural language search queries to find a particular frame of the video. The results are really amazing in my opinion... If you want to experiment with it yourself, I prepared a Colab notebook…

Excellent work. If it could take image set as input, then perhaps we can use this to identify our self in a random Internet video e.g. Lengthy tourist video in which you suspect you could have been covered as you were there at that place on that day. There are people already looking for such solution(I've added the link to that discussion on my profile).

I think this is a valuable application, but I don't think CLIP is well suited for it. The power of CLIP comes from training a model to jointly "understand" text and images. If you are looking at identifying a particular person there are more suitable designs for face recognition.

Re: Show HN: Search inside YouTube videos using natural language queries

#32

Post by vladoh »

woko wrote: post #26
vladoh wrote: post #25

Earlier quoted context omitted.

Thanks for sharing! :) Yes, the feature computation on the images has to be sone only once and the representation is very efficient - 512 float16 values per image.

Yes, I know. :D Your previous project with Unsplash made me try a similar approach [1] for banners of video games on Steam. [1] https://github.com/woctezuma/steam-image-search

Cool application! I wasn't aware that there are so many images available on Steam...

Re: Show HN: Search inside YouTube videos using natural language queries

#33

Post by vladoh »

gbennett71 wrote: post #27

Downloading YouTube videos is a Terms of Use violation, no?

I haven't read the Terms of Use in details, but I guess it depends on what you do with the data. I actually don't store, distribute or sell the videos, so I hope it is OK in the scope of such personal project :)

Re: Show HN: Search inside YouTube videos using natural language queries

#34

Post by vladoh »

woko wrote: post #29
suyash wrote: post #18

The problem still exists that you have to provide it the YouTube video to search within, would be nice if there was a tool to search across all of YouTube.

Which would require an easy access to all of the videos, which only Googe/Youtube itself has. Many nice things could be done, but the platform (or the data owner) has all the power in its hands.

Yeah, indexing all YouTube videos is surely possible, but out of the scope of a personal project :)

Re: Show HN: Search inside YouTube videos using natural language queries

#36

Post by mncharity »

Years ago, when I prototyped an orders-of-magnitude physical-properties playground web app, I found the development bottleneck was searching for video clips and images. For example, find clips showing the heartbeat of a {goldfish,mouse,cat,dog,child,adult,horse,elephant,whale}, for a association (metabolic rate scales with body mass). Jiggling an oom kg scale, maybe you're shown a cat or mouse, and maybe a mouse heart patter, and whale's slow swish. Providing a massive (sorry) hidden curriculum. They all exist on youtube, and might be fair use for OER content, but finding them was not plausible. And still isn't, even with commercial use of stock videos.

In the 1950's, the first Powers of Ten zoom book was hand drawn from books over years. Around 1980, a PoT film and book could use photos, but still good people made mistakes. Now creating a PoT zoom book can be homework. A video a school project. An XR a professional project. Technology, media search, acquisition, and handling costs, throttles science education content.

Necessary but not sufficient, of course. The first book might have been imagined and created earlier, but wasn't. 1950's astronomy textbooks needn't have had the color of the Sun wrong, and the same now in 2020. Though, anticipated difficulty of creation does throttle imagination...

If OP search were deployed on youtube, and fair use in its current form was allowed to survive, providing a historical step-change in the abundance and accessibility of reusable content, how might you imagine using that?

Re: Show HN: Search inside YouTube videos using natural language queries

#37

Post by jameshart »

I’m curious: why hasn’t google done this already?

Seems like a key part of ‘organizing all the world’s information’, right? Making YouTube searchable would open up their massive content library to discovery, and enable people to find content without having to rely on The Algorithm guessing what they might want to see...

In general the switch from users going to the internet and asking it to find them certain things, to going to the internet and having things selected for them and offered up as ‘recommendations’ seems like a key shift in how the internet functions to disseminate ideas...

Re: Show HN: Search inside YouTube videos using natural language queries

#38

Post by woko »

vladoh wrote: post #32
woko wrote: post #26

Earlier quoted context omitted.

Yes, I know. :D Your previous project with Unsplash made me try a similar approach [1] for banners of video games on Steam. [1] https://github.com/woctezuma/steam-image-search

Cool application! I wasn't aware that there are so many images available on Steam...

Yes, Steam has grown a lot. Last I checked, there were ~50k apps.

Edit: 50,630 apps according to https://www.gamedatacrunch.com/

As I focused on vertical banners, the list was smaller (~30k apps). This is equivalent to the *lite* version of Unsplash's dataset.

Re: Show HN: Search inside YouTube videos using natural language queries

#39

Post by vladoh »

jameshart wrote: post #37

I’m curious: why hasn’t google done this already? Seems like a key part of ‘organizing all the world’s information’, right? Making YouTube searchable would open up their massive content library to discovery, and enable people to find content without having to rely on The Algorithm guessing what they might want to see... In general the switch from users going to the internet and asking it to find them certain things,…

CLIP is quite a new model published by OpenAI in January. Their work is novel and pushed the state-of-the-art in this area by a lot. I'm sure that Google is also working on similar applications.

And I think they already have something similar. Recently, I've seen search results on Google that point you to a specific time of a YouTube video...

Post reply on HN