Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Search inside YouTube videos using natural language queries

github.com

Post reply on HN
Print view View original

21–30 of 45 posts

Re: Show HN: Search inside YouTube videos using natural language queries

#21

Post by vladoh »

Crazyontap wrote: post #20
vladoh wrote: post #2

This is a demonstration of the abilities of OpenAI's CLIP neural network. The tool will download a YouTube video, extract frames at regular intervals and precompute the feature vectors of each frame using CLIP. You can then use natural language search queries to find a particular frame of the video. The results are really amazing in my opinion... If you want to experiment with it yourself, I prepared a Colab notebook…

Can it work for more advance keywords like say, "traffic violation" where it spots a car jumping red light or pedestrian not using a crosswalk, etc? It could be very useful to help with law enforcement.

I think it can. However, you will likely need a bigger model. Currently, OpenAI shares only their small model and I hope they will soon release bigger ones!

Re: Show HN: Search inside YouTube videos using natural language queries

#22

Post by vladoh »

ramraj07 wrote: post #19
vladoh wrote: post #2

This is a demonstration of the abilities of OpenAI's CLIP neural network. The tool will download a YouTube video, extract frames at regular intervals and precompute the feature vectors of each frame using CLIP. You can then use natural language search queries to find a particular frame of the video. The results are really amazing in my opinion... If you want to experiment with it yourself, I prepared a Colab notebook…

This is amazing. I'm going to get this running on my Dropbox. Curious what it gets out of scanned documents as well.

There is one caveat to be aware of - the image is cropped to a square in the center and scaled down to 224x224. So small details will be lost, for example if you want to run it on scanned documents. Photos work great though.

I tried it on the 2M photos from the Unsplash dataset: https://github.com/haltakov/natural-language-image-search

Re: Show HN: Search inside YouTube videos using natural language queries

#23

Post by woko »

mandeepj wrote: post #12
vladoh wrote: post #2

This is a demonstration of the abilities of OpenAI's CLIP neural network. The tool will download a YouTube video, extract frames at regular intervals and precompute the feature vectors of each frame using CLIP. You can then use natural language search queries to find a particular frame of the video. The results are really amazing in my opinion... If you want to experiment with it yourself, I prepared a Colab notebook…

> The tool will download a YouTube video, extract frames at regular intervals That should be able to scale well :-)

For info, the same tool works well with 2 million images found in the Unsplash dataset [1]. Features only have to be computed once for the dataset, and only the feature vector for the user query has to be computed on the fly. Then matching features can be done in a manner that scales well.

So, the present tool does not scale because the videos are part of the user query, but a company with an easy access to the videos and the computational power to pre-encode the frames as features could create a search engine based on CLIP.

[1] https://github.com/haltakov/natural-language-image-search

Re: Show HN: Search inside YouTube videos using natural language queries

#24

Post by vladoh »

canada_dry wrote: post #16
vladoh wrote: post #2

This is a demonstration of the abilities of OpenAI's CLIP neural network. The tool will download a YouTube video, extract frames at regular intervals and precompute the feature vectors of each frame using CLIP. You can then use natural language search queries to find a particular frame of the video. The results are really amazing in my opinion... If you want to experiment with it yourself, I prepared a Colab notebook…

Great demo. Wondering whether it would be more efficient if extracting frames where the content has changed (e.g. over a threshold and/or all I-frames)? Also, could this be used to identify event type in videos? I'd love to run my 25 years of home videos through this an have it annotate: "Christmas, birthday, park, camping...".

Yes, this is definitely possible. You can maybe try computing some kind of image distance between frames or some keyframe extraction.

Once you compute the features, the search is very efficient! I tried it for searching in the 2M photos dataset from Unsplash and it takes like 2-3 seconds: https://github.com/haltakov/natural-language-image-search

I plan to run my personal photos through it :)

Re: Show HN: Search inside YouTube videos using natural language queries

#25

Post by vladoh »

woko wrote: post #23
mandeepj wrote: post #12

Earlier quoted context omitted.

> The tool will download a YouTube video, extract frames at regular intervals That should be able to scale well :-)

For info, the same tool works well with 2 million images found in the Unsplash dataset [1]. Features only have to be computed once for the dataset, and only the feature vector for the user query has to be computed on the fly. Then matching features can be done in a manner that scales well. So, the present tool does not scale because the videos are part of the user query, but a company with an easy access to the video…

Thanks for sharing! :)

Yes, the feature computation on the images has to be sone only once and the representation is very efficient - 512 float16 values per image.

Re: Show HN: Search inside YouTube videos using natural language queries

#26

Post by woko »

vladoh wrote: post #25
woko wrote: post #23

Earlier quoted context omitted.

For info, the same tool works well with 2 million images found in the Unsplash dataset [1]. Features only have to be computed once for the dataset, and only the feature vector for the user query has to be computed on the fly. Then matching features can be done in a manner that scales well. So, the present tool does not scale because the videos are part of the user query, but a company with an easy access to the video…

Thanks for sharing! :) Yes, the feature computation on the images has to be sone only once and the representation is very efficient - 512 float16 values per image.

Yes, I know. :D Your previous project with Unsplash made me try a similar approach [1] for banners of video games on Steam.

[1] https://github.com/woctezuma/steam-image-search

Re: Show HN: Search inside YouTube videos using natural language queries

#28

Post by Abishek_Muthian »

vladoh wrote: post #2

This is a demonstration of the abilities of OpenAI's CLIP neural network. The tool will download a YouTube video, extract frames at regular intervals and precompute the feature vectors of each frame using CLIP. You can then use natural language search queries to find a particular frame of the video. The results are really amazing in my opinion... If you want to experiment with it yourself, I prepared a Colab notebook…

Excellent work.

If it could take image set as input, then perhaps we can use this to identify our self in a random Internet video e.g. Lengthy tourist video in which you suspect you could have been covered as you were there at that place on that day.

There are people already looking for such solution(I've added the link to that discussion on my profile).

Re: Show HN: Search inside YouTube videos using natural language queries

#29

Post by woko »

suyash wrote: post #18

The problem still exists that you have to provide it the YouTube video to search within, would be nice if there was a tool to search across all of YouTube.

Which would require an easy access to all of the videos, which only Googe/Youtube itself has.

Many nice things could be done, but the platform (or the data owner) has all the power in its hands.

Post reply on HN