Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Search inside YouTube videos using natural language queries

github.com

Post reply on HN
Print view View original

11–20 of 45 posts

Re: Show HN: Search inside YouTube videos using natural language queries

#11

Post by vladoh »

pininja wrote: post #5
vladoh wrote: post #2

This is a demonstration of the abilities of OpenAI's CLIP neural network. The tool will download a YouTube video, extract frames at regular intervals and precompute the feature vectors of each frame using CLIP. You can then use natural language search queries to find a particular frame of the video. The results are really amazing in my opinion... If you want to experiment with it yourself, I prepared a Colab notebook…

This is very cool! Does this produce a occurrence index by any chance? It would be neat to explore a word map of a video.

I pushed a small update and you will now see a heatmap displaying the score of the search query for each frame.

Re: Show HN: Search inside YouTube videos using natural language queries

#12

Post by mandeepj »

vladoh wrote: post #2

This is a demonstration of the abilities of OpenAI's CLIP neural network. The tool will download a YouTube video, extract frames at regular intervals and precompute the feature vectors of each frame using CLIP. You can then use natural language search queries to find a particular frame of the video. The results are really amazing in my opinion... If you want to experiment with it yourself, I prepared a Colab notebook…

> The tool will download a YouTube video, extract frames at regular intervals

That should be able to scale well :-)

Re: Show HN: Search inside YouTube videos using natural language queries

#14

Post by mockingbirdy »

vladoh wrote: post #2

This is a demonstration of the abilities of OpenAI's CLIP neural network. The tool will download a YouTube video, extract frames at regular intervals and precompute the feature vectors of each frame using CLIP. You can then use natural language search queries to find a particular frame of the video. The results are really amazing in my opinion... If you want to experiment with it yourself, I prepared a Colab notebook…

Awesome! I'm currently working on the exact same thing (but with OCR added). Thank you for releasing this.

Re: Show HN: Search inside YouTube videos using natural language queries

#15

Post by canada_dry »

mockingbirdy wrote: post #14
vladoh wrote: post #2

This is a demonstration of the abilities of OpenAI's CLIP neural network. The tool will download a YouTube video, extract frames at regular intervals and precompute the feature vectors of each frame using CLIP. You can then use natural language search queries to find a particular frame of the video. The results are really amazing in my opinion... If you want to experiment with it yourself, I prepared a Colab notebook…

Awesome! I'm currently working on the exact same thing (but with OCR added). Thank you for releasing this.

> the exact same thing (but with OCR added)

Hmmm... what does "with OCR added" mean? If there is text in the video (e.g. street sign) that it can also be searched??

Re: Show HN: Search inside YouTube videos using natural language queries

#16

Post by canada_dry »

vladoh wrote: post #2

This is a demonstration of the abilities of OpenAI's CLIP neural network. The tool will download a YouTube video, extract frames at regular intervals and precompute the feature vectors of each frame using CLIP. You can then use natural language search queries to find a particular frame of the video. The results are really amazing in my opinion... If you want to experiment with it yourself, I prepared a Colab notebook…

Great demo.

Wondering whether it would be more efficient if extracting frames where the content has changed (e.g. over a threshold and/or all I-frames)?

Also, could this be used to identify event type in videos? I'd love to run my 25 years of home videos through this an have it annotate: "Christmas, birthday, park, camping...".

Re: Show HN: Search inside YouTube videos using natural language queries

#17

Post by mockingbirdy »

canada_dry wrote: post #15
mockingbirdy wrote: post #14

Earlier quoted context omitted.

Awesome! I'm currently working on the exact same thing (but with OCR added). Thank you for releasing this.

> the exact same thing (but with OCR added) Hmmm... what does "with OCR added" mean? If there is text in the video (e.g. street sign) that it can also be searched??

No, that wouldn't work too well. It's for YouTubers who stream their desktop screens and I need to extract some information to automatically process it. The desktop streams always look very similar so I don't need advanced AI/neural nets to extract that.

Re: Show HN: Search inside YouTube videos using natural language queries

#19

Post by ramraj07 »

vladoh wrote: post #2

This is a demonstration of the abilities of OpenAI's CLIP neural network. The tool will download a YouTube video, extract frames at regular intervals and precompute the feature vectors of each frame using CLIP. You can then use natural language search queries to find a particular frame of the video. The results are really amazing in my opinion... If you want to experiment with it yourself, I prepared a Colab notebook…

This is amazing. I'm going to get this running on my Dropbox. Curious what it gets out of scanned documents as well.

Re: Show HN: Search inside YouTube videos using natural language queries

#20

Post by Crazyontap »

vladoh wrote: post #2

This is a demonstration of the abilities of OpenAI's CLIP neural network. The tool will download a YouTube video, extract frames at regular intervals and precompute the feature vectors of each frame using CLIP. You can then use natural language search queries to find a particular frame of the video. The results are really amazing in my opinion... If you want to experiment with it yourself, I prepared a Colab notebook…

Can it work for more advance keywords like say, "traffic violation" where it spots a car jumping red light or pedestrian not using a crosswalk, etc?

It could be very useful to help with law enforcement.

Post reply on HN