Live data from Hacker News

Show HN: Gemini can now natively embed video, so I built sub-second video search

github.com

21–30 of 118 posts

Re: Show HN: Gemini can now natively embed video, so I built sub-second video search

#21
post #8

very cool, anybody have apparent use cases for this?

I think a good use case would be searching for certain products or videos across social media (TikTok and Instagram). especially useful for shopping, maybe

Branding/marketing monitoring companies would be all over this.

Re: Show HN: Gemini can now natively embed video, so I built sub-second video search

#22

> No transcription, no frame captioning, no intermediate text. If there is text on the video (like a caption or wtv), will the embedding capture that? Never thought about this before. If the video has audio, does the embedding capture that too?

Yes to both. The embedding is over raw video frames, so anything visible (text, signs, captions) gets captured in the vector. And Gemini Embedding 2 extracts the audio track and embeds it alongside the visual frames. So a query like 'someone yelling' would theoretically match on audio. My dashcam footage doesn't have audio though, so I haven't tested that side yet.

Re: Show HN: Gemini can now natively embed video, so I built sub-second video search

#23

Does anyone know of an open weights models that can embed video? Would love to experiment locally with this.

Not aware of any that do native video-to-vector embedding the way Gemini Embedding 2 does. There are CLIP-based models (like VideoCLIP) that embed frames individually, but they don't process temporal video. you'd need to average frame embeddings which loses a lot.

Would love to see open-weight models with this capability since it would eliminate the API cost and the privacy concern of uploading footage.

Re: Show HN: Gemini can now natively embed video, so I built sub-second video search

#24
post #11

Where is the Exit to this dystopia?

The Matrix style human pods: we live in blissful ignorance in the Matrix, while the LLMs extract more and more compute power from us so some CEO somewhere can claim they have now replaced all humans with machines in their business.

Ever get the feeling that the universe is reading your mind? Maybe there's some truth to that after all.

Re: Show HN: Gemini can now natively embed video, so I built sub-second video search

#27

What a brilliant idea! is this all done locally? That's incredible.

While the vector store is local, it is sending the data to Gemini's API for embedding. (Which if using a paid API key is probably fine for most use cases, no long term retention/training etc.)

Re: Show HN: Gemini can now natively embed video, so I built sub-second video search

#29
This is a really cool implementation—embeddings still often feel like magic to me. That said, this exact use case is sort of also my biggest point of concern with where AI takes us, much more so than most of the common AI risks you hear lots of chatter about. We live in a world absolutely loaded with cameras now but ultimately retain some semblance of semi-anonymity/privacy in public by virtue of the fact that nobody can actually watch or review all of the video from those cameras except when there is a compelling reason to do so, but these technologies are making that a much more realistic proposition.

The presence of cameras everywhere is considerably more concerning than the status quo, to me at least, when there is an AI watching and indexing every second of every feed—where camera owners or manufacturers or governments could set simple natural language parameters for highly specific people or activities notify about. There are obviously compelling and easy-to-sell cases here that will surely drive adoption as it becomes cost effective: get an alert to crime in progress, get an alert when a neighbor who doesn't clean up after his dog, get an alert when someone has fallen...but the potential implications of living in a panopticon like this if not well regulated are pretty ugly.

Post reply on HN