[deleted]
The killer app of Gemini Pro 1.5 is using video as an input
61–70 of 507 posts
Re: The killer app of Gemini Pro 1.5 is using video as an input
#62These things seem great for casual use, but not trustworthy enough for archival work, for example. The world needs casual-use tools, too, but there are bigger impact use cases in the pipeline. I'd love for these things to communicate when they're shaky on an interpretation, for example. Maybe pairing it with a different model and using an adversarial approach? Getting a confidence rating on existing messy data where the source is available for a second pass could be a good use case.
Looking at this, however, my hope is soured by the exponentially growing power of our law enforcement's panopticon. The existing shitty, buggy facial recognition system is already bad, but making automated fingerprints of people's movements based on their face combined with text on clothing and bags, the logos on your shoes, protest signs, alerting authorities if people have certain bumper stickers or books, recording the data on every card made visible when people open their wallets at public transit hubs or to pay for coffee or groceries, or set up a cheap remote camera across the street from a library to make a big list of every book checked out correlated with facial recognition... I mean, damn. Even in the private sector affording retailers the ability to make mass databases of any logo you've had on you when walking into their stores... or any stores considering it will be data brokers who keep it. Considering how much privacy our society has killed with the data we have, I'm genuinely concerned about what they will make next. Attempts to limit Facebook, et al may well seem quaint pretty soon. How about criminal applications? You can get a zoom camera with incredible range for short money, and surely it wouldn't be that hard to find a counter in front of a window where people show sensitive documents. Even just putting a phone with the camera facing out in your shirt pocket and walking around a target rich environment could be useful when you can comb through that gathered data looking for patterns, too.
That said, I'm not in security, law enforcement, crime, or marketing data collection so maybe I'm full of beans and just being neurotic.
Edit: if you're going to downvote me, surely you're capable of articulating your opposition in a comment, no?
Re: The killer app of Gemini Pro 1.5 is using video as an input
#63Note that a video is just a sequence of images: OpenAI has a demo with GPT-4-Vision that sends a list of frames to the model with a similar effect: https://cookbook.openai.com/examples/gpt_with_vision_for_vid... If GPT-4-Vision supported function calling/structured data for guaranteed JSON output, that would be nice though. There's shenanigans you can do with ffmpeg to output every-other-frame to halve the costs too.…
We've done extensive comparisons against GPT-4V for video inputs in our technical report: https://storage.googleapis.com/deepmind-media/gemini/gemini_... . Most notably, at 1FPS the GPT-4V API errors out around 3-4 mins, while 1.5 Pro supports upto an hour of video inputs.
The average shot length in modern movies is between 4 and 16 seconds and around 1 minute for a scene.
Re: The killer app of Gemini Pro 1.5 is using video as an input
#64How would the results compare to: 1. Video frames are sampled (based on frame clarity) 2. The images are fed to OCR, with their content outputed as: Frame X: 3. The accomulated text is given to an average LLM (Mistral) and asked the same request mentioned by the author (creating a JSON file containing book information) Wouldn't we get something similar? maybe if a more sophisticed AI is used? So the monopoly on Gemin…
Sure, you can slice a video up into images and process them separately - that's apparently how Gemini Pro works, it uses one frame from every second of video. But you still need a REALLY long context length to work with that information - the magic combination here is 1,000,000 tokens combined with good multi-model image inputs.
I see, but I was wondering about the partial transferability of this feature to other LLMs
But fair enough, context length is key in this scenario
Re: The killer app of Gemini Pro 1.5 is using video as an input
#65Re: The killer app of Gemini Pro 1.5 is using video as an input
#66Can it work on traffic I wonder? Automatic number-plate recognition (ALPR)
Re: The killer app of Gemini Pro 1.5 is using video as an input
#67Re: The killer app of Gemini Pro 1.5 is using video as an input
#68I was just today thinking that AI assisted editing could be a nice interface. You could watch the image and work mostly by speaking.
Computer could pull the images based on description. Make first assembly edit and give alternatives. Ok drop that shot, cut from this shot when the characters eyes leave the frame, replace this take etc. There is something in editing that feels contained enough that in can be described with language.
Re: The killer app of Gemini Pro 1.5 is using video as an input
#69Note that a video is just a sequence of images: OpenAI has a demo with GPT-4-Vision that sends a list of frames to the model with a similar effect: https://cookbook.openai.com/examples/gpt_with_vision_for_vid... If GPT-4-Vision supported function calling/structured data for guaranteed JSON output, that would be nice though. There's shenanigans you can do with ffmpeg to output every-other-frame to halve the costs too.…
We've done extensive comparisons against GPT-4V for video inputs in our technical report: https://storage.googleapis.com/deepmind-media/gemini/gemini_... . Most notably, at 1FPS the GPT-4V API errors out around 3-4 mins, while 1.5 Pro supports upto an hour of video inputs.
>while 1.5 Pro supports upto an hour of video inputs
At what price, tho?
Re: The killer app of Gemini Pro 1.5 is using video as an input
#70To me the 'It didn’t get all of them' is what makes me think this AI thing is just a toy. Don't get me wrong, it's marvelous as it is, but it only is useful (I use ollama + mistral 7B) when I know nothing, if I do have some understanding of the topic at hand it just becomes plain wrong. Hopefully I will be corrected.