How does this particular use case stack up against OCR?
The killer app of Gemini Pro 1.5 is using video as an input
11–20 of 507 posts
Re: The killer app of Gemini Pro 1.5 is using video as an input
#12Note that a video is just a sequence of images: OpenAI has a demo with GPT-4-Vision that sends a list of frames to the model with a similar effect: https://cookbook.openai.com/examples/gpt_with_vision_for_vid... If GPT-4-Vision supported function calling/structured data for guaranteed JSON output, that would be nice though. There's shenanigans you can do with ffmpeg to output every-other-frame to halve the costs too.…
Prompt injection via Video?
Re: The killer app of Gemini Pro 1.5 is using video as an input
#13hehe, this is great, I was just (2 days ago) playing with a similar problem in a web app form: browsing books in the foreign literature section of a Portuguese bookstore! My (less serious) ultimate goal is a universal sock pairing app: never fold your socks together again, just dump them in the drawer and ask the phone to find a match when you need them! This seems more like a visual segmentation problem though and s…
I employ a different strategy: I own 25 pairs of the same gray socks (gray was chosen so that it matches most outfits) and I just wear those all the time. Obviously I do own other socks (for suits etc.) but it has cumulatively saved me hours of sock searching.
Re: The killer app of Gemini Pro 1.5 is using video as an input
#14Earlier quoted context omitted.
No it's individual frames https://developers.googleblog.com/2024/02/gemini-15-availabl... "Gemini 1.5 Pro can also reason across up to 1 hour of video. When you attach a video, Google AI Studio breaks it down into thousands of frames (without audio),..." But it's very likely individual frames at 1 frame/s https://storage.googleapis.com/deepmind-media/gemini/gemini_... "Figure 5 | When prompted with a 45 minute Buster…
Despite that being in their blog post, I'm skeptical. I tried uploading a single frame of the video as an image and it consumed 258 tokens. The 7s video was 1,841 tokens. I think it's more complicated than just "split the video into frames and process those" - otherwise I would expect the token count for the video to be much higher than that. UPDATE ... posted that before you edited your post to link to the Gemini 1.…
I mean, that's just over 7 frames, or one frame/s of video. There are likely fewer then that many I-frames in your video.
Re: The killer app of Gemini Pro 1.5 is using video as an input
#15>GPT-4 Video and LLaVA expanded that to images.
A little error in the page: GPT-4V stands for vision, not video.
Re: The killer app of Gemini Pro 1.5 is using video as an input
#16How does this particular use case stack up against OCR?
I think OCR would fair pretty poorly on such messy visuals.
Not to mention the partially obscured titles that Gemini guessed well, which would be impossible for an OCR.
Re: The killer app of Gemini Pro 1.5 is using video as an input
#17Earlier quoted context omitted.
No it's individual frames https://developers.googleblog.com/2024/02/gemini-15-availabl... "Gemini 1.5 Pro can also reason across up to 1 hour of video. When you attach a video, Google AI Studio breaks it down into thousands of frames (without audio),..." But it's very likely individual frames at 1 frame/s https://storage.googleapis.com/deepmind-media/gemini/gemini_... "Figure 5 | When prompted with a 45 minute Buster…
Despite that being in their blog post, I'm skeptical. I tried uploading a single frame of the video as an image and it consumed 258 tokens. The 7s video was 1,841 tokens. I think it's more complicated than just "split the video into frames and process those" - otherwise I would expect the token count for the video to be much higher than that. UPDATE ... posted that before you edited your post to link to the Gemini 1.…
Added a note about this to my post: https://simonwillison.net/2024/Feb/21/gemini-pro-video/#imag...
Re: The killer app of Gemini Pro 1.5 is using video as an input
#18>GPT-4 Video and LLaVA expanded that to images. A little error in the page: GPT-4V stands for vision, not video.
Thanks, fixed.
Re: The killer app of Gemini Pro 1.5 is using video as an input
#19So you have to get invited to use Gemini Pro 1.5 right? EDIT: there is a waitlist here https://aistudio.google.com/app/waitlist/97445851
Re: The killer app of Gemini Pro 1.5 is using video as an input
#20Note that a video is just a sequence of images: OpenAI has a demo with GPT-4-Vision that sends a list of frames to the model with a similar effect: https://cookbook.openai.com/examples/gpt_with_vision_for_vid... If GPT-4-Vision supported function calling/structured data for guaranteed JSON output, that would be nice though. There's shenanigans you can do with ffmpeg to output every-other-frame to halve the costs too.…
We've done extensive comparisons against GPT-4V for video inputs in our technical report: https://storage.googleapis.com/deepmind-media/gemini/gemini_....
Most notably, at 1FPS the GPT-4V API errors out around 3-4 mins, while 1.5 Pro supports upto an hour of video inputs.