It’s sad that Google ai studio is not available in Canada.
The killer app of Gemini Pro 1.5 is using video as an input
21–30 of 507 posts
Re: The killer app of Gemini Pro 1.5 is using video as an input
#22I wonder if the real killer app is Googles hardware scale verses OpenAi' s(or what Microsoft gives them). Seems like nothing Google's done has been particular surprising to OpenAi's team, it's just they have such huge scale maybe they can iterate faster.
Re: The killer app of Gemini Pro 1.5 is using video as an input
#23Everyone is missing the point, it seems (please BOFH me when wrong);
Its not going to be all about "llms" and this app or that app...
They all will talk, just like any other ecosystem, but this one is going to be different... it can ferret out connections as BGP will route.
Gimme an AI from here, with this context, and that one and yes, please Id like another...
and it will create soft LLMs - temporal ones dedicated to their prompt and will pull from the tentriles of knowledge it can grasp and give you the result.
AI creates IRL Human Ephemeral Storage.
Re: The killer app of Gemini Pro 1.5 is using video as an input
#24hehe, this is great, I was just (2 days ago) playing with a similar problem in a web app form: browsing books in the foreign literature section of a Portuguese bookstore! My (less serious) ultimate goal is a universal sock pairing app: never fold your socks together again, just dump them in the drawer and ask the phone to find a match when you need them! This seems more like a visual segmentation problem though and s…
I employ a different strategy: I own 25 pairs of the same gray socks (gray was chosen so that it matches most outfits) and I just wear those all the time. Obviously I do own other socks (for suits etc.) but it has cumulatively saved me hours of sock searching.
[dead]
Re: The killer app of Gemini Pro 1.5 is using video as an input
#25Earlier quoted context omitted.
The number of tokens used for videos - 1,841 for my 7s video, 6,049 for 22s - suggests to me that this is a much more efficient way of processing content than individual frames. For structured data extraction I also like not having to run pseudo-OCR on hundreds of frames and then combine the results myself.
No it's individual frames https://developers.googleblog.com/2024/02/gemini-15-availabl... "Gemini 1.5 Pro can also reason across up to 1 hour of video. When you attach a video, Google AI Studio breaks it down into thousands of frames (without audio),..." But it's very likely individual frames at 1 frame/s https://storage.googleapis.com/deepmind-media/gemini/gemini_... "Figure 5 | When prompted with a 45 minute Buster…
The model is fed individual frames from the movie BUT the movie is segmented into scenes. These scenes, are held in context for 5-10 scenes, depending on their length. If the video exceeds a specific length or better said a threshold of scenes it creates an index and summary. So yes technically the model looks at individual frames but it's a bit more tooling behind it.
Re: The killer app of Gemini Pro 1.5 is using video as an input
#26I wonder if the real killer app is Googles hardware scale verses OpenAi' s(or what Microsoft gives them). Seems like nothing Google's done has been particular surprising to OpenAi's team, it's just they have such huge scale maybe they can iterate faster.
And the fact that Google are on their own hardware platform, not dependent on Nvidia for supply or hardware features.
Re: The killer app of Gemini Pro 1.5 is using video as an input
#27The real frustrating thing about this is how Gemini 1.5 is a marketing ploy only.
Not even 1.0 Ultra is available in the GCP API. only for their "allowlist" clients.
Re: The killer app of Gemini Pro 1.5 is using video as an input
#28I feel that while youtubers and influencers are heavily interested in video tools, most average users aren’t that interested in creating video.
I write a lot more email than sending out videos and the value of those videos is mostly just for sharing my life with friends and family, but my emails are often related to important professional communications.
I don’t think video tools will ever reach the level of usefulness to everyday consumers that generative writing tools create.
Re: The killer app of Gemini Pro 1.5 is using video as an input
#29Note that a video is just a sequence of images: OpenAI has a demo with GPT-4-Vision that sends a list of frames to the model with a similar effect: https://cookbook.openai.com/examples/gpt_with_vision_for_vid... If GPT-4-Vision supported function calling/structured data for guaranteed JSON output, that would be nice though. There's shenanigans you can do with ffmpeg to output every-other-frame to halve the costs too.…
We've done extensive comparisons against GPT-4V for video inputs in our technical report: https://storage.googleapis.com/deepmind-media/gemini/gemini_... . Most notably, at 1FPS the GPT-4V API errors out around 3-4 mins, while 1.5 Pro supports upto an hour of video inputs.
So that 3-4 mins at 1FPS means you are using about 500 to 700 tokens per image, which means you are using `detail: high` with something like 1080p to feed to gpt-4-vision-preview (unless you have another private endpoint).
The gemini 1.5 pro uses about 258 tokens per frame (2.8M tokens for 10856 frames).
Are those comparable?
Re: The killer app of Gemini Pro 1.5 is using video as an input
#30Wow, only 256 tokens per frame? I guess a picture isn’t worth a thousand words, just ~192.