I find really hard to understand how a system like this can STILL be fooled by the Scunthorpe issue (this time with "cocktail"). Aren't LLM supposed to be good at context?
The killer app of Gemini Pro 1.5 is using video as an input
41–50 of 507 posts
Re: The killer app of Gemini Pro 1.5 is using video as an input
#42How would the results compare to:
1. Video frames are sampled (based on frame clarity)
2. The images are fed to OCR, with their content outputed as:
Frame X:
3. The accomulated text is given to an average LLM (Mistral) and asked the same request mentioned by the author (creating a JSON file containing book information)
Wouldn't we get something similar? maybe if a more sophisticed AI is used? So the monopoly on Gemini Pro for video processing (specifically when it comes to handling text present inside the video) is not really a sustainable advantage? or am I missing something (as this is something beyond just a fancy OCR hooked into a LLM? as the model would be able to tell that this text is on a book for instance?)
Re: The killer app of Gemini Pro 1.5 is using video as an input
#43This is nice. but since google is probably training on it's vast google books data set, i'm not extremely surprised.
Re: The killer app of Gemini Pro 1.5 is using video as an input
#44Everyone is missing the point, it seems (please BOFH me when wrong); Its not going to be all about "llms" and this app or that app... They all will talk, just like any other ecosystem, but this one is going to be different... it can ferret out connections as BGP will route. Gimme an AI from here, with this context, and that one and yes, please Id like another... and it will create soft LLMs - temporal ones dedicated…
Pre-emptive temporal curated LLMs in ..x0x
Meatbag translation: The pre-emptive is the cancer that will kill us.
Fuck you:
* insurance
* taxes
* health...
(what MAY this body-populous do, based on LLM-x trained on accuarial q and reduce from Human to cellular.
How fucking cyberpunk dystopian would one like to get.
The scariest wave of intellect is those that create technology before we had such technology "well, weve always been that way...
Robots (AI) have no such "I would like to play in the yard"
Re: The killer app of Gemini Pro 1.5 is using video as an input
#45I wonder if it could identify new books with titles it's never seen before.
Re: The killer app of Gemini Pro 1.5 is using video as an input
#46The tech is legitimately impressive and exciting, but I couldn't help but chuckle at the revenge of the Scunthorpe problem:
> It looks like the safety filter may have taken offense to the word “Cocktail”!
Re: The killer app of Gemini Pro 1.5 is using video as an input
#47Title should have input added to the end "The killer app of Gemini Pro 1.5 is video input" Seems like a good way to do video moderation (YouTube) at scale, if they can keep costs down...
Probably overkill for content moderation, I'd think. You can identify bad words looking only at audio, and you can probably do nearly as good a job of identifying violence and nudity examining still images. And at YouTube scale, I imagine the main problem with moderation isn't so much as being correct, but of scaling. statista.com (what's up with that site, anyway?) suggests that YouTube adds something like 8 hours of video per second. I didn't run the numbers, but I'm pretty sure that's way too much to cost effectively throw something like Gemini Pro at.
Re: The killer app of Gemini Pro 1.5 is using video as an input
#48That is impressive at first glance, no question. To stay with the example of the bookshelf, you would only follow this path for several or very many books, as in the example with the cookbooks. I have no idea how good the Geminis or GPTs of this world currently are, but let's optimistically assume a 3% error rate due to hallucinations or something. If I want to be sure that the results are correct, then I have to go…
This right here.
I'm currently building out some code that should go in production in the next week or two and simply because of this we are using LLM to prefill data and then have a human look over it.
For our use case the LLM prefilling the data is significantly faster but if it ever gets to the point of that not needing to happen it would take a task whichtakes about 3 hours ( now down to one hour ) and make it a task that takes 3 minutes.
Will LLMs ever get to the point where it is perfectly reliable ( or at least with an error margin low enough for our use case ), I don't think so.
It does make for a very cheap accelerator though.
Re: The killer app of Gemini Pro 1.5 is using video as an input
#49How would the results compare to: 1. Video frames are sampled (based on frame clarity) 2. The images are fed to OCR, with their content outputed as: Frame X: 3. The accomulated text is given to an average LLM (Mistral) and asked the same request mentioned by the author (creating a JSON file containing book information) Wouldn't we get something similar? maybe if a more sophisticed AI is used? So the monopoly on Gemin…
Sure, you can slice a video up into images and process them separately - that's apparently how Gemini Pro works, it uses one frame from every second of video.
But you still need a REALLY long context length to work with that information - the magic combination here is 1,000,000 tokens combined with good multi-model image inputs.
Re: The killer app of Gemini Pro 1.5 is using video as an input
#50Title should have input added to the end "The killer app of Gemini Pro 1.5 is video input" Seems like a good way to do video moderation (YouTube) at scale, if they can keep costs down...
Probably overkill for content moderation, I'd think. You can identify bad words looking only at audio, and you can probably do nearly as good a job of identifying violence and nudity examining still images. And at YouTube scale, I imagine the main problem with moderation isn't so much as being correct, but of scaling. statista.com (what's up with that site, anyway?) suggests that YouTube adds something like 8 hours o…
For now, but in a year?
You could also stagger the moderation to reduce costs. E.g.
Text analysis: 2 views
Audio analysis: 300 views
Frame analysis: 5,000 views
I would be very surprised if even 20% of content uploaded to YouTube passes 300 views.