Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The killer app of Gemini Pro 1.5 is using video as an input

simonwillison.net

Post reply on HN
Print view View original

91–100 of 507 posts

Re: The killer app of Gemini Pro 1.5 is using video as an input

#91

Post by tekni5 »

worldsayshi wrote: post #59
tekni5 wrote: post #57

I was thinking about this a while back, once AI is able to analyze video, images and text and do so cheap & efficiently. It's game over for privacy, like completely. Right now massive corps have tons of data on us, but they can't really piece it together and understand everything. With powerful AI every aspect of your digital life can be understood. The potential here is insane, it can be used for so many different t…

Unless you live in the EU and have laws that should protect you from that.

What happens if it's a datamining third party bot? That can check your social media accounts, create an in-depth profile on you, every image, video, post you've made has been recorded and understood. It knows everything about you, every product you use, where you have been, what you like, what you hate, everything packaged and ready to be sold to an advertiser, or the government, etc.

Re: The killer app of Gemini Pro 1.5 is using video as an input

#92

Post by yazaddaruvala »

system2 wrote: post #86
MyFirstSass wrote: post #71

Ok, crazy tangent; Where agents will potentially become extremely useful/dystopian is when they just silently watch your entire screen at all times. Isolated, encrypted and local preferably. Imagine it just watching you coding for months, planning stuff, researching things, it could potentially give you personal and professional advice from deep knowledge about you. "I noticed you code this way, may i recommend this…

If 7 second video consumed 1k token, I'd assume the budget must be insane to process such prompt.

Unlikely to be a prompt. It would need to be some form of fine tuning like LORA.

Re: The killer app of Gemini Pro 1.5 is using video as an input

#93

Post by MyFirstSass »

system2 wrote: post #86
MyFirstSass wrote: post #71

Ok, crazy tangent; Where agents will potentially become extremely useful/dystopian is when they just silently watch your entire screen at all times. Isolated, encrypted and local preferably. Imagine it just watching you coding for months, planning stuff, researching things, it could potentially give you personal and professional advice from deep knowledge about you. "I noticed you code this way, may i recommend this…

If 7 second video consumed 1k token, I'd assume the budget must be insane to process such prompt.

Yeah not feasible with todays methods and rag / lora shenanigans, but the way the field is moving i wouldn't be surprised if new decoder paradigms made it possible.

Saw this yesterday, 1M context window but haven't had any time to look into it, just an example new developments happening every week:

https://www.reddit.com/r/LocalLLaMA/comments/1as36v9/anyone_...

Re: The killer app of Gemini Pro 1.5 is using video as an input

#94

Post by Invictus0 »

system2 wrote: post #86
MyFirstSass wrote: post #71

Ok, crazy tangent; Where agents will potentially become extremely useful/dystopian is when they just silently watch your entire screen at all times. Isolated, encrypted and local preferably. Imagine it just watching you coding for months, planning stuff, researching things, it could potentially give you personal and professional advice from deep knowledge about you. "I noticed you code this way, may i recommend this…

If 7 second video consumed 1k token, I'd assume the budget must be insane to process such prompt.

That's a 7 second video from an HD camera. When recording a screen, you only really need to consider whats changing on the screen.

Re: The killer app of Gemini Pro 1.5 is using video as an input

#95

Post by cubefox »

So it is only about 256 tokens per image. I think the standard text tokenization method encodes two bytes per token, resulting in around 65.000 different tokens. If the same holds for images, given that they have the same price in the API, that would be just 512 bytes per image. Which seems impossibly low considering that the AI is still able to read those book titles. I don't understand what is going on here.

Re: The killer app of Gemini Pro 1.5 is using video as an input

#96

Post by simonw »

nox101 wrote: post #89
chefandy wrote: post #62

These things seem great for casual use, but not trustworthy enough for archival work, for example. The world needs casual-use tools, too, but there are bigger impact use cases in the pipeline. I'd love for these things to communicate when they're shaky on an interpretation, for example. Maybe pairing it with a different model and using an adversarial approach? Getting a confidence rating on existing messy data where…

honest question: Why is it bad? I see that posted over and over. Right now I watch SF and LA feel like 3rd world countries. Nothing appears to be enforced. Traffic laws, car break-ins, car theft, garage break-ins, house break-ins. I'd personally choose a little less privacy if it meant less people were getting injured by drivers ignoring the traffic laws and, less people were having to shell out for all the costs ass…

It's bad because while you may trust the government right now, there are no guarantees that a government you do NOT trust won't be elected in the future.

Also important to consider that government institutions are made up of individuals. Do you want a police officer who is the abuser in an bad domestic situation being given the power to track their partner using the resources made available to them in their work?

Re: The killer app of Gemini Pro 1.5 is using video as an input

#97

Post by mdanger007 »

MyFirstSass wrote: post #87
CamperBob2 wrote: post #82

Earlier quoted context omitted.

I liked this idea better in THX-1138.

One of the movies i've had on my watch list for far too long, thanks for reminding me. But yeah, dystopia is right down the same road we're all going right now.

Reading The Four by Scott Galloway, Apple, Facebook, Google, and Amazon were dominating the market 7 years ago generating 2.3 trillion in wealth. They're worth double that now.

The Four, especially with its AI, is going to control the market in ways that will have a deep impact on government and society.

Re: The killer app of Gemini Pro 1.5 is using video as an input

#99

Post by loudmax »

At the end of the article, a single image of the bookshelf uploaded to Gemini is 258 tokens. Gemini then responds with a listing of book titles, coming to 152 tokens.

Does anyone understand where the information for the response came from? That is, does Gemini hold onto the original uploaded non-tokenized image, then run an OCR on it to read those titles? Or are all those book titles somehow contained in those 258 tokens?

If it's the later, it seems amazing that these tokens contain that much information.

Re: The killer app of Gemini Pro 1.5 is using video as an input

#100

Post by ugh123 »

CamelCaseName wrote: post #50
CobrastanJorji wrote: post #47

Earlier quoted context omitted.

Probably overkill for content moderation, I'd think. You can identify bad words looking only at audio, and you can probably do nearly as good a job of identifying violence and nudity examining still images. And at YouTube scale, I imagine the main problem with moderation isn't so much as being correct, but of scaling. statista.com (what's up with that site, anyway?) suggests that YouTube adds something like 8 hours o…

For now, but in a year? You could also stagger the moderation to reduce costs. E.g. Text analysis: 2 views Audio analysis: 300 views Frame analysis: 5,000 views I would be very surprised if even 20% of content uploaded to YouTube passes 300 views.

Or.. google supplies some kind of local LLM tool which processes your videos before uploaded. You pay for the gpu/electricity costs. Obviously this would need to be done in a way that can't be hacked/manipulated. Might need to be highly integrated with a backend service that manages the analyzed frames from the local machine and verifies hashes/tokens after the video is fully uploaded to YouTube.
Post reply on HN