Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The killer app of Gemini Pro 1.5 is using video as an input

simonwillison.net

Post reply on HN
Print view View original

71–80 of 507 posts

Re: The killer app of Gemini Pro 1.5 is using video as an input

#71

Post by MyFirstSass »

Ok, crazy tangent;

Where agents will potentially become extremely useful/dystopian is when they just silently watch your entire screen at all times. Isolated, encrypted and local preferably.

Imagine it just watching you coding for months, planning stuff, researching things, it could potentially give you personal and professional advice from deep knowledge about you. "I noticed you code this way, may i recommend this pattern" or "i noticed you have signs of this diagnosis from the way you move your mouse and consume content, may i recommend this lifestyle change".

I wonder how long before something like that is feasible, ie a model you install that is constantly updated, but also constantly merged with world data so it becomes more intelligent on two fronts, and can follow as hardware and software advances over the years.

Such a model would be dangerously valuable to corporations / bad actors as it would mirror your psyche and remember so much about you - so it would have to be running with a degree of safety i can't even imagine, or you'd be cloneable or loose all privacy.

Re: The killer app of Gemini Pro 1.5 is using video as an input

#72

Post by londons_explore »

tekni5 wrote: post #57

I was thinking about this a while back, once AI is able to analyze video, images and text and do so cheap & efficiently. It's game over for privacy, like completely. Right now massive corps have tons of data on us, but they can't really piece it together and understand everything. With powerful AI every aspect of your digital life can be understood. The potential here is insane, it can be used for so many different t…

> I bet it will be used to sell more targeted goods and services.

Plenty of companies have been shoving all the unstructured data they have about you and your friends into a big neural net to predict which ad you're most likely to click for a decade now...

Re: The killer app of Gemini Pro 1.5 is using video as an input

#73

Post by elzbardico »

Really. I am not that impressed. It is not something radically different from doing the same thing with a still photo which by now is trivial for those models.

What is being tested here doesn't require a video. It is not showing to be able to derive any meaning from a short clip. It is fucking doing very fancy OCR, that's all.

What would impress me is if shown a clip of an open chest surgery it was able to comment what surgery is being done, which technique is being used, or if shown video of construction workers, be able to figure out what is the building technique, what they are actually doing, telling that the guy with the yellow shirt is not following safety regulations by not wearing a helmet.

Re: The killer app of Gemini Pro 1.5 is using video as an input

#74

Post by bonoboTP »

IshKebab wrote: post #51

What do the tokens for an image even look like? I understand that tokens for text are just fragments of text... but that obviously doesn't make sense for images.

The image is subdivided by a grid and the resulting patches are fed through a linear encoder to get the token embeddings.

Re: The killer app of Gemini Pro 1.5 is using video as an input

#75

Post by YetAnotherNick »

worldsayshi wrote: post #59
tekni5 wrote: post #57

I was thinking about this a while back, once AI is able to analyze video, images and text and do so cheap & efficiently. It's game over for privacy, like completely. Right now massive corps have tons of data on us, but they can't really piece it together and understand everything. With powerful AI every aspect of your digital life can be understood. The potential here is insane, it can be used for so many different t…

Unless you live in the EU and have laws that should protect you from that.

Is it true or more of a myth? Based on my online read, Europe has "think of the children" narrative as common if not more than other parts of the world. They tried hard to ban encryption in apps many times.[1]

[1]: https://proton.me/blog/eu-council-encryption-vote-delayed

Re: The killer app of Gemini Pro 1.5 is using video as an input

#76

Post by AI_beffr »

he calls this technology "exciting." it makes me shudder. i have been contemplating this for a decade, this specific thing, and now it really is right in front of us. what happens when the useful data within any image or video stream can be extracted into the form of text and descriptions? a model of the world or of a country will emerge that you can hold in your hand. you can know the exact whereabouts of anyone at any time. you can know anything at any time. a real-time model of a country. and AI will be able to digest this model and answer questions about it. any government that has possession of such a system will wield absolute control in a way that has never been possible before. it will have massive implications. liberal democracy will no longer be viable as an economic of political framework. jeff bezos once said that we are essentially lucky that the most efficient way for resources to be utilized is in a decentralized manner. the fact that liberty is the strongest model economically, where everyone acts independently, is a happy coincidence. centralized economies, otherwise known as communism, havent worked in the past but that will change because with the power of AI, and with the real-time model and control-loop that it will make possible, the most efficient way to manage and deploy resources will be with one central management entity. in other words, an advanced AI will do literally everything for us, human labor will be made worthless, and countries that stick to the old ways will simply be made obsolete. inevitably, the AI-driven countries, with their pathetic blobs of parasitic human enclaves hanging off their tits, will move in on the old countries and destroy them for some inane reason such as needing more space to store antimatter. whatever.

even without looking all the way into the future, these AI video and image digesting tools will give birth to new and horrifying possibilities for bad actors in the government. their ability to steam roll over peoples lives in a bureaucratic stupor will be completely out of control. this seems like a sure thing but it doesnt seem likely at all that AI will be proactively and bravely used to counter-balance the negative uses by concerned citizens. people need to open their eyes to the possibility that different levels of technology are like points on a landscape -- not necessarily getting better or worse with time or "progress."

Re: The killer app of Gemini Pro 1.5 is using video as an input

#77

Post by az226 »

MyFirstSass wrote: post #71

Ok, crazy tangent; Where agents will potentially become extremely useful/dystopian is when they just silently watch your entire screen at all times. Isolated, encrypted and local preferably. Imagine it just watching you coding for months, planning stuff, researching things, it could potentially give you personal and professional advice from deep knowledge about you. "I noticed you code this way, may i recommend this…

Rewind.ai

Re: The killer app of Gemini Pro 1.5 is using video as an input

#78

Post by nomel »

belter wrote: post #12
minimaxir wrote: post #2

Note that a video is just a sequence of images: OpenAI has a demo with GPT-4-Vision that sends a list of frames to the model with a similar effect: https://cookbook.openai.com/examples/gpt_with_vision_for_vid... If GPT-4-Vision supported function calling/structured data for guaranteed JSON output, that would be nice though. There's shenanigans you can do with ffmpeg to output every-other-frame to halve the costs too.…

Prompt injection via Video?

Probably: https://simonwillison.net/2023/Oct/14/multi-modal-prompt-inj...

Re: The killer app of Gemini Pro 1.5 is using video as an input

#79

Post by elzbardico »

CamelCaseName wrote: post #50
CobrastanJorji wrote: post #47

Earlier quoted context omitted.

Probably overkill for content moderation, I'd think. You can identify bad words looking only at audio, and you can probably do nearly as good a job of identifying violence and nudity examining still images. And at YouTube scale, I imagine the main problem with moderation isn't so much as being correct, but of scaling. statista.com (what's up with that site, anyway?) suggests that YouTube adds something like 8 hours o…

For now, but in a year? You could also stagger the moderation to reduce costs. E.g. Text analysis: 2 views Audio analysis: 300 views Frame analysis: 5,000 views I would be very surprised if even 20% of content uploaded to YouTube passes 300 views.

People assume that we can scale the capabilities of LLMs indefinitely, I on the other side strongly suspect we are probably getting close to diminishing returns territory.

There's only so much you can do by guessing the next probably token in a stream. We will probably need something else to achieve what people think that will soon be done with LLMs.

Like Elon Musk probably realizing that computer vision is not enough for full self-driving, I expect we will soon reach the limits of what can be done with LLMs.

Re: The killer app of Gemini Pro 1.5 is using video as an input

#80

Post by tekni5 »

londons_explore wrote: post #72
tekni5 wrote: post #57

I was thinking about this a while back, once AI is able to analyze video, images and text and do so cheap & efficiently. It's game over for privacy, like completely. Right now massive corps have tons of data on us, but they can't really piece it together and understand everything. With powerful AI every aspect of your digital life can be understood. The potential here is insane, it can be used for so many different t…

> I bet it will be used to sell more targeted goods and services. Plenty of companies have been shoving all the unstructured data they have about you and your friends into a big neural net to predict which ad you're most likely to click for a decade now...

Sure but not images and video. Now they can look at a picture of your room and label everything you own, etc.
Post reply on HN