Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The killer app of Gemini Pro 1.5 is using video as an input

simonwillison.net

Post reply on HN
Print view View original

161–170 of 507 posts

Re: The killer app of Gemini Pro 1.5 is using video as an input

#161

Post by lawlessone »

oconnor663 wrote: post #158
MyFirstSass wrote: post #71

Ok, crazy tangent; Where agents will potentially become extremely useful/dystopian is when they just silently watch your entire screen at all times. Isolated, encrypted and local preferably. Imagine it just watching you coding for months, planning stuff, researching things, it could potentially give you personal and professional advice from deep knowledge about you. "I noticed you code this way, may i recommend this…

A version of this that seems both easier and less weird would be an AI that listens to you all the time when you're learning a foreign language. Imagine how much faster you could learn, and how much more native you could ultimately get, if you had something that could buzz your watch whenever you said something wrong. And of course you'd calibrate it to understand what level you're at and not spam you constantly. I w…

>assuming it was voluntary...

Imagine if it was wrong about something. But every time you tried to submit the bug report it disables your arms via Nueralink.

Re: The killer app of Gemini Pro 1.5 is using video as an input

#162

Post by crooked-v »

DariusKocar wrote: post #143
MyFirstSass wrote: post #71

Ok, crazy tangent; Where agents will potentially become extremely useful/dystopian is when they just silently watch your entire screen at all times. Isolated, encrypted and local preferably. Imagine it just watching you coding for months, planning stuff, researching things, it could potentially give you personal and professional advice from deep knowledge about you. "I noticed you code this way, may i recommend this…

I'm working on this! https://www.perfectmemory.ai/ It's encrypted (on top of Bitlocker) and local. There's all this competition who makes the best, most articulate LLM. But the truth is that off-the-shelf 7B models can put sentences together with no problem. It's the context they're missing.

I feel like the storage requirements are really going to be these issue for these apps/services that run on "take screenshots and OCR them" functionality with LLMs. If you're using something like this a huge part of the value proposition is in the long term, but until something has a more efficient way to function, even a 1-year history is impractical for a lot of people.

For example, consider the classic situation of accidentally giving someone the same Christmas that you did a few years back. A sufficiently powerful personal LLM that 'remembers everything' could absolutely help with that (maybe even give you a nice table of the gifts you've purchased online, who they were for, and what categories of items would complement a previous gift), but only if it can practically store that memory for a multi-year time period.

Re: The killer app of Gemini Pro 1.5 is using video as an input

#163

Post by ssl-3 »

underdeserver wrote: post #126
ghxst wrote: post #115

Earlier quoted context omitted.

That's why we would want it to run locally! Think about a fully personalized model that can work out some simple tasks / code while you're going out for groceries, or potentially more complex tasks while you're sleeping.

It's local to your employer 's computer.

It can be.

It can also be local to my own computer. People do write software while they're away from work.

Re: The killer app of Gemini Pro 1.5 is using video as an input

#164

Post by mostlysimilar »

underdeserver wrote: post #126
ghxst wrote: post #115

Earlier quoted context omitted.

That's why we would want it to run locally! Think about a fully personalized model that can work out some simple tasks / code while you're going out for groceries, or potentially more complex tasks while you're sleeping.

It's local to your employer 's computer.

Corporations would absolutely force this until it could do your job and then fire you the second they could.

Re: The killer app of Gemini Pro 1.5 is using video as an input

#166

Post by lucubratory »

oconnor663 wrote: post #158
MyFirstSass wrote: post #71

Ok, crazy tangent; Where agents will potentially become extremely useful/dystopian is when they just silently watch your entire screen at all times. Isolated, encrypted and local preferably. Imagine it just watching you coding for months, planning stuff, researching things, it could potentially give you personal and professional advice from deep knowledge about you. "I noticed you code this way, may i recommend this…

A version of this that seems both easier and less weird would be an AI that listens to you all the time when you're learning a foreign language. Imagine how much faster you could learn, and how much more native you could ultimately get, if you had something that could buzz your watch whenever you said something wrong. And of course you'd calibrate it to understand what level you're at and not spam you constantly. I w…

I think even aside from the more outlandish ideas like that one, just having a fluent native speaker to talk to as much as you want would be incredibly valuable. Even more valuable if they are smart/educated enough to act as a language teacher. High-quality LLMs with a conversational interface capable of seamless language switching are an absolute killer app for language learning.

A use that seems scientifically possible but technically difficult would be to have an LLM help you engage in essentially immersion learning. Set up something like a pihole, but instead of cutting out ads it intercepts all the content you're consuming (webpages, text, video, images) and translates it to the language you're learning. The idea would be that you don't have to go out and find whole new sources of language to set yourself with a different language's information ecosystem, you can just press a button and convert your current information ecosystem to the language you want to learn. If something like that could be implemented it would be incredibly valuable.

Re: The killer app of Gemini Pro 1.5 is using video as an input

#167

Post by ChrisClark »

Animats wrote: post #106
MyFirstSass wrote: post #71

Ok, crazy tangent; Where agents will potentially become extremely useful/dystopian is when they just silently watch your entire screen at all times. Isolated, encrypted and local preferably. Imagine it just watching you coding for months, planning stuff, researching things, it could potentially give you personal and professional advice from deep knowledge about you. "I noticed you code this way, may i recommend this…

> Imagine it just watching you coding for months, planning stuff, researching things, it could potentially give you personal and professional advice from deep knowledge about you. And then announcing "I can do your job now. You're fired."

That sounds a lot like Learning To Be Me, by Greg Egan. Just not quite as advanced, or inside your head.

Re: The killer app of Gemini Pro 1.5 is using video as an input

#168

Post by londons_explore »

tekni5 wrote: post #80
londons_explore wrote: post #72

Earlier quoted context omitted.

> I bet it will be used to sell more targeted goods and services. Plenty of companies have been shoving all the unstructured data they have about you and your friends into a big neural net to predict which ad you're most likely to click for a decade now...

Sure but not images and video. Now they can look at a picture of your room and label everything you own, etc.

yes including images and video. It's been basically standard practice to take each piece of user data and turn it into an embedding vector, then combine all the vectors with some time/relevancy weighting or neural net, then use the resulting vector to predict user click through rates for ads. (which effectively determines which ad the user will see).

Re: The killer app of Gemini Pro 1.5 is using video as an input

#170

Post by halamadrid »

CamelCaseName wrote: post #50
CobrastanJorji wrote: post #47

Earlier quoted context omitted.

Probably overkill for content moderation, I'd think. You can identify bad words looking only at audio, and you can probably do nearly as good a job of identifying violence and nudity examining still images. And at YouTube scale, I imagine the main problem with moderation isn't so much as being correct, but of scaling. statista.com (what's up with that site, anyway?) suggests that YouTube adds something like 8 hours o…

For now, but in a year? You could also stagger the moderation to reduce costs. E.g. Text analysis: 2 views Audio analysis: 300 views Frame analysis: 5,000 views I would be very surprised if even 20% of content uploaded to YouTube passes 300 views.

It should be far less than 20%.

I guess it could also be associated with views per time period to optimize better. If the video is interesting, people will share and more views will happen quickly.

Post reply on HN