Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The killer app of Gemini Pro 1.5 is using video as an input

simonwillison.net

Post reply on HN
Print view View original

271–280 of 507 posts

Re: The killer app of Gemini Pro 1.5 is using video as an input

#271

Post by Solvency »

oconnor663 wrote: post #158
MyFirstSass wrote: post #71

Ok, crazy tangent; Where agents will potentially become extremely useful/dystopian is when they just silently watch your entire screen at all times. Isolated, encrypted and local preferably. Imagine it just watching you coding for months, planning stuff, researching things, it could potentially give you personal and professional advice from deep knowledge about you. "I noticed you code this way, may i recommend this…

A version of this that seems both easier and less weird would be an AI that listens to you all the time when you're learning a foreign language. Imagine how much faster you could learn, and how much more native you could ultimately get, if you had something that could buzz your watch whenever you said something wrong. And of course you'd calibrate it to understand what level you're at and not spam you constantly. I w…

I love how in a sea of navel-gazing ideas, this one is randomly being downvoted to oblivion. Does HN hate learning new languages or something?

Re: The killer app of Gemini Pro 1.5 is using video as an input

#272

Post by galaxyLogic »

ghxst wrote: post #115
Animats wrote: post #106

Earlier quoted context omitted.

> Imagine it just watching you coding for months, planning stuff, researching things, it could potentially give you personal and professional advice from deep knowledge about you. And then announcing "I can do your job now. You're fired."

That's why we would want it to run locally! Think about a fully personalized model that can work out some simple tasks / code while you're going out for groceries, or potentially more complex tasks while you're sleeping.

"AI Companion" is a bit like spouse. You are married to it in the long run, unless you decide to divorce it. Definitely TRUST is the basis of marrage, and it should be the same for AI models.

As in human marriage, there should be a law that said your AI-companion cannot be compelled to testify against you :-)

Re: The killer app of Gemini Pro 1.5 is using video as an input

#273

Post by dustingetz »

MyFirstSass wrote: post #71

Ok, crazy tangent; Where agents will potentially become extremely useful/dystopian is when they just silently watch your entire screen at all times. Isolated, encrypted and local preferably. Imagine it just watching you coding for months, planning stuff, researching things, it could potentially give you personal and professional advice from deep knowledge about you. "I noticed you code this way, may i recommend this…

thoughtcrime

Re: The killer app of Gemini Pro 1.5 is using video as an input

#274

Post by justworkout »

superb-owl wrote: post #193

The “cocktail” thing is real. A while back I tried to get DALLE to imagine characters from Moby Dick [1], but it completely refused. You’d think an AI company could come up with a better obscenity filter! [1] https://superb-owl.link/shapes-of-stories/#1513

I couldn't even get Google Gemini to generate a picture of, verbatim, "a man eating". It gave me a long winded lecture about how it's offensive and I should consider changing my views on the world. It does this with virtually any topic.

Re: The killer app of Gemini Pro 1.5 is using video as an input

#275

Post by ramenbytes »

searchableguy wrote: post #270
MyFirstSass wrote: post #71

Ok, crazy tangent; Where agents will potentially become extremely useful/dystopian is when they just silently watch your entire screen at all times. Isolated, encrypted and local preferably. Imagine it just watching you coding for months, planning stuff, researching things, it could potentially give you personal and professional advice from deep knowledge about you. "I noticed you code this way, may i recommend this…

I pre-ordered the rewind pendant. It will listen 24/7 and help you figure out what happened. I bet meta is thinking of doing this with quest once the battery life improves. https://rewind.ai/pendant

Black Mirror strikes again.

Re: The killer app of Gemini Pro 1.5 is using video as an input

#276

Post by bonoboTP »

MyFirstSass wrote: post #71

Ok, crazy tangent; Where agents will potentially become extremely useful/dystopian is when they just silently watch your entire screen at all times. Isolated, encrypted and local preferably. Imagine it just watching you coding for months, planning stuff, researching things, it could potentially give you personal and professional advice from deep knowledge about you. "I noticed you code this way, may i recommend this…

It doesn't even have to coach you at your job, simply a LLM-powered fuzzy retrieval would be great. Where did I put that file three weeks ago? What was that trick that I had to do to fix that annoying OS config issue? I recall seeing a tweet about a paper that did xyz about half a year ago, what was it called again?

Of course taking notes and bookmarking things is possible, but you can't include everything and it takes a lot of discipline to keep things neatly organized.

So we take it for granted that every once in a while we forget things, and can't find them again with web searching.

But with the new LLMs and multimodal models, in principle this can be solved. Just describe the thing you want to recall in vague natural language and the model will find it.

And this kind of retrieval is just one thing. But if it works well, we may also grow to rely on it a lot. Just as many who use GPS in the car never really learn the mental map of the city layout and can't drive around without it. Yeah, I know that some ancient philosopher derided the invention of books the same way (will make our memory lazy). But it can make us less capable by ourselves, but much more capable when augmented with this kind of near-perfect memory.

Re: The killer app of Gemini Pro 1.5 is using video as an input

#278

Post by andiareso »

elzbardico wrote: post #73

Really. I am not that impressed. It is not something radically different from doing the same thing with a still photo which by now is trivial for those models. What is being tested here doesn't require a video. It is not showing to be able to derive any meaning from a short clip. It is fucking doing very fancy OCR, that's all. What would impress me is if shown a clip of an open chest surgery it was able to comment wh…

But it can do that…

Re: The killer app of Gemini Pro 1.5 is using video as an input

#279

Post by bonoboTP »

chamomeal wrote: post #122
MyFirstSass wrote: post #71

Ok, crazy tangent; Where agents will potentially become extremely useful/dystopian is when they just silently watch your entire screen at all times. Isolated, encrypted and local preferably. Imagine it just watching you coding for months, planning stuff, researching things, it could potentially give you personal and professional advice from deep knowledge about you. "I noticed you code this way, may i recommend this…

Not crazy! I listened to a software engineering daily episode about pieces.app. Right now it’s some dev productivity tool or something, but in the interview the guy laid out a crazy vision that sounds like what you’re talking about. He was talking about eventually having an agent that watches your screen and remembers what you do across all apps, and can store it and share it with you team. So you could say “how does…

Even later it will be ingesting camera feeds from your AR glasses and listening in on your conversations, so you can remember what you agreed on. Just like automated meeting notes with Zoom which already exists, but it will be for real life 24/7.

Speech-to-text works. OCR works. LLMs are quite good at getting the semantics of the extracted text. Image understanding is pretty good too already. Just with the things that already exist right now, you can go most of the way.

And the CCTV cameras will also all be processed through something like it.

Re: The killer app of Gemini Pro 1.5 is using video as an input

#280

Post by 1shooner »

searchableguy wrote: post #270
MyFirstSass wrote: post #71

Ok, crazy tangent; Where agents will potentially become extremely useful/dystopian is when they just silently watch your entire screen at all times. Isolated, encrypted and local preferably. Imagine it just watching you coding for months, planning stuff, researching things, it could potentially give you personal and professional advice from deep knowledge about you. "I noticed you code this way, may i recommend this…

I pre-ordered the rewind pendant. It will listen 24/7 and help you figure out what happened. I bet meta is thinking of doing this with quest once the battery life improves. https://rewind.ai/pendant

This service says it's local and privacy-first, but it sends to OpenAI?

>Our service, Ask Rewind, integrates OpenAI’s ChatGPT, allowing for the extraction of key information from your device’s audio and video files to produce relevant and personalized outputs in response to your inputs and questions.

Post reply on HN