Live data from Hacker News

The killer app of Gemini Pro 1.5 is using video as an input

simonwillison.net

301–310 of 507 posts

Re: The killer app of Gemini Pro 1.5 is using video as an input

#301
post #214

Earlier quoted context omitted.

I'm working on this! https://www.perfectmemory.ai/ It's encrypted (on top of Bitlocker) and local. There's all this competition who makes the best, most articulate LLM. But the truth is that off-the-shelf 7B models can put sentences together with no problem. It's the context they're missing.

Any plan to implement this on macOS or Linux?

macOS: https://screenmemory.app/

This is my application, it does not have AI running on top.

Re: The killer app of Gemini Pro 1.5 is using video as an input

#302

I can't access that Google AI Studio link because I'm in some strange place called the UK so I'm unable to verify or prototype with it currently. People at Deepmind, what's with that?

I can’t either because I’m using a strange new device called a phone and it says my device width is too small to support.

Re: The killer app of Gemini Pro 1.5 is using video as an input

#303

Ok, crazy tangent; Where agents will potentially become extremely useful/dystopian is when they just silently watch your entire screen at all times. Isolated, encrypted and local preferably. Imagine it just watching you coding for months, planning stuff, researching things, it could potentially give you personal and professional advice from deep knowledge about you. "I noticed you code this way, may i recommend this…

Aside. Is this your first Sass or Saas?

Re: The killer app of Gemini Pro 1.5 is using video as an input

#304
post #77

Earlier quoted context omitted.

Rewind.ai

I have tried Rewind and found it very disappointing. Transcripts were of very poor quality and the screen capture timeline proved useless to me.

If I may do some advertising, I specifically disliked the timeline in Rewind.ai so much so that I built my own application https://screenmemory.app. In fact the timeline is what I work on the most and have the most plans for.

Re: The killer app of Gemini Pro 1.5 is using video as an input

#305
post #131

Earlier quoted context omitted.

> encrypted and local of course Only for people who'd pay for that. Free users would become the product.

I noticed you code this way, may i recommend a Lenovo Thinkpad with an Intel Xeon processor? You're sure to "wish everything was a Lenovo."

Certainly! Here is a list of great thinkpads.

The x230 is a popular and interesting thinkpad with a powerful i5 processor suitable for today’s needs.

The T60 can also suit your needs and is one of the last IBM thinkpads. It featured the latest Intel mobile processor at the time of its release.

If you want the most powerful thinkpad the T440p is sure to suit you perfectly without leaving your morals behind.

Re: The killer app of Gemini Pro 1.5 is using video as an input

#307
post #287

Earlier quoted context omitted.

It doesn't even have to coach you at your job, simply a LLM-powered fuzzy retrieval would be great. Where did I put that file three weeks ago? What was that trick that I had to do to fix that annoying OS config issue? I recall seeing a tweet about a paper that did xyz about half a year ago, what was it called again? Of course taking notes and bookmarking things is possible, but you can't include everything and it tak…

Eventually someone will realise that it'd also be great for telling you where you left your keys, if it'd film everything you see instead of just your screen.

I simply am not going to have my entire life filmed by an form of technology, I don't care what the advantages are. There's a limit to the level of dystopian dependent uses of these technologies I'm going to put up with. I sincerely hope the majority of the human race feels the same way.

Re: The killer app of Gemini Pro 1.5 is using video as an input

#308
In the same vein as "agents watching your screen" - what about "agents watching your posture"? Pages like [0] and [1] exist because people experience great benefits from becoming (even slightly) more aware of the way they are holding their bodies. Imagine this idea taken to the extreme, with a local agent intelligently reminding you to tighten your core, square your shoulders, relax your tongue, or warning of potential incoming RSI?

[0] https://news.ycombinator.com/item?id=35939206

[1] https://static.virtualsaleslab.com/vsl-poc/ppp/

Re: The killer app of Gemini Pro 1.5 is using video as an input

#309
post #214

Earlier quoted context omitted.

I'm working on this! https://www.perfectmemory.ai/ It's encrypted (on top of Bitlocker) and local. There's all this competition who makes the best, most articulate LLM. But the truth is that off-the-shelf 7B models can put sentences together with no problem. It's the context they're missing.

Any plan to implement this on macOS or Linux?

I got 90% of this built on Linux (around KDE Wayland) before other interests/priorities took over:

https://github.com/Zetaphor/screendiary/

Re: The killer app of Gemini Pro 1.5 is using video as an input

#310
post #308

In the same vein as "agents watching your screen" - what about "agents watching your posture"? Pages like [0] and [1] exist because people experience great benefits from becoming (even slightly) more aware of the way they are holding their bodies. Imagine this idea taken to the extreme, with a local agent intelligently reminding you to tighten your core, square your shoulders, relax your tongue, or warning of potenti…

> He sat as still as he could on the narrow bench, with his hands crossed on his knee. He had already learned to sit still. If you made unexpected movements they yelled at you from the telescreen.
Post reply on HN