Live data from Hacker News

The killer app of Gemini Pro 1.5 is using video as an input

simonwillison.net

171–180 of 507 posts

Re: The killer app of Gemini Pro 1.5 is using video as an input

#171

"So Google’s new Gemini chatbot is racist as fuck." https://twitter.com/JoshWalkos/status/1760423141942178037

This has nothing to do with the model's capabilities and isn't substantially different from the vast majority of mainstream values in content moderation on social media.

Re: The killer app of Gemini Pro 1.5 is using video as an input

#172

Ok, crazy tangent; Where agents will potentially become extremely useful/dystopian is when they just silently watch your entire screen at all times. Isolated, encrypted and local preferably. Imagine it just watching you coding for months, planning stuff, researching things, it could potentially give you personal and professional advice from deep knowledge about you. "I noticed you code this way, may i recommend this…

I'm working on this! https://www.perfectmemory.ai/ It's encrypted (on top of Bitlocker) and local. There's all this competition who makes the best, most articulate LLM. But the truth is that off-the-shelf 7B models can put sentences together with no problem. It's the context they're missing.

Your website and blog are very low on details on how this is working. Downloading and installing an mai directly feels unsafe imo. Especially when I don't know how this software is working. Is it recording a video, performing OCR continuously, taking just screenshots

No mention of using any LLMs in there at all which is how you are presenting it in your comment here.

Re: The killer app of Gemini Pro 1.5 is using video as an input

#173

Earlier quoted context omitted.

I'm working on this! https://www.perfectmemory.ai/ It's encrypted (on top of Bitlocker) and local. There's all this competition who makes the best, most articulate LLM. But the truth is that off-the-shelf 7B models can put sentences together with no problem. It's the context they're missing.

I feel like the storage requirements are really going to be these issue for these apps/services that run on "take screenshots and OCR them" functionality with LLMs. If you're using something like this a huge part of the value proposition is in the long term, but until something has a more efficient way to function, even a 1-year history is impractical for a lot of people. For example, consider the classic situation o…

It's not that bad. With Perfect Memory AI I see ~9GB a month. That's 108 GB/year. HDD/SSDs are getting bigger than that every year. The storage also varies by what you do, your workflow and display resolution. Here's an article I wrote on my finding of storage requirements. https://www.perfectmemory.ai/support/storage-resources/stora...

And if you want to use the data for LLM only, then you don't need to store the screenshots at all. Then it's ~ 15MB a month

Re: The killer app of Gemini Pro 1.5 is using video as an input

#174

Ok, crazy tangent; Where agents will potentially become extremely useful/dystopian is when they just silently watch your entire screen at all times. Isolated, encrypted and local preferably. Imagine it just watching you coding for months, planning stuff, researching things, it could potentially give you personal and professional advice from deep knowledge about you. "I noticed you code this way, may i recommend this…

I'm working on this! https://www.perfectmemory.ai/ It's encrypted (on top of Bitlocker) and local. There's all this competition who makes the best, most articulate LLM. But the truth is that off-the-shelf 7B models can put sentences together with no problem. It's the context they're missing.

Basically looks like rewind.ai but for the PC?

Re: The killer app of Gemini Pro 1.5 is using video as an input

#175

Earlier quoted context omitted.

please consider commenting more thoughtfully. I understand this is a joke but we don't want this site to devolve into Reddit.

[flagged]

I don't have a flag button or option my account is new

Re: The killer app of Gemini Pro 1.5 is using video as an input

#176

Earlier quoted context omitted.

I'm working on this! https://www.perfectmemory.ai/ It's encrypted (on top of Bitlocker) and local. There's all this competition who makes the best, most articulate LLM. But the truth is that off-the-shelf 7B models can put sentences together with no problem. It's the context they're missing.

Basically looks like rewind.ai but for the PC?

exactly. the UI is shockingly similar

Re: The killer app of Gemini Pro 1.5 is using video as an input

#177
It may end being truly a killer app.

It is already bad for privacy the amount of video that is around, but increasing some orders how fast, easy and scalable may be processing it may increase the amount that is processed, even if is not perfect identifying what is there. And that by different actors, not just governments or intelligence agencies.

Now match that what is happening right now in Palestine in the present or somewhere else in a not so far future.

Re: The killer app of Gemini Pro 1.5 is using video as an input

#178

[flagged]

Rude!

I knew putting a video of a bookshelp up would risk people judging me based on the books they saw there, but I got over that by deciding that if anyone did that it would reflect badly on them, not badly on me.

(Unsurprisingly, I own a lot more books than the ones visible in those videos.)

Re: The killer app of Gemini Pro 1.5 is using video as an input

#179

As it says the audio is stripped/removed from video before processing, wonder how well it'd do if asked to transcribe by lip reading?

It looks like it actually only considers one frame for every second of video, so that certainly wouldn't work.

Re: The killer app of Gemini Pro 1.5 is using video as an input

#180

[flagged]

That’s a harsh judgement to make without considering whether, for example, he might mostly keep work books in the bookshelf where he works. I’d also suggest doing a little homework, since you could easily correct that false impression from his public presence.
Post reply on HN