Live data from Hacker News

The killer app of Gemini Pro 1.5 is using video as an input

simonwillison.net

201–210 of 507 posts

Re: The killer app of Gemini Pro 1.5 is using video as an input

#201
post #197

The killer app of ai is robots. Like, literally killer but also farmer, cleaner, builder etc

And car driver.

When I heard about how Tesla was training it's AI - without describing objects but instead through direct observation - it reminded me of Heinlein's "Door Into Summer" (1956). Heinlein's character teaches a multipurpose robot how to do any tedious human task through direct observation.

Re: The killer app of Gemini Pro 1.5 is using video as an input

#203

Earlier quoted context omitted.

I'm working on this! https://www.perfectmemory.ai/ It's encrypted (on top of Bitlocker) and local. There's all this competition who makes the best, most articulate LLM. But the truth is that off-the-shelf 7B models can put sentences together with no problem. It's the context they're missing.

I feel like the storage requirements are really going to be these issue for these apps/services that run on "take screenshots and OCR them" functionality with LLMs. If you're using something like this a huge part of the value proposition is in the long term, but until something has a more efficient way to function, even a 1-year history is impractical for a lot of people. For example, consider the classic situation o…

This reminds me of how Sherlock, Spotlight and its iterations came to be. It was very resource intensive to index everything and keep a live db, until it was not.

Re: The killer app of Gemini Pro 1.5 is using video as an input

#204

Earlier quoted context omitted.

I feel like the storage requirements are really going to be these issue for these apps/services that run on "take screenshots and OCR them" functionality with LLMs. If you're using something like this a huge part of the value proposition is in the long term, but until something has a more efficient way to function, even a 1-year history is impractical for a lot of people. For example, consider the classic situation o…

It's not that bad. With Perfect Memory AI I see ~9GB a month. That's 108 GB/year. HDD/SSDs are getting bigger than that every year. The storage also varies by what you do, your workflow and display resolution. Here's an article I wrote on my finding of storage requirements. https://www.perfectmemory.ai/support/storage-resources/stora... And if you want to use the data for LLM only, then you don't need to store the sc…

Is the 15mb basically embeddings from the video screenshots? What would it recall if there isn't the screenshots saved?

Re: The killer app of Gemini Pro 1.5 is using video as an input

#205

Earlier quoted context omitted.

It's not that bad. With Perfect Memory AI I see ~9GB a month. That's 108 GB/year. HDD/SSDs are getting bigger than that every year. The storage also varies by what you do, your workflow and display resolution. Here's an article I wrote on my finding of storage requirements. https://www.perfectmemory.ai/support/storage-resources/stora... And if you want to use the data for LLM only, then you don't need to store the sc…

> That's 108 GB/year. HDD/SSDs are getting bigger than that every year. Cries in MacBook Pro

Outboard TB 3/4 storage only seems expensive until you price it against Apple's native storage. Is it slower? Of course! Is it fast enough? Probably.

Re: The killer app of Gemini Pro 1.5 is using video as an input

#206
post #145

Earlier quoted context omitted.

Unless its open sourced :)

In modern world open code often doesn't mean much. E.g. Chrome is opensourced. And yet no one really contributes to it or has any say over the direction its going: https://twitter.com/RickByers/status/1715568535731155100

The graph seems to be that browsers are able to focus more resources towards improving the browser than improving the browser engine to meet their needs. If the browser engine already has what they need there is less of need for companies to dig deep into the internals. It's a sign of maturity and also a sign that open source work is properly being funded.

Re: The killer app of Gemini Pro 1.5 is using video as an input

#207
post #35

Title should have input added to the end "The killer app of Gemini Pro 1.5 is video input" Seems like a good way to do video moderation (YouTube) at scale, if they can keep costs down...

Probably overkill for content moderation, I'd think. You can identify bad words looking only at audio, and you can probably do nearly as good a job of identifying violence and nudity examining still images. And at YouTube scale, I imagine the main problem with moderation isn't so much as being correct, but of scaling. statista.com (what's up with that site, anyway?) suggests that YouTube adds something like 8 hours o…

> Probably overkill for content moderation

Content moderation is one of the hardest task we have at hand, we're burning though human souls looking at god awful stuff, lose their sanity, because simple filters just won't cut it.

For instance right now many rules exclude all nudity and the false positive rate is through the roof, while some of the nudity should actually be allowed and the rule in itself is hurting and should ideally be changed.

Even with our current simplistic rules I don't see automatic filters doing their job ("let me talk to an human" is our collective cry for help). When setting up more sensible rules ("nudity is OK when not sexualized, but not of minors, except for babies, if the viewer's coubtry allows for it"), I assume the resources and tuning needed to make that work on an automated systems would be of epic scale.

Re: The killer app of Gemini Pro 1.5 is using video as an input

#208

Earlier quoted context omitted.

[flagged]

I don't have a flag button or option my account is new

click on time since posted to get a flag button - exactly why hn works this way is a bit of a mystery, to me at least

Re: The killer app of Gemini Pro 1.5 is using video as an input

#209

Earlier quoted context omitted.

> That's 108 GB/year. HDD/SSDs are getting bigger than that every year. Cries in MacBook Pro

Outboard TB 3/4 storage only seems expensive until you price it against Apple's native storage. Is it slower? Of course! Is it fast enough? Probably.

I recently moved my macOS installation to an external Thunderbolt drive - it's faster than the internal SSD.

Re: The killer app of Gemini Pro 1.5 is using video as an input

#210
post #35

Title should have input added to the end "The killer app of Gemini Pro 1.5 is video input" Seems like a good way to do video moderation (YouTube) at scale, if they can keep costs down...

Probably overkill for content moderation, I'd think. You can identify bad words looking only at audio, and you can probably do nearly as good a job of identifying violence and nudity examining still images. And at YouTube scale, I imagine the main problem with moderation isn't so much as being correct, but of scaling. statista.com (what's up with that site, anyway?) suggests that YouTube adds something like 8 hours o…

That’s only 8 calls with a full context window per second. If that costs so much it makes Google do a double take, then maybe these AI things are just too expensive.

If it costs $1 per call, then over a year the entire perfect moderation of Youtube would cost roughly $250M. That seems sort of reasonable?

But probably pointless for most videos that are never watched by anyone other than the uploader, so maybe you just do this thing before anyone else watches the video and cut your costs by 50+%

Post reply on HN