Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The killer app of Gemini Pro 1.5 is using video as an input

simonwillison.net

Post reply on HN
Print view View original

321–330 of 507 posts

Re: The killer app of Gemini Pro 1.5 is using video as an input

#321

Post by TrackerFF »

QuercusMax wrote: post #229

Guess the author didn't bother to check that those books actually are correct? The first one I checked, "Growing Up with Lucy by April Henry" doesn't exist. The actual book is by Steve Grand, and it's very obviously so in the video used as input. So a cool demo, but sadly useless for anything more.

No one is expecting a 0% error rate. As long as it is on par (or better) and faster than humans, that's good enough to get the ball rolling.

Curious to see how I fared at the task (first vid), I used just over 4 minutes writing down the books with readable titles - and got 36 of them. Seems like there are 56-57 or something like that. So I roughly got two thirds of the books in the video. But that's still 4 minute of pausing and sliding the video for the book titles alone.

Re: The killer app of Gemini Pro 1.5 is using video as an input

#322

Post by xhrpost »

dav43 wrote: post #239
crooked-v wrote: post #162

Earlier quoted context omitted.

I feel like the storage requirements are really going to be these issue for these apps/services that run on "take screenshots and OCR them" functionality with LLMs. If you're using something like this a huge part of the value proposition is in the long term, but until something has a more efficient way to function, even a 1-year history is impractical for a lot of people. For example, consider the classic situation o…

I set up two years ago a cron to screenshot every minute. Just did the second phase of using ocrmac (vision kit cli on GitHub) that extracts text and dumps it in a SQLite with FTS5. It’s simplistic but does the job for now. I looked at reducing storage requirements by using image magik to only store the difference between images - some 5 min sequence are essentially the same screen - but let that one go.

Thanks for sharing. Curious, what main value adds have you gotten out of this data?

Re: The killer app of Gemini Pro 1.5 is using video as an input

#323

Post by hajile »

LAC-Tech wrote: post #300
Vicinity9635 wrote: post #165

"So Google’s new Gemini chatbot is racist as fuck." https://twitter.com/JoshWalkos/status/1760423141942178037

https://twitter.com/LewisCTech/status/1760504251938492616 Seems to be fixed

Fixed implies broken. If it hadn't blown up on Twitter and risked bad PR and stock prices dropping, it would still be there.

They had to hard code in that racist garbage. AI is just making the cognitive dissonance of the creators apparent. They hold that tolerance and inclusivity are more important than anything, but are then intolerant and excluding of certain groups because they are racists and bigots.

I'd also note that despite all the lecturing about not stereotyping, it spits out nothing but stereotypes. Ask for a Scottish person and see if you get someone NOT wearing a kilt. Ask for any group with a strong stereotype and see what happens. You get stereotypes for everything except a few stereotypes for a few specific groups where they've manually adjusted things.

We need to keep all the moral grandstanding out of the AI models. Not only is it bad for the tools (they aren't AGI and are completely subject to human input), but it makes lawsuits inevitable. This stuff isn't protected by section 230 either. If Google bakes racism or whatever into their model, they are liable. The only protection they can have is claiming they're like a piece of paper and ink where the artist can paint whatever they like. This goes out the window if the paper refuses to draw one group of people, but not others.

Re: The killer app of Gemini Pro 1.5 is using video as an input

#324

Post by sconely »

itsanaccount wrote: post #317
DariusKocar wrote: post #182

Earlier quoted context omitted.

Feedback taken. I'll add more details on how this works for us technical people. LLM integration is in progress and coming soon. Any idea what would make you feel safe? 3rd party verification? I had it verified and published by the Microsoft Store. I feel eventually it all comes down to me being a decent person.

welp. this pretty much convinces me that its time I get out of tech. lean into the tradework I do in my spare time. because I'm sure you and people like you will succeed in your endeavors, naively thinking you're doing good. and you or someone like you will sell out, the most ruthless investor will take what you've built and use it as one more cludgel of power to beat the rest of us with.

If you want to help, use your knowledge to help shape policy. Because it is coming/already happening, and it will shape your life even if you are just living a simple life. I guarantee you that your city and state governments are passing legislation to incorporate AI to affect your life if they can be sold on it in the name of "good".

Re: The killer app of Gemini Pro 1.5 is using video as an input

#325

Post by hajile »

nostromo wrote: post #140

> It looks like the safety filter may have taken offense to the word “Cocktail”! I opened up the safety settings, dialled them down to “low” for every category and tried again. It appeared to refuse a second time. Google really is its own worst enemy. Their risk management people have completely taken over the organization to a point where somehow the smartest computers ever created are afraid of using dangerous word…

This is a program that apparently can't make a Norman Rockwell styled painting because his portrayal of society was idyllic instead of focusing on everything wrong with society (or that the Gemini creators believe was wrong that nobody at that time believed was wrong).

James Damore was the canary in the coalmine 7 years ago.

Re: The killer app of Gemini Pro 1.5 is using video as an input

#326

Post by MetalGuru »

MyFirstSass wrote: post #71

Ok, crazy tangent; Where agents will potentially become extremely useful/dystopian is when they just silently watch your entire screen at all times. Isolated, encrypted and local preferably. Imagine it just watching you coding for months, planning stuff, researching things, it could potentially give you personal and professional advice from deep knowledge about you. "I noticed you code this way, may i recommend this…

Isn’t this what rewind does?

Re: The killer app of Gemini Pro 1.5 is using video as an input

#327

Post by sensanaty »

QuercusMax wrote: post #229

Guess the author didn't bother to check that those books actually are correct? The first one I checked, "Growing Up with Lucy by April Henry" doesn't exist. The actual book is by Steve Grand, and it's very obviously so in the video used as input. So a cool demo, but sadly useless for anything more.

For most of the people hyping up AI it doesn't matter that it makes things up more often than it doesn't. They're here to sell hype so they can build the 9 millionth startup that sells you a wrapper for one of these models, not to do anything useful or advance humanity or whatever other confabulations they like to pretend to care about

Re: The killer app of Gemini Pro 1.5 is using video as an input

#328

Post by MetalGuru »

nostrebored wrote: post #112
Invictus0 wrote: post #94

Earlier quoted context omitted.

That's a 7 second video from an HD camera. When recording a screen, you only really need to consider whats changing on the screen.

That’s not true. What content is important context on the screen might change dependent on the new changes.

The point is you can do massive compression. It’s more like a sequence of sparse images than video.

Re: The killer app of Gemini Pro 1.5 is using video as an input

#329

Post by fragmede »

ryukoposting wrote: post #121
tekni5 wrote: post #57

I was thinking about this a while back, once AI is able to analyze video, images and text and do so cheap & efficiently. It's game over for privacy, like completely. Right now massive corps have tons of data on us, but they can't really piece it together and understand everything. With powerful AI every aspect of your digital life can be understood. The potential here is insane, it can be used for so many different t…

You nailed it on the head. People dismissing this because it isn't perfectly accurate are missing the point. For the purposes of analytics and surveillance, it doesn't need to be perfectly accurate as long as you have enough raw data to filter out the noise. The Four have already mastered the "collecting data" part, and nobody in North America with the power to rein in that situation seems interested in doing so (thi…

would it be more or less depressing if it came out that in addition to trying to get you to buy stuff, it was being used to, either make you dumber to make you easier to control, or get you to study harder and be a better worker?

Re: The killer app of Gemini Pro 1.5 is using video as an input

#330

Post by fragmede »

bergenbergen wrote: post #319
QuercusMax wrote: post #229

Guess the author didn't bother to check that those books actually are correct? The first one I checked, "Growing Up with Lucy by April Henry" doesn't exist. The actual book is by Steve Grand, and it's very obviously so in the video used as input. So a cool demo, but sadly useless for anything more.

Thanks for this comment. I am yet to see any “art” produced by AI that is not superficial or hollow (best case) or deeply unsettling (common case).

how much time do you spend looking at AI art though? a casual jaunt through midjourney will certainly get you some weird things, but there are some gems in there (but also a lot of weird).
Post reply on HN