Live data from Hacker News

The killer app of Gemini Pro 1.5 is using video as an input

simonwillison.net

101–110 of 507 posts

Re: The killer app of Gemini Pro 1.5 is using video as an input

#101
post #99

At the end of the article, a single image of the bookshelf uploaded to Gemini is 258 tokens. Gemini then responds with a listing of book titles, coming to 152 tokens. Does anyone understand where the information for the response came from? That is, does Gemini hold onto the original uploaded non-tokenized image, then run an OCR on it to read those titles? Or are all those book titles somehow contained in those 258 to…

I would LOVE to understand that myself.

Re: The killer app of Gemini Pro 1.5 is using video as an input

#102

Ok, crazy tangent; Where agents will potentially become extremely useful/dystopian is when they just silently watch your entire screen at all times. Isolated, encrypted and local preferably. Imagine it just watching you coding for months, planning stuff, researching things, it could potentially give you personal and professional advice from deep knowledge about you. "I noticed you code this way, may i recommend this…

Why watch your screen when you could feed in video from a wearable pair of glasses like those Instagram Ray Bans. And why stop at video when you could have it record and learn from a mic that is always on. And you might as well throw in a feed of your GPS location and biometrics from your smart watch.

When you consider it, we aren't very far away from that at all.

Re: The killer app of Gemini Pro 1.5 is using video as an input

#103
post #77

Ok, crazy tangent; Where agents will potentially become extremely useful/dystopian is when they just silently watch your entire screen at all times. Isolated, encrypted and local preferably. Imagine it just watching you coding for months, planning stuff, researching things, it could potentially give you personal and professional advice from deep knowledge about you. "I noticed you code this way, may i recommend this…

Rewind.ai

I have tried Rewind and found it very disappointing. Transcripts were of very poor quality and the screen capture timeline proved useless to me.

Re: The killer app of Gemini Pro 1.5 is using video as an input

#104

Earlier quoted context omitted.

One of the movies i've had on my watch list for far too long, thanks for reminding me. But yeah, dystopia is right down the same road we're all going right now.

Reading The Four by Scott Galloway, Apple, Facebook, Google, and Amazon were dominating the market 7 years ago generating 2.3 trillion in wealth. They're worth double that now. The Four, especially with its AI, is going to control the market in ways that will have a deep impact on government and society.

Yeah, that's one of the developments i'm unable to spin positively.

As technological society advances the threshold to enter the market with anything not completely laughable becomes exponentially harder, only consolidating old money or the already established right?

What i found so amazing about the early internet, or even just the internet 2.0 was the possibility to create a platform/marketplace/magazine or whatever, and actually have it take off and get a little of the shared growth.

But now it seems all growth has become centralised to a few apps and marketplaces and the barrier to entry is getting harder by the hour.

Ie. being an entrepreneur is harder now because of tech and market consolidation. But potentially mirrored in previous eras like the industrialisation - i'm just not sure we'll get another "reset" like that to allow new players.

Please someone explain how this is wrong and there's still hope for the tech entrepreneurs / sideprojects!

Re: The killer app of Gemini Pro 1.5 is using video as an input

#105
post #70

To me the 'It didn’t get all of them' is what makes me think this AI thing is just a toy. Don't get me wrong, it's marvelous as it is, but it only is useful (I use ollama + mistral 7B) when I know nothing, if I do have some understanding of the topic at hand it just becomes plain wrong. Hopefully I will be corrected.

[deleted]

Re: The killer app of Gemini Pro 1.5 is using video as an input

#106

Ok, crazy tangent; Where agents will potentially become extremely useful/dystopian is when they just silently watch your entire screen at all times. Isolated, encrypted and local preferably. Imagine it just watching you coding for months, planning stuff, researching things, it could potentially give you personal and professional advice from deep knowledge about you. "I noticed you code this way, may i recommend this…

> Imagine it just watching you coding for months, planning stuff, researching things, it could potentially give you personal and professional advice from deep knowledge about you.

And then announcing "I can do your job now. You're fired."

Re: The killer app of Gemini Pro 1.5 is using video as an input

#107
post #99

At the end of the article, a single image of the bookshelf uploaded to Gemini is 258 tokens. Gemini then responds with a listing of book titles, coming to 152 tokens. Does anyone understand where the information for the response came from? That is, does Gemini hold onto the original uploaded non-tokenized image, then run an OCR on it to read those titles? Or are all those book titles somehow contained in those 258 to…

I'm not sure about Gemini, but OpenAI GTP-V bills at roughly a token per 40x40px square. It isn't clear to me these actually processed as units, but rather it seems like they tried to approximate the cost structure to match text.

Re: The killer app of Gemini Pro 1.5 is using video as an input

#108
post #89

These things seem great for casual use, but not trustworthy enough for archival work, for example. The world needs casual-use tools, too, but there are bigger impact use cases in the pipeline. I'd love for these things to communicate when they're shaky on an interpretation, for example. Maybe pairing it with a different model and using an adversarial approach? Getting a confidence rating on existing messy data where…

honest question: Why is it bad? I see that posted over and over. Right now I watch SF and LA feel like 3rd world countries. Nothing appears to be enforced. Traffic laws, car break-ins, car theft, garage break-ins, house break-ins. I'd personally choose a little less privacy if it meant less people were getting injured by drivers ignoring the traffic laws and, less people were having to shell out for all the costs ass…

Sadly, you should disabuse yourself of the notion that our government will only use these powers in our best interest by looking at COINTELPRO, manufactured evidence for invading Iraq, mass incarceration based on nonviolent crimes, surveilling and prosecuting rape victims who live in the wrong jurisdictions for seeking abortions, police treatment of people who speak out against them (they'll have access, too,) the red scare, etc. etc. etc. And that's entirely ignoring what we may be subject to by other governments. Even the increasing polarity between partisan political entities is concerning. If our country is run by someone comfortable with encouraging their supporters to violently put down opposition, do you want them supported by agencies that have access to this stuff? If you are, should everybody else have to be?

One way I gauge where we are is to compare it to what people previously considered problematic. We've witnessed a tectonic shift in the overton window for reasonable surveillance-- each incremental change is presented as a reasonable, prudent step that a preponderance of people agree is beneficial. However, if you compiled the changes that have taken place and presented to someone from 1984, for example, they'd be understandably shocked.

For people that have the correct ideas about what to believe, what to say, what to do, and how to do it according to everyone from their municipal jurisdictions to the federal government and all of it's arms, it's probably not a problem. Can we accept the government installing machinery to squash everybody else?

Speeding and red light camera tickets are one thing-- they selectively capture stills of people who have likely committed a crime. Camera networks that track all cars movement by recording license plate sightings are more representative of what the future looks like. Think I'm being paranoid? It's already implemented: https://turnto10.com/news/local/providence-police-department...

Edit: again, if you're going to downvote me, surely you're capable of articulating your opposition in a comment, no?

Re: The killer app of Gemini Pro 1.5 is using video as an input

#109

The tech is legitimately impressive and exciting, but I couldn't help but chuckle at the revenge of the Scunthorpe problem: > It looks like the safety filter may have taken offense to the word “Cocktail”!

> > It looks like the safety filter may have taken offense to the word “Cocktail”!

It's almost as if they got some intern to "code" the correctness filter using some AI coding assistant!

Re: The killer app of Gemini Pro 1.5 is using video as an input

#110

Ok, crazy tangent; Where agents will potentially become extremely useful/dystopian is when they just silently watch your entire screen at all times. Isolated, encrypted and local preferably. Imagine it just watching you coding for months, planning stuff, researching things, it could potentially give you personal and professional advice from deep knowledge about you. "I noticed you code this way, may i recommend this…

I would hate that so much.
Post reply on HN