Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The killer app of Gemini Pro 1.5 is using video as an input

simonwillison.net

Post reply on HN
Print view View original

281–290 of 507 posts

Re: The killer app of Gemini Pro 1.5 is using video as an input

#281

Post by zacmps »

simonw wrote: post #125
zacmps wrote: post #117

Earlier quoted context omitted.

Remember, if it's using a similar tokeniser to GPT-4 (cl100k_base iirc), each token has a dimension of ~100,000. So 258x100,000 is a space of 25,800,000 floats, using f16 (a total guess) that's 51.6kB, probably enough to represent the image at ok quality with JPG.

I don't think that's right. A token in GPT-4 is a single integer, not a vector of floats. Input to a model gets embedded into vectors later, but the actual tokens are pretty tiny.

Ah true, I guess it's still 258 positions by 100,000 possible tokens though.

Re: The killer app of Gemini Pro 1.5 is using video as an input

#282

Post by wincy »

nostromo wrote: post #140

> It looks like the safety filter may have taken offense to the word “Cocktail”! I opened up the safety settings, dialled them down to “low” for every category and tried again. It appeared to refuse a second time. Google really is its own worst enemy. Their risk management people have completely taken over the organization to a point where somehow the smartest computers ever created are afraid of using dangerous word…

When you consider the Gorilla in the room, it makes more sense. Google is absolutely terrified of a repeat of classifying black people as great apes. [0] Apparently this apprehension is so great that both iOS and Android have an inability to tag “gorilla” in images.

[0] https://www.wsj.com/articles/BL-DGB-42522

Re: The killer app of Gemini Pro 1.5 is using video as an input

#283

Post by slimsag »

xyzelement wrote: post #267
dEnigma wrote: post #252

> It looks like the safety filter may have taken offense to the word “Cocktail”! I'm definitely not a fan of these severely hamstrung by default models. Especially as it seems to be based on an extremely puritan ethical system.

We're months into this technology being available so it's not a surprise that the various "safeties" have not been perfectly tuned. Perhaps Google knew they couldn't be perfect right now and they could err on the side of the model refusing to talk about cocktails, or err on the side of it gladly spouting about cocks. They may have made a perfectly valid choice for the moment.

If you want a great example of how this plays out long-term, look no further than algospeak[0] - the new lingo created by censorship algorithms like those on youtube and tiktok.

[0] https://www.nytimes.com/2022/11/19/style/tiktok-avoid-modera...

Re: The killer app of Gemini Pro 1.5 is using video as an input

#285

Post by [deleted] »

elzbardico wrote: post #73

Really. I am not that impressed. It is not something radically different from doing the same thing with a still photo which by now is trivial for those models. What is being tested here doesn't require a video. It is not showing to be able to derive any meaning from a short clip. It is fucking doing very fancy OCR, that's all. What would impress me is if shown a clip of an open chest surgery it was able to comment wh…

[deleted]

Re: The killer app of Gemini Pro 1.5 is using video as an input

#287

Post by Nition »

bonoboTP wrote: post #276
MyFirstSass wrote: post #71

Ok, crazy tangent; Where agents will potentially become extremely useful/dystopian is when they just silently watch your entire screen at all times. Isolated, encrypted and local preferably. Imagine it just watching you coding for months, planning stuff, researching things, it could potentially give you personal and professional advice from deep knowledge about you. "I noticed you code this way, may i recommend this…

It doesn't even have to coach you at your job, simply a LLM-powered fuzzy retrieval would be great. Where did I put that file three weeks ago? What was that trick that I had to do to fix that annoying OS config issue? I recall seeing a tweet about a paper that did xyz about half a year ago, what was it called again? Of course taking notes and bookmarking things is possible, but you can't include everything and it tak…

Eventually someone will realise that it'd also be great for telling you where you left your keys, if it'd film everything you see instead of just your screen.

Re: The killer app of Gemini Pro 1.5 is using video as an input

#290

Post by [deleted] »

crooked-v wrote: post #162
DariusKocar wrote: post #143

Earlier quoted context omitted.

I'm working on this! https://www.perfectmemory.ai/ It's encrypted (on top of Bitlocker) and local. There's all this competition who makes the best, most articulate LLM. But the truth is that off-the-shelf 7B models can put sentences together with no problem. It's the context they're missing.

I feel like the storage requirements are really going to be these issue for these apps/services that run on "take screenshots and OCR them" functionality with LLMs. If you're using something like this a huge part of the value proposition is in the long term, but until something has a more efficient way to function, even a 1-year history is impractical for a lot of people. For example, consider the classic situation o…

[deleted]
Post reply on HN