Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The killer app of Gemini Pro 1.5 is using video as an input

simonwillison.net

Post reply on HN
Print view View original

461–470 of 507 posts

Re: The killer app of Gemini Pro 1.5 is using video as an input

#461

Post by llm_nerd »

famouswaffles wrote: post #458
llm_nerd wrote: post #451

Earlier quoted context omitted.

>This is not at all how this works. You can literally convert the tokens returned from a video to text. What do you even think tokens are? Like seriously, before you write another word on this feel free to call the API and retrieve tokens for a video or image. Now go through the magical process of converting those tokens back to their text form. It isn't some magical hyper-dimensional, inside-out spatial encoding tha…

>You can literally convert the tokens returned from a video to text. What do you even think tokens are? Tokens are patches of each image. It's amazing to me how people will confidently spout utter nonsense. It only takes looking at the technical report for the Gemini models to see that you're completely wrong. https://arxiv.org/abs/2312.11805 >The visual encoding of Gemini models is inspired by our own foundational w…

>It's amazing to me how people will confidently spout utter nonsense.

Ok.

You seem to be conflating some things, evident when you suddenly dropped the ViT paper as evidentiary. During the analysis of images, tiles and transformers (such as a ViT) are used. This is the model of processing the image to obtain useful information, such as to do OCR (you might notice that that word used repeatedly in the Google paper).

But to actually use the image, context has to be drawn from it. This is pretty bog standard OCR, object detection and classification, sentiment analysis, etc. This yields tokens.

Have you called the API and generated tokens from an image yet? Try it. You'll find they aren't as magical and mysterious as you believe, and your quasi-understanding of a ViT is not relevant to the tokens retrieved from a multimodal LLM.

There is the notion of semantic image tokens, which is an inner property of the analysis engine for images (and, conversely, the generation engine) but it is not what we're talking about. If an image was somehow collapsed into a 16x16 array of integers and amazingly it could still tell you the words on books and the objects that appear, that would be amazing. Too amazing.

Re: The killer app of Gemini Pro 1.5 is using video as an input

#462

Post by SkyBelow »

Cthulhu_ wrote: post #456
ginko wrote: post #430

Earlier quoted context omitted.

Assuming the person interacting with it is an adult, does it need any guard rails at all?

Yes it does, I don't want AI generating something that is illegal in my country. And it cannot make assumptions about where I live, due to VPNs and the like.

Doesn't this lead to the AI only being able to generate content that is legal in every country? That seems like a pretty bad standard and one that might even be impossible to meet given some countries with odd laws against specific things. If there were any countries which restricted speaking out against the government, should the AI be unable to generate anything deemed critical of those governments?

Also, if these are used in a professional setting, there is an even stricter criteria of not generating anything deemed inappropriate for that society. That might seem okay if we stick to an American only view (but even that I wouldn't actually bet on), but what happens if your AI shows things that violate very strong cultural norms of other societies, especially if those cultural norms run counter to our own?

Re: The killer app of Gemini Pro 1.5 is using video as an input

#463

Post by glenneroo »

kristofferR wrote: post #247
lillecarl wrote: post #196

Earlier quoted context omitted.

PerfectMemory is only available on Windows at the moment.

https://Rewind.ai is the macOS equivalent

Except that Rewind uses chatGPT whereas this runs entirely locally. I would like to note though that Anonymous Analytics are enabled as well as auto-updates, both of which I disabled for privacy reasons. Encryption is also disabled by default. I just blocked everything with my firewall for peace of mind :)

Re: The killer app of Gemini Pro 1.5 is using video as an input

#464

Post by beepbooptheory »

exodust wrote: post #436
beepbooptheory wrote: post #376

Earlier quoted context omitted.

So, when you read here that they are fixing it, is that a good thing to you? Do you think that means they are turning down the censorship knob? Because in reality they are only replacing the feedback they already have in place with different feedback. Again, there is simply no such thing as an "uncensored" model if what you mean by that is something that performs as well as Gemini (or whatever) but has zero external…

I'm lost on most of your reply. No worries. My armchair knowledge of AI tells me there's degrees of influence from the safety teams about what is permitted and what is not permitted. My preference for "unconstrained" AI is a preference for less degrees of safety and more permissions. A preference for accuracy and objective truth over guardrails to words, facts, images, ideas. The original definition of "woke" is mora…

Got it. Good luck with all that I guess! Hope you find your sorcerer.

Re: The killer app of Gemini Pro 1.5 is using video as an input

#465

Post by arbuge »

minimaxir wrote: post #2

Note that a video is just a sequence of images: OpenAI has a demo with GPT-4-Vision that sends a list of frames to the model with a similar effect: https://cookbook.openai.com/examples/gpt_with_vision_for_vid... If GPT-4-Vision supported function calling/structured data for guaranteed JSON output, that would be nice though. There's shenanigans you can do with ffmpeg to output every-other-frame to halve the costs too.…

How is sound handled?

All I see in the Gemini docs is a terse sentence that says it isn’t included, which doesn’t sound like an optimal solution.

Re: The killer app of Gemini Pro 1.5 is using video as an input

#466

Post by Cthulhu_ »

te_chris wrote: post #411
dEnigma wrote: post #252

> It looks like the safety filter may have taken offense to the word “Cocktail”! I'm definitely not a fan of these severely hamstrung by default models. Especially as it seems to be based on an extremely puritan ethical system.

Silicon Valley has been auto-parodic morals-wise for a while. Hell, just the basics of you can have super violent gaming but woe-betide you look at anything sex related in the appstores is intensely comedic. America desperately tries to export its puritanism but most of us just shrug (along with many Americans). Surely it's hard to argue that being open about sex (for consenting adults) is infinitely preferable to a…

And it's not even the SV companies themselves per se, it's their partners like credit card companies that will have nothing to with it, citing "think of the children".

Re: The killer app of Gemini Pro 1.5 is using video as an input

#467

Post by llm_nerd »

famouswaffles wrote: post #455
llm_nerd wrote: post #453

Earlier quoted context omitted.

Completely wrong. Well, aside from the edited in bit about OCR. Of course there isn't a separate run to do OCR because that was literally the first step during image analysis. You know, before the conversion to simple tokens.

There's no run to any OCR, first step or not. And you have no idea what you're talking about.

You understand that OCR is the process of extracting text from images, right? You know, such as what Gemini does, and they reference repeatedly in their paper. I have absolutely no idea why you repeatedly make some bizarre distinction about it being a "separate process".

Okay, it's been fun talking to you but feel free to have the last word. Good luck.

Re: The killer app of Gemini Pro 1.5 is using video as an input

#468

Post by famouswaffles »

llm_nerd wrote: post #461
famouswaffles wrote: post #458

Earlier quoted context omitted.

>You can literally convert the tokens returned from a video to text. What do you even think tokens are? Tokens are patches of each image. It's amazing to me how people will confidently spout utter nonsense. It only takes looking at the technical report for the Gemini models to see that you're completely wrong. https://arxiv.org/abs/2312.11805 >The visual encoding of Gemini models is inspired by our own foundational w…

>It's amazing to me how people will confidently spout utter nonsense. Ok. You seem to be conflating some things, evident when you suddenly dropped the ViT paper as evidentiary. During the analysis of images, tiles and transformers (such as a ViT) are used. This is the model of processing the image to obtain useful information, such as to do OCR (you might notice that that word used repeatedly in the Google paper). Bu…

>But to actually use the image, context has to be drawn from it. This is pretty bog standard OCR, object detection and classification, sentiment analysis, etc. This yields tokens

None of that is necessary for an Autoregressive Transformer. You can train the transformer to predict text tokens given interleaved image and text input tokens in the context window.

Google have already told us how this works. Read the Flamingo or Pali papers. You are wrong. Very wrong.

It's incredible that people will crucify LLMs for "hallucinating" but then there are humans like you running around.

Re: The killer app of Gemini Pro 1.5 is using video as an input

#469

Post by famouswaffles »

llm_nerd wrote: post #467
famouswaffles wrote: post #455

Earlier quoted context omitted.

There's no run to any OCR, first step or not. And you have no idea what you're talking about.

You understand that OCR is the process of extracting text from images, right? You know, such as what Gemini does, and they reference repeatedly in their paper. I have absolutely no idea why you repeatedly make some bizarre distinction about it being a "separate process". Okay, it's been fun talking to you but feel free to have the last word. Good luck.

The transformer (Gemini) predicts text with image and text in the context window. That's it.

OCR, Object detection etc all come from the transformer predicting text. Read the Flamingo paper.

Re: The killer app of Gemini Pro 1.5 is using video as an input

#470

Post by itsanaccount »

sconely wrote: post #324
itsanaccount wrote: post #317

Earlier quoted context omitted.

welp. this pretty much convinces me that its time I get out of tech. lean into the tradework I do in my spare time. because I'm sure you and people like you will succeed in your endeavors, naively thinking you're doing good. and you or someone like you will sell out, the most ruthless investor will take what you've built and use it as one more cludgel of power to beat the rest of us with.

If you want to help, use your knowledge to help shape policy. Because it is coming/already happening, and it will shape your life even if you are just living a simple life. I guarantee you that your city and state governments are passing legislation to incorporate AI to affect your life if they can be sold on it in the name of "good".

I live next to the Amish, trust me my township isn't passing anything related to AI.

For a reality check, name one instance of policy that has stopped the amoral march of tech being a tool of power to the hands of the few? Last one I can name is when they broke up Ma Bell. Now of course you can pick Verizon or AT&T, so that worked. /s

And that was 42 years ago.

Post reply on HN