"So Google’s new Gemini chatbot is racist as fuck." https://twitter.com/JoshWalkos/status/1760423141942178037
https://twitter.com/LewisCTech/status/1760504251938492616 Seems to be fixed
[dead]
The killer app of Gemini Pro 1.5 is using video as an input
491–500 of 507 posts
Re: The killer app of Gemini Pro 1.5 is using video as an input
#492> It looks like the safety filter may have taken offense to the word “Cocktail”! I opened up the safety settings, dialled them down to “low” for every category and tried again. It appeared to refuse a second time. Google really is its own worst enemy. Their risk management people have completely taken over the organization to a point where somehow the smartest computers ever created are afraid of using dangerous word…
It's become absurd. Look at how creators now talk in their videos. "He tried to unalive himself". We are changing the way we speak to please these stupid algorithms when the context is the same.
Or "self-deletion" instead of "suicide".
Re: The killer app of Gemini Pro 1.5 is using video as an input
#493Earlier quoted context omitted.
A token is a single integer from a dictionary of a given model's vocabulary (e.g. GPT-4 has a vocab of ~100k different tokens, Gemma has ~256k). You are discussing embeddings which are a deeper, different element of models. https://platform.openai.com/tokenizer In the given example the video was condensed to a sequence of 258 tokens, and clearly it was a very minimalist, almost-entirely-ocr extraction from the video.
Yeah but we're not talking about LLMs here but vision transformers, which don't use the same type of token vocabulary to produce embeddings from the input as the LLMs do. The pixel data is much more dense than a few characters is, per token. I looked it up - the original ViT models directly projected for example 16x16 pixel patches into 768-dimensional "tokens". So a 224x224 image ended up as 14*14=196 "tokens" each…
>Yeah but we're not talking about LLMs here but vision transformers
We ultimately are. Gemini is a multimodal model whose core function is an LLM. This doesn't mean that everything flows through the same pathway -- different modalities have different paths -- but eventually there is fusion through which a common representations appears. It's where the worlds combine. That parlance is often tokens, though it obviously depends upon the architecture and we simply don't have those details for Gemini (the paper is extremely superficial). The fact that it will ingest massive videos and then post-facto answer arbitrary queries on it is a good clue, however.
>This blog-post has the specific number
It's a great link and an enjoyable read, and while the ViT plays a critical role in virtually all image analysis pipelines, including in Gemini where it is a part of OCR, object detection, etc, the numbers you are referring to do not map to tokens.
E.g. the 768 dimensions are nothing more than the underlying image data for the tile. e.g. 16x16x3 channels. I'm unaware of any ViT resources that refers to those vectors (vectorized because that's the form GPUs like) as tokens. This system could lazily reuse it, but the way processing happens in ViTs would make that a completely irrational overlap of terms.
The role that a token plays in that description is the classifier -- basically the output that classifies each tile.
Ultimately the number of tokens that Google or OpenAI assign to processing an image or video is a billing artifact because tokens are the measure by which things are billed. However you can ask these systems for the tokens representing an image and it will be exactly what one would expect. Indeed, the brilliance of image (and thus video) analysis in these multimodal systems is not nearly as deep as first glances might assume, and often it can derive nothing more than the most obvious classifications. e.g. classifications made without knowing anything about what the user specifically wants. It is usually fantastic at things like OCR, which happens to be a very common need.
These systems obviously have different usage patterns. I can do simultaneous processing where the image and command work in concert, image analysis deep diving on specifically those elements that are wanted (but that would otherwise be ignored). Or I can do the classic feed a video or an image and then ask questions where the dominant model is to tokenize the video or images using the common flow (OCR, object detection, etc), create a token narrative, and then answer the question from the narrative.
Re: The killer app of Gemini Pro 1.5 is using video as an input
#494Re: The killer app of Gemini Pro 1.5 is using video as an input
#495Earlier quoted context omitted.
I'm lost on most of your reply. No worries. My armchair knowledge of AI tells me there's degrees of influence from the safety teams about what is permitted and what is not permitted. My preference for "unconstrained" AI is a preference for less degrees of safety and more permissions. A preference for accuracy and objective truth over guardrails to words, facts, images, ideas. The original definition of "woke" is mora…
Got it. Good luck with all that I guess! Hope you find your sorcerer.
You've missed the metaphor, so I'll explain. The sorcerer is AI, and the mountain is the years of innovation by humanity to get there. We don't want all that effort "wokified" to please the easily offended. Or to please the overly obsessed ambassadors of DEI politics such as the founder of Google's "AI Responsibility" initiative, Jen Gennai. Your snarky responses don't make fun reading, btw.
Re: The killer app of Gemini Pro 1.5 is using video as an input
#496Earlier quoted context omitted.
"AI Companion" is a bit like spouse. You are married to it in the long run, unless you decide to divorce it. Definitely TRUST is the basis of marrage, and it should be the same for AI models. As in human marriage, there should be a law that said your AI-companion cannot be compelled to testify against you :-)
But unlike a spouse you can reset it back to an earlier state you preferred.
That's a noteworthy difference. Maybe AI only becomes truly "human" when it can't be reset. Maybe only then we can truly trust it - because it has the capability to betray us and yet it won't. (if it does then we don't trust it any more)
Re: The killer app of Gemini Pro 1.5 is using video as an input
#497Note that a video is just a sequence of images: OpenAI has a demo with GPT-4-Vision that sends a list of frames to the model with a similar effect: https://cookbook.openai.com/examples/gpt_with_vision_for_vid... If GPT-4-Vision supported function calling/structured data for guaranteed JSON output, that would be nice though. There's shenanigans you can do with ffmpeg to output every-other-frame to halve the costs too.…
How is sound handled? All I see in the Gemini docs is a terse sentence that says it isn’t included, which doesn’t sound like an optimal solution.
It doesn’t appear to be using the sound from the video, but elsewhere in the report for Gemini 1.5 pro it mentions it can handle sound directly as an input, without first transcribing it to text (including a chart that makes the point it’s much more accurate than transcribing text with whisper and then querying it using GPT-4).
But I don’t think it went into detail about how exactly that works, and I’m not sure if the API/front end has a good way to handle that.
Re: The killer app of Gemini Pro 1.5 is using video as an input
#498Earlier quoted context omitted.
Can we please argue on the thing being discussed rather than where it is common? Are you saying influencing life through ads and putting me in jail have similar effect on me? If you combine all laws of my country I am pretty sure I would have broken few unintentionally. If government wants to just put me in jail they could retroactively find any of my past instance if they have the data. This is not some theoretical…
The "thing being discussed" is the efficacy of privacy laws. They work well, and the fact that you haven't been put on trial for your 'crimes' yet is tacit evidence. In the real world, both corporations and governments are your enemy. You're mistakenly looking at it as a relativist comparison; the people influencing your life through advertising work with the people who put you in jail. They aggregate and sell data t…
But everything is relativist. There is and can't be any absolute privacy. We need to find the biggest gain we can have in privacy with minimal impact to economy. And making laws for online ads is the worst in terms of ROI. It impacts economy and millions of people could work because of ads and it offers very low benefit.
> you haven't been put on trial for your 'crimes' yet
I know someone who has been put to trial.
> They aggregate and sell data to Palantir
See here we are going to speculative domain. If there are companies who I trust not to do that, it would be big tech not because they are good, but because they know the value of data and are the ones which can extract highest value. And in any case it would require breaking TOS as companies list out their partners. And if we are entering illegal, anyways laws won't help with this.
Re: The killer app of Gemini Pro 1.5 is using video as an input
#499Ok, crazy tangent; Where agents will potentially become extremely useful/dystopian is when they just silently watch your entire screen at all times. Isolated, encrypted and local preferably. Imagine it just watching you coding for months, planning stuff, researching things, it could potentially give you personal and professional advice from deep knowledge about you. "I noticed you code this way, may i recommend this…
I'm working on this! https://www.perfectmemory.ai/ It's encrypted (on top of Bitlocker) and local. There's all this competition who makes the best, most articulate LLM. But the truth is that off-the-shelf 7B models can put sentences together with no problem. It's the context they're missing.
I installed it and kept it open for a full day but apparently it hasn't "saved" anything, and even if I open a Wiki page and a few minutes later search for that page, it returns nothing. Tried reading the Support FAQs on the website to no avail. Screen recording is on.
Re: The killer app of Gemini Pro 1.5 is using video as an input
#500One thing that has held back a lot of computer automation is context. For example, an app can know your geographic location, it can know your activity level, it can know what messages you send and receive, but it doesn't know you, it doesn't know how you think, what influences you, what perspective you have, what is happening in the world around you. An LLM that can look at images of the world around you constantly and continuously extracted details could provide an enormous amount of context that could feed automation. So your personal assistant knows why now is the right time to offer particular suggestions and knows why you would be interested (I'm anthropomorphizing as a shortcut). Huge privacy issues here also.
As an example, using AI to detect cognitive decline. A senior person is losing their balance more often recently as detected by an accelerometer. Are they experiencing a sudden cognitive decline? Part of the context might be that they had a visit from grandchildren recently and the children spent the day playing and left stuff scattered all over the house. Hence there is more stuff to trip over. Without the ability to extract that context the accelometer readings are difficult to interpret.