Earlier quoted context omitted.
The whole matter of tokens from video is one that has a lot of ambiguity, and is often presented as if these are some unique weird encoding of the contents of the video. But logically the only possible tokenization of videos (or images, or series of images ala video) is basically an image to text model that takes each frame and generates descriptive language -- in English in Gemini -- to describe the contents of the…
This is not at all how this works. There's no separate model. Yes there's unique tokenization, if not the video as a whole then for each image. The whole video is ~1800 tokens because Gemini gets video as a series of images in context at 1 frame/s. Each image is about 258 tokens because a token in image transformer terms is literally a patch of the image. https://arxiv.org/abs/2010.11929
You can literally convert the tokens returned from a video to text. What do you even think tokens are?
Like seriously, before you write another word on this feel free to call the API and retrieve tokens for a video or image. Now go through the magical process of converting those tokens back to their text form. It isn't some magical hyper-dimensional, inside-out spatial encoding that yields impossible compression.
This process is obvious and logical if actually thought through.
>Each image is about 258 tokens
Because Google set that as the "budget" and truncates accordingly. Again, call the API with an image or video and then convert those tokens to text.
>https://arxiv.org/abs/2010.11929
This is super weird, and does not remotely prove your point. I literally spend most of my days in ViTs, but thanks for the link.