DeepSeek OCR

Post by krackers » Mon, Oct 27, 2025, 11:10 PM UTC

krackers wrote: post #247

miki123211 wrote: post #87

Earlier quoted context omitted.

Text tokens are quantized and represent subword units, vision tokens only exist in the embedding space. The way text tokenization works in LLMs is that you have a "lookup table" of (small) token ids to (large) vector embeddings. To pass text to the LLM, you split it at token boundaries, convert strings to token ids, and then construct the "context", a matrix where each row is a vector taken from that lookup table. Tr…

Thank you, this makes sense! As [1] puts it pithily >Image-patch tokens make better use of the high-dimensional embedding space than text tokens do. That seems to imply it's not necessarily something unique about images, just a byproduct of having better conversion from "raw input -> embeddings" [2]. Although there is a certain elegance of handling both images and text with the same method. [1] https://twitter.com/c0…