Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

60% Fable cost cut by converting code to images and having the model OCR it

github.com

Post reply on HN
Print view View original

71–80 of 115 posts

Re: 60% Fable cost cut by converting code to images and having the model OCR it

#71

Post by Taek »

yorwba wrote: post #62
hn_throwaway_99 wrote: post #30

Earlier quoted context omitted.

This is really fascinating to me. I was reading this article and originally agreed with you, "I mean, under the covers it's got to be converting to text tokens at some point, so there is no way it's actually cheaper for Claude itself to execute." But then there is a comment below talking about how DeepSeek was able to get a huge improvement in compression by using visual tokens, https://news.ycombinator.com/item?id=4…

LLMs have a very bloated in-memory representation for text, on the order of megabytes of KV cache per byte of text. Meanwhile, for images a lossy representation is considered acceptable and it only takes up maybe a kilobyte of KV cache per byte of image. So if you can render your text into a hundred bytes of image per byte of text and then lossily expand it into 100 kB of KV cache per byte of text, you come out ahead…

I don't think it's that bad, if I recall correctly it's about 8 kilobytes per token, and a token can be 3-4 characters so you're talking ~2 kilobytes per character.

An image token I recall is something like 16x16, so you get 32 bytes of overhead per pixel. And a character is minimally like 20 pixels including the whitespace, so you've jumped from 4 characters per token to maybe 12.

So 3x savings... which actually maps pretty closely to 60% savings.

Re: 60% Fable cost cut by converting code to images and having the model OCR it

#72

Post by satvikpendem »

geor9e wrote: post #17
ricardobeat wrote: post #4

Earlier quoted context omitted.

It’s not a loophole, it just happens that encoding information as optical tokens is much more efficient than text.

Step back and think about it another way - ask which scenario is more likely: Some random person discovered a 60% across the board gain in all LLMs, using an extremely simple trick that none of the labs noticed in all these years. That trick being to rasterize 8bit characters into 8x8 pixels in a big image. 60% in a market worth trillions of dollars. or Anthropic's marketing team arbitrarily prices tokens to drive gr…

An economist walks past a hundred dollar bill on the ground because someone would've picked it up already if it were real.

Re: 60% Fable cost cut by converting code to images and having the model OCR it

#73

Post by calebkaiser »

jayd16 wrote: post #66
yorwba wrote: post #51

Earlier quoted context omitted.

The size is indeed smaller, because text tokens and image tokens are embedded as vectors of the same size, but text tokens typically only cover a few characters, while image tokens typically cover many pixels, so many that you can fit more characters in there. So the same text takes up fewer tokens as an image, and hence requires less time and memory to process. You could also imagine models where text tokens cover m…

So split the difference and start encoding input at the words or phrases level?

Lots of researchers have done just this! There's a really rich history of research + lots of contemporary work on different encoding/representation strategies. This might be interesting to you: https://sbert.net/

What makes the DeepSeek-OCR and related results exciting to some researchers is less about the fact that you could devise a tokenization scheme that has fewer tokens, and more about how well it works.

Re: 60% Fable cost cut by converting code to images and having the model OCR it

#75

Post by esafak »

yorwba wrote: post #62
hn_throwaway_99 wrote: post #30

Earlier quoted context omitted.

This is really fascinating to me. I was reading this article and originally agreed with you, "I mean, under the covers it's got to be converting to text tokens at some point, so there is no way it's actually cheaper for Claude itself to execute." But then there is a comment below talking about how DeepSeek was able to get a huge improvement in compression by using visual tokens, https://news.ycombinator.com/item?id=4…

LLMs have a very bloated in-memory representation for text, on the order of megabytes of KV cache per byte of text. Meanwhile, for images a lossy representation is considered acceptable and it only takes up maybe a kilobyte of KV cache per byte of image. So if you can render your text into a hundred bytes of image per byte of text and then lossily expand it into 100 kB of KV cache per byte of text, you come out ahead…

Can we get some refs for this number? If true it sounds like poor design.

Re: 60% Fable cost cut by converting code to images and having the model OCR it

#76

Post by electrotype »

mrbnprck wrote: post #67
electrotype wrote: post #44

What about: "Read this document online : [URL]" and you add your text/context to an online document? Would that reduce the number of tokens used too?

Documents are processed as tokens as well, unless its bitmap is ocr'd. Images tho are natively compatible with Multi-Modal LLMs, so theres no image->text translation layer in between. It's that the unit of cost is different (e.g. "visual token" vs text token)

I see. I was thinking that it might be different if the document wasn't provided by you directly, but instead if the LLM fetched it itself online.

Re: 60% Fable cost cut by converting code to images and having the model OCR it

#79

Post by sebmellen »

quantummagic wrote: post #45
sebmellen wrote: post #31

Earlier quoted context omitted.

It’s so painful to read the LLM-compressed explanations. I can’t exactly identify what it is, but it’s an immediate tell and literally requires twice the effort to comprehend. For example: > Honest caveat, visible in the clip: the pxpipe arm answered the count first and needed one follow-up nudge to also print the ledger balance in the requested one-line format; the plain arm followed the format on the first try. Leg…

Here's one rewrite that would have helped: To be fair, as you can see in the clip, the two models handled the prompt slightly differently. The pxpipe variant gave the right count initially but needed a quick follow-up to output the ledger balance in a single line. The standard model, on the other hand, nailed the formatting on its first try. We've completely solved readability here on Fable; our only real hurdle left…

Yeah, but as you say, this is another LLM rewriting it. The amount of noncontextual information is nauseating and destroys the point of a README (in my humble opinion).

A human might have written a disclaimer like this:

> When not using Fable, pxpipe may require additional follow-ups to precisely follow your formatting instructions.

This kind of garbled information dump is very inconsiderate of the reader, and all good writing is considerate of the audience consuming it.

Re: 60% Fable cost cut by converting code to images and having the model OCR it

#80

Post by Tuna-Fish »

esafak wrote: post #75
yorwba wrote: post #62

Earlier quoted context omitted.

LLMs have a very bloated in-memory representation for text, on the order of megabytes of KV cache per byte of text. Meanwhile, for images a lossy representation is considered acceptable and it only takes up maybe a kilobyte of KV cache per byte of image. So if you can render your text into a hundred bytes of image per byte of text and then lossily expand it into 100 kB of KV cache per byte of text, you come out ahead…

Can we get some refs for this number? If true it sounds like poor design.

It's not quite as bad as the parent made it out to be, the largest I've seen is 32kB per token (where sometimes, a token represents a byte, but usually it represents more than one.)

It's forced by the nature of how LLMs use vector embeddings for language.

Basically, a single token in a LLM is represented as a n-element vector, where n is the "hidden dimension", also known as model dimension. In order for the model to be smart, the hidden dimension needs to be large, on the order of 2^16 on top-tier models. Elements of this vector are typically quantized to 2-byte floats, or sometimes smaller. Every possible fact is embedded as a direction in this very many dimensional vector space, and a token is related to a fact if the vector representing that token points into a similar direction as that fact. You can do vector math about these things, famously for most trained models, if you find the vector embedding for king, man, woman and queen, and calculate king - man + woman, the result is very close to queen.

(Does that mean that there are 2^16 possible different kinds facts about things in this model? No, because high-dimensional geometry is very unintuitively powerful. The facts are not axis-aligned, and they don't need to be perfectly non-orthogonal. This matters, because the numbers of individual vectors you can fit into a single 2^16 dimensional space that are orthogonal with each other (all angles 90degrees) is of course 2^16. But, if you allow for almost orthogonal vectors, the number is larger than the amount of atoms in the universe. If this sounds wacky, for people with a CS background it can help to think it working a bit like a bloom filter, in that collisions are possible. Although in actuality they are theoretical, because 2^16 is a very large number.)

Post reply on HN