Are we really re-discovering that compressed binary formats are more efficient data representations?
60% Fable cost cut by converting code to images and having the model OCR it
91–100 of 115 posts
Re: 60% Fable cost cut by converting code to images and having the model OCR it
#92Are we really re-discovering that compressed binary formats are more efficient data representations?
No?
The image is still getting run through OCR and turned back into text before being fed into the LLM. There is no efficiency gain here, rather we have learned that Anthropic is applying a discount to text fed in via OCR.
Re: 60% Fable cost cut by converting code to images and having the model OCR it
#93Earlier quoted context omitted.
That's like saying quantization isn't real because the frontier labs aren't using it in their production inference. This is a lossy process, it produces worse results. It might be worth it for some situations, but applying it to everything would just be making your SOTA model worse
Isn't this just quantization with extra steps? Can converting the text to an image really be a better way to lossily compress it? (Not that I have any idea what I'm talking about on this topic.)
I also have no idea what I'm talking about, but to me this seems closer to the "caveman mode" that some people use to compress info into fewer tokens. Going through the image tokenizer allows you to leave the source text untouched while still gaining (some of?) the benefits
Re: 60% Fable cost cut by converting code to images and having the model OCR it
#94Earlier quoted context omitted.
Of course it isn't A text encoding uses 8bits per character on average, tokenization further compresses that An image font would be 25 bits if 5x5, and most fonts are 12 pixels high Of course it isn't efficient, this is a pricing inefficiency and a hack to exploit it (even the author describes it as an exploit)
You are wrong. Text tokens are high-dimensional vectors, not 8 bits per character. Every token has a deep embedding, e.g. 1024 float values per text token. DeepSeek-OCR proved 10x+ compression from visual embedding of text, which was a groundbreaking result. [1] Very cool to see OP's project hacking on this principle. It's still not lossless, as noted in the github, but is a promising research direction. [1] https://…
Well, then we could presumably also add lossy compression to texts, without passing through images first
Re: 60% Fable cost cut by converting code to images and having the model OCR it
#95Ahhh my eyes the vibe coded readme
It’s so painful to read the LLM-compressed explanations. I can’t exactly identify what it is, but it’s an immediate tell and literally requires twice the effort to comprehend. For example: > Honest caveat, visible in the clip: the pxpipe arm answered the count first and needed one follow-up nudge to also print the ledger balance in the requested one-line format; the plain arm followed the format on the first try. Leg…
It's about information density.
Most LLMs by default seem to write both text and code with low information density.
You can kind of get around it by prompting them to optimize for compactness, but most just let it run with a more generic prompt.
Re: 60% Fable cost cut by converting code to images and having the model OCR it
#96Earlier quoted context omitted.
Yeah, but as you say, this is another LLM rewriting it. The amount of noncontextual information is nauseating and destroys the point of a README (in my humble opinion). A human might have written a disclaimer like this: > When not using Fable, pxpipe may require additional follow-ups to precisely follow your formatting instructions. This kind of garbled information dump is very inconsiderate of the reader, and all go…
To be fair to the rewriting LLM, it was given only that single paragraph, with no other context and asked only to make it more comprehensible. But your point still stands. Here's what it said about _your_ rewrite: That human rewrite is excellent. It ruthlessly cuts out the "narrative" of the test case (the transaction counts, the video clip, the "first try vs second try" details) and extracts the only piece of inform…
[dead]
Re: 60% Fable cost cut by converting code to images and having the model OCR it
#97In Gemini at least, if you look at how they process PDFs, they do an OCR and then feed the text + image to the model, without charging you for the text tokens (I believe). So my guess is that Claude’s backend is doing the same — so this hack is probably more of a loophole in token accounting that might get closed if Claude is doing what Gemini does
This is really fascinating to me. I was reading this article and originally agreed with you, "I mean, under the covers it's got to be converting to text tokens at some point, so there is no way it's actually cheaper for Claude itself to execute." But then there is a comment below talking about how DeepSeek was able to get a huge improvement in compression by using visual tokens, https://news.ycombinator.com/item?id=4…
I am trying to get rough summaries of long PDFs of scanned pages of text. At first I was doing OCR and passing the (tens of thousands of) characters into the LLM, which works, but it's expensive.
I asked Gemini how to save costs and it said just send in all the images of the pages instead. Instinctively, as a developer, it's hard to fathom how sending 200 images is cheaper than sending the text, but it definitely works.
Re: 60% Fable cost cut by converting code to images and having the model OCR it
#98In Gemini at least, if you look at how they process PDFs, they do an OCR and then feed the text + image to the model, without charging you for the text tokens (I believe). So my guess is that Claude’s backend is doing the same — so this hack is probably more of a loophole in token accounting that might get closed if Claude is doing what Gemini does
This is really fascinating to me. I was reading this article and originally agreed with you, "I mean, under the covers it's got to be converting to text tokens at some point, so there is no way it's actually cheaper for Claude itself to execute." But then there is a comment below talking about how DeepSeek was able to get a huge improvement in compression by using visual tokens, https://news.ycombinator.com/item?id=4…
>This is really fascinating to me. I was reading this article and originally agreed with you, "I mean, under the covers it's got to be converting to text tokens at some point, so there is no way it's actually cheaper for Claude itself to execute."
It'd be weird if they were doing this, since it would mean the context window size was a lie and that the API would presumably reject requests whose expanded form went over the 1m limit. For someone using pxpipe with an effective context compression of 90% in some instances, it'd hit the limit at barely 100k.
Re: 60% Fable cost cut by converting code to images and having the model OCR it
#99Are we really re-discovering that compressed binary formats are more efficient data representations?
No? The image is still getting run through OCR and turned back into text before being fed into the LLM. There is no efficiency gain here, rather we have learned that Anthropic is applying a discount to text fed in via OCR.
I don't think this is what is happening, IMO. The models can genuinely "read" the text off the images, but usually at a less-than-perfect ratio, and it uses less tokens for the model on visual input than it does actually using OCR to convert them into text and then sending that in. I do not think there is any intermediate stage where they are applying a free OCR in this situation. (I realize that happens in some situations)
Re: 60% Fable cost cut by converting code to images and having the model OCR it
#100In Gemini at least, if you look at how they process PDFs, they do an OCR and then feed the text + image to the model, without charging you for the text tokens (I believe). So my guess is that Claude’s backend is doing the same — so this hack is probably more of a loophole in token accounting that might get closed if Claude is doing what Gemini does
This is really fascinating to me. I was reading this article and originally agreed with you, "I mean, under the covers it's got to be converting to text tokens at some point, so there is no way it's actually cheaper for Claude itself to execute." But then there is a comment below talking about how DeepSeek was able to get a huge improvement in compression by using visual tokens, https://news.ycombinator.com/item?id=4…
“I mean, under the covers it's got to be converting to text tokens at some point”
Multi-modal models do actually natively tokenize images, though. So it doesn’t have to be converted to text for it to work. They may do it anyway for accuracy, but it’s not at all required.
Effectively an image is scaled to a standard size, rasterized / cut up, and each cut is assigned a separate token, much in the same way text is tokenized. Train the model on this as well and you’ll end up having a model that can understand images.