Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

60% Fable cost cut by converting code to images and having the model OCR it

github.com

Post reply on HN
Print view View original

81–90 of 115 posts

Re: 60% Fable cost cut by converting code to images and having the model OCR it

#81

Post by ptx »

Klathmon wrote: post #61
geor9e wrote: post #23

Earlier quoted context omitted.

If the trick were genuinely useful, and was well circulated months ago, the resource-starved inference providers would have squeezed this trick dry already, instead of wasting 60% of their tokens, waiting for users to implement it themselves in 5 minutes of effort.

That's like saying quantization isn't real because the frontier labs aren't using it in their production inference. This is a lossy process, it produces worse results. It might be worth it for some situations, but applying it to everything would just be making your SOTA model worse

Isn't this just quantization with extra steps? Can converting the text to an image really be a better way to lossily compress it? (Not that I have any idea what I'm talking about on this topic.)

Re: 60% Fable cost cut by converting code to images and having the model OCR it

#82

Post by michaelt »

aabhay wrote: post #10

In Gemini at least, if you look at how they process PDFs, they do an OCR and then feed the text + image to the model, without charging you for the text tokens (I believe). So my guess is that Claude’s backend is doing the same — so this hack is probably more of a loophole in token accounting that might get closed if Claude is doing what Gemini does

Not necessarily. See the paper See "DeepSeek-OCR: Contexts Optical Compression" [1]

One option, when an image is fed into an LLM, is to divide it into tiles, then those tiles pass through a 'vision encoder' neural network to make 'vision tokens' which are then input into the LLM much like text tokens are. Obviously you train the vision encoder and LLM to understand one another. This is known as an 'end-to-end OCR model'.

And it turns out, once you've trained a model to do this, you can vary the number of 'vision tokens' used to represent a given text document by scaling an image of a document up or down, and see what happens. You also get a load of other parameters like patch size and vision encoder complexity and so on.

Turns out it works really well; in some tests they used 90% fewer input tokens, but still got 97% output performance.

[1] https://arxiv.org/abs/2510.18234

Re: 60% Fable cost cut by converting code to images and having the model OCR it

#84

Post by brumar »

Tangentially related: I don't think OCR is the right term and I am generally vocal about that. But seeing this unquestioned here, I am wondering if I am the one who is wrong here. Is it ok to call this OCR? To me ocr means text in the end, not visual tokens.

Re: 60% Fable cost cut by converting code to images and having the model OCR it

#85

Post by lubujackson »

sebmellen wrote: post #31
aabhay wrote: post #5

Ahhh my eyes the vibe coded readme

It’s so painful to read the LLM-compressed explanations. I can’t exactly identify what it is, but it’s an immediate tell and literally requires twice the effort to comprehend. For example: > Honest caveat, visible in the clip: the pxpipe arm answered the count first and needed one follow-up nudge to also print the ledger balance in the requested one-line format; the plain arm followed the format on the first try. Leg…

[dead]

Re: 60% Fable cost cut by converting code to images and having the model OCR it

#86

Post by parsimo2010 »

brumar wrote: post #84

Tangentially related: I don't think OCR is the right term and I am generally vocal about that. But seeing this unquestioned here, I am wondering if I am the one who is wrong here. Is it ok to call this OCR? To me ocr means text in the end, not visual tokens.

OCR means optical character recognition. The terms do not require a direct transcription, but that is mostly what OCR meant in the past. If you’re using an LLM’s vision capability to pass in text and the LLM actually understands it, then I would say that it recognized the characters, hence OCR seems okay to use.

Re: 60% Fable cost cut by converting code to images and having the model OCR it

#87

Post by devmor »

brumar wrote: post #84

Tangentially related: I don't think OCR is the right term and I am generally vocal about that. But seeing this unquestioned here, I am wondering if I am the one who is wrong here. Is it ok to call this OCR? To me ocr means text in the end, not visual tokens.

So if I OCR a document, edit it, and print it, OCR didn't happen?

Re: 60% Fable cost cut by converting code to images and having the model OCR it

#88

Post by TurdF3rguson »

brumar wrote: post #84

Tangentially related: I don't think OCR is the right term and I am generally vocal about that. But seeing this unquestioned here, I am wondering if I am the one who is wrong here. Is it ok to call this OCR? To me ocr means text in the end, not visual tokens.

It's not. OCR is not what the vision model is doing here. We're used to using OCR as a verb but it's more accurate to say the model "visioned" it.

Also, some models still do OCR and it's usually way more expensive that way.

Re: 60% Fable cost cut by converting code to images and having the model OCR it

#90

Post by quantummagic »

sebmellen wrote: post #79
quantummagic wrote: post #45

Earlier quoted context omitted.

Here's one rewrite that would have helped: To be fair, as you can see in the clip, the two models handled the prompt slightly differently. The pxpipe variant gave the right count initially but needed a quick follow-up to output the ledger balance in a single line. The standard model, on the other hand, nailed the formatting on its first try. We've completely solved readability here on Fable; our only real hurdle left…

Yeah, but as you say, this is another LLM rewriting it. The amount of noncontextual information is nauseating and destroys the point of a README (in my humble opinion). A human might have written a disclaimer like this: > When not using Fable, pxpipe may require additional follow-ups to precisely follow your formatting instructions. This kind of garbled information dump is very inconsiderate of the reader, and all go…

To be fair to the rewriting LLM, it was given only that single paragraph, with no other context and asked only to make it more comprehensible. But your point still stands. Here's what it said about _your_ rewrite:

That human rewrite is excellent. It ruthlessly cuts out the "narrative" of the test case (the transaction counts, the video clip, the "first try vs second try" details) and extracts the only piece of information an actual user reading a README cares about: what to expect when using the software.

Which suggests some ideas that should have been included in the prompt, to get closer to your ideal rewrite.

Post reply on HN