Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

60% Fable cost cut by converting code to images and having the model OCR it

github.com

Post reply on HN
Print view View original

61–70 of 115 posts

Re: 60% Fable cost cut by converting code to images and having the model OCR it

#61

Post by Klathmon »

geor9e wrote: post #23
vineyardmike wrote: post #20

Earlier quoted context omitted.

> Some random person discovered a 60% across the board gain in all LLMs, using an extremely simple trick that none of the labs noticed in all these years of multi-trillion dollar growth DeepSeek published a pretty well circulated paper on exactly this many months ago. It just hasn’t been attempted and shared publicly, asa retrofit, AFAIK. Also, it’s no free lunch, the readme indicates that this “use images” hack is l…

If the trick were genuinely useful, and was well circulated months ago, the resource-starved inference providers would have squeezed this trick dry already, instead of wasting 60% of their tokens, waiting for users to implement it themselves in 5 minutes of effort.

That's like saying quantization isn't real because the frontier labs aren't using it in their production inference.

This is a lossy process, it produces worse results. It might be worth it for some situations, but applying it to everything would just be making your SOTA model worse

Re: 60% Fable cost cut by converting code to images and having the model OCR it

#62

Post by yorwba »

hn_throwaway_99 wrote: post #30
aabhay wrote: post #10

In Gemini at least, if you look at how they process PDFs, they do an OCR and then feed the text + image to the model, without charging you for the text tokens (I believe). So my guess is that Claude’s backend is doing the same — so this hack is probably more of a loophole in token accounting that might get closed if Claude is doing what Gemini does

This is really fascinating to me. I was reading this article and originally agreed with you, "I mean, under the covers it's got to be converting to text tokens at some point, so there is no way it's actually cheaper for Claude itself to execute." But then there is a comment below talking about how DeepSeek was able to get a huge improvement in compression by using visual tokens, https://news.ycombinator.com/item?id=4…

LLMs have a very bloated in-memory representation for text, on the order of megabytes of KV cache per byte of text. Meanwhile, for images a lossy representation is considered acceptable and it only takes up maybe a kilobyte of KV cache per byte of image. So if you can render your text into a hundred bytes of image per byte of text and then lossily expand it into 100 kB of KV cache per byte of text, you come out ahead!

Whether such lossy compression is acceptable for your use case is up to you.

Re: 60% Fable cost cut by converting code to images and having the model OCR it

#63

Post by trueno »

aabhay wrote: post #5

Ahhh my eyes the vibe coded readme

dead giveaway for me that something someone made and wants to share = they dont understand what they put together enough to like speak to it with some level of authority.

people can make some really useful stuff with AI especially when its a domain they're already an expert in, and it would go a long ways for them to just sit and explain that 1. they used AI to help 2. their own words to explain what the heck they put together, especially if they can speak to some of the limitations AI has working with it. just goes a long way to demonstrate this guys stuff is worth tinkering with because he has a good grasp on what was created

for 99% of the stuff out there now people are literally operating in domains they don't understand at all, i just close my tab when i see the damn vibe coded readmes

Re: 60% Fable cost cut by converting code to images and having the model OCR it

#65

Post by cwmoore »

georgemcbay wrote: post #48
quantummagic wrote: post #45

Earlier quoted context omitted.

Here's one rewrite that would have helped: To be fair, as you can see in the clip, the two models handled the prompt slightly differently. The pxpipe variant gave the right count initially but needed a quick follow-up to output the ledger balance in a single line. The standard model, on the other hand, nailed the formatting on its first try. We've completely solved readability here on Fable; our only real hurdle left…

Reads a little bit better, but still reads like a writer getting paid by the word, which I guess is fitting.

Or maybe: like a copywriter, paid by the dopamine hit

Re: 60% Fable cost cut by converting code to images and having the model OCR it

#66

Post by jayd16 »

yorwba wrote: post #51
rightbyte wrote: post #39

Earlier quoted context omitted.

> a compression strategy To me compression implies smaller size? However new line chars seems to be removed in the pic so I guess it could be expressed in fewer bytes than the original text with further compression ...

The size is indeed smaller, because text tokens and image tokens are embedded as vectors of the same size, but text tokens typically only cover a few characters, while image tokens typically cover many pixels, so many that you can fit more characters in there. So the same text takes up fewer tokens as an image, and hence requires less time and memory to process. You could also imagine models where text tokens cover m…

So split the difference and start encoding input at the words or phrases level?

Re: 60% Fable cost cut by converting code to images and having the model OCR it

#67

Post by mrbnprck »

electrotype wrote: post #44

What about: "Read this document online : [URL]" and you add your text/context to an online document? Would that reduce the number of tokens used too?

Documents are processed as tokens as well, unless its bitmap is ocr'd.

Images tho are natively compatible with Multi-Modal LLMs, so theres no image->text translation layer in between. It's that the unit of cost is different (e.g. "visual token" vs text token)

Re: 60% Fable cost cut by converting code to images and having the model OCR it

#69

Post by satvikpendem »

hn_throwaway_99 wrote: post #30
aabhay wrote: post #10

In Gemini at least, if you look at how they process PDFs, they do an OCR and then feed the text + image to the model, without charging you for the text tokens (I believe). So my guess is that Claude’s backend is doing the same — so this hack is probably more of a loophole in token accounting that might get closed if Claude is doing what Gemini does

This is really fascinating to me. I was reading this article and originally agreed with you, "I mean, under the covers it's got to be converting to text tokens at some point, so there is no way it's actually cheaper for Claude itself to execute." But then there is a comment below talking about how DeepSeek was able to get a huge improvement in compression by using visual tokens, https://news.ycombinator.com/item?id=4…

Baidu released a faster OCR model as well: https://github.com/baidu/Unlimited-OCR
Post reply on HN