Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

60% Fable cost cut by converting code to images and having the model OCR it

github.com

Post reply on HN
Print view View original

51–60 of 115 posts

Re: 60% Fable cost cut by converting code to images and having the model OCR it

#51

Post by yorwba »

rightbyte wrote: post #39
calebkaiser wrote: post #24

Earlier quoted context omitted.

Nah, optical compression is a thing. You see it in a lot of different areas in ML. In this case, the "trick" has been known for a while, and belongs to a whole world of compression research. But I think where you're maybe getting mixed up is in where that 60% gain is coming from. It's not a 60% percent reduction in cost for 100% of the same output. If you have a model and input text A, and you fix the seed etc. and r…

> a compression strategy To me compression implies smaller size? However new line chars seems to be removed in the pic so I guess it could be expressed in fewer bytes than the original text with further compression ...

The size is indeed smaller, because text tokens and image tokens are embedded as vectors of the same size, but text tokens typically only cover a few characters, while image tokens typically cover many pixels, so many that you can fit more characters in there. So the same text takes up fewer tokens as an image, and hence requires less time and memory to process.

You could also imagine models where text tokens cover many characters and image tokens just a few pixels, which would invert the relationship, but this is typically suboptimal for the applications people have in mind when they train a model.

Re: 60% Fable cost cut by converting code to images and having the model OCR it

#52

Post by NooneAtAll3 »

DANmode wrote: post #34
hn_throwaway_99 wrote: post #30

Earlier quoted context omitted.

This is really fascinating to me. I was reading this article and originally agreed with you, "I mean, under the covers it's got to be converting to text tokens at some point, so there is no way it's actually cheaper for Claude itself to execute." But then there is a comment below talking about how DeepSeek was able to get a huge improvement in compression by using visual tokens, https://news.ycombinator.com/item?id=4…

It wouldn’t, they’re subsidizing it for training. Edit: didn’t realize this occurred on local models(!!), this is smarter https://news.ycombinator.com/item?id=48779884

can't explain with subsidies a model you host yourself (like deepseek)

Re: 60% Fable cost cut by converting code to images and having the model OCR it

#53

Post by Gooblebrai »

aabhay wrote: post #10

In Gemini at least, if you look at how they process PDFs, they do an OCR and then feed the text + image to the model, without charging you for the text tokens (I believe). So my guess is that Claude’s backend is doing the same — so this hack is probably more of a loophole in token accounting that might get closed if Claude is doing what Gemini does

Claude Science has a tool to extract the PDF but not sure if it's OCR'ing it.

Re: 60% Fable cost cut by converting code to images and having the model OCR it

#55

Post by OSaMaBiNLoGiN »

Saw a Tweet a while ago from someone (maybe Carmack, maybe Geohot, maybe Karpathy?) wondering if images were just the better option.

Since then I've been using images with very simply worded prompts whenever I'm informing an agent of what is happening. Sometimes no text in the prompt at all.

It has been very very effective.

That being said, this isn't really what Karpathy was talking about. But it got me thinking a bit, and that got me to a much nicer workflow.

Re: 60% Fable cost cut by converting code to images and having the model OCR it

#56

Post by cindyllm »

OSaMaBiNLoGiN wrote: post #55

Saw a Tweet a while ago from someone (maybe Carmack, maybe Geohot, maybe Karpathy?) wondering if images were just the better option. Since then I've been using images with very simply worded prompts whenever I'm informing an agent of what is happening. Sometimes no text in the prompt at all. It has been very very effective. That being said, this isn't really what Karpathy was talking about. But it got me thinking a b…

[dead]

Re: 60% Fable cost cut by converting code to images and having the model OCR it

#58

Post by measurablefunc »

NooneAtAll3 wrote: post #52
DANmode wrote: post #34

Earlier quoted context omitted.

It wouldn’t, they’re subsidizing it for training. Edit: didn’t realize this occurred on local models(!!), this is smarter https://news.ycombinator.com/item?id=48779884

can't explain with subsidies a model you host yourself (like deepseek)

Then you are paying for the electricity. It's not physically possible to do more computation & not use more energy b/c every arithmetic operation requires a minimum amount of energy so more operations = more energy.

Re: 60% Fable cost cut by converting code to images and having the model OCR it

#59

Post by gamblor956 »

legel wrote: post #14
TZubiri wrote: post #12

Earlier quoted context omitted.

Of course it isn't A text encoding uses 8bits per character on average, tokenization further compresses that An image font would be 25 bits if 5x5, and most fonts are 12 pixels high Of course it isn't efficient, this is a pricing inefficiency and a hack to exploit it (even the author describes it as an exploit)

You are wrong. Text tokens are high-dimensional vectors, not 8 bits per character. Every token has a deep embedding, e.g. 1024 float values per text token. DeepSeek-OCR proved 10x+ compression from visual embedding of text, which was a groundbreaking result. [1] Very cool to see OP's project hacking on this principle. It's still not lossless, as noted in the github, but is a promising research direction. [1] https://…

People really need to read their cites and not just the summaries.

The paper notes two things:

1) While the compression ratio for visual text is better than it is for regular text, but the absolute space required is still higher for the images. OPs were talking about the space required, not the ratio.

2) The results of the OCR must still be fed into a text-based LLM for linguistic processing. Otherwise, all you have achieved is turning an image into a bunch of text.

Re: 60% Fable cost cut by converting code to images and having the model OCR it

#60

Post by [deleted] »

measurablefunc wrote: post #58
NooneAtAll3 wrote: post #52

Earlier quoted context omitted.

can't explain with subsidies a model you host yourself (like deepseek)

Then you are paying for the electricity. It's not physically possible to do more computation & not use more energy b/c every arithmetic operation requires a minimum amount of energy so more operations = more energy.

[deleted]
Post reply on HN