Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

60% Fable cost cut by converting code to images and having the model OCR it

github.com

Post reply on HN
Print view View original

111–115 of 115 posts

Re: 60% Fable cost cut by converting code to images and having the model OCR it

#111

Post by numeri »

nazgul17 wrote: post #105
hn_throwaway_99 wrote: post #30

Earlier quoted context omitted.

This is really fascinating to me. I was reading this article and originally agreed with you, "I mean, under the covers it's got to be converting to text tokens at some point, so there is no way it's actually cheaper for Claude itself to execute." But then there is a comment below talking about how DeepSeek was able to get a huge improvement in compression by using visual tokens, https://news.ycombinator.com/item?id=4…

That was DeepSeek OCR, not a DeepSeek lineage LLM. If the idea is introduced in LLMs, then you're right. But Gemini is not doing that, not yet. This is something I literally discussed with Claude last week, but took its word for it.

Deep seek OCR is an LLM, just one trained/post-trained specifically for OCR.

Exact details of text to image compression ratios are of course extremely dependent on the model architecture, training data, training objectives, etc., so there's probably not too much justification for generalizing to all models

Re: 60% Fable cost cut by converting code to images and having the model OCR it

#112

Post by numeri »

ptx wrote: post #81
Klathmon wrote: post #61

Earlier quoted context omitted.

That's like saying quantization isn't real because the frontier labs aren't using it in their production inference. This is a lossy process, it produces worse results. It might be worth it for some situations, but applying it to everything would just be making your SOTA model worse

Isn't this just quantization with extra steps? Can converting the text to an image really be a better way to lossily compress it? (Not that I have any idea what I'm talking about on this topic.)

No, quantization is applied to model weights or the KV cache (the model activations of all past tokens), and is just storing everything with lower precision (carefully, so that it doesn't hurt performance much).

Sending an image of text instead of text reduces the number of input tokens, but they're still being processed by the model at the same precision. This probably also hurts performance in some way – the question is by how much.

Re: 60% Fable cost cut by converting code to images and having the model OCR it

#113

Post by brumar »

parsimo2010 wrote: post #86
brumar wrote: post #84

Tangentially related: I don't think OCR is the right term and I am generally vocal about that. But seeing this unquestioned here, I am wondering if I am the one who is wrong here. Is it ok to call this OCR? To me ocr means text in the end, not visual tokens.

OCR means optical character recognition. The terms do not require a direct transcription, but that is mostly what OCR meant in the past. If you’re using an LLM’s vision capability to pass in text and the LLM actually understands it, then I would say that it recognized the characters, hence OCR seems okay to use.

But characters only exist when we ask the model about this and it does it best to do this projection if asked. Vision model are richer than that. It "understands" visually a document. If it was only about characters, then there will be no way it beats the traditional pipelines of image->text->extractions or obtain the kind of results we see in this article. Vision models are more than characters recognition and OCR term don't do it justice IMHO.

Re: 60% Fable cost cut by converting code to images and having the model OCR it

#114

Post by TZubiri »

netsharc wrote: post #109
TZubiri wrote: post #38

Earlier quoted context omitted.

then it's no longer an image, as the one in the github repo, you would be encoding the text as characters and sending it as an image. You can achieve this by changing the extension of an image file from .bmp to .txt Guys, not to be mean, but maybe chill with the state of the art research and go back to studying fundamentals.

If you want to be a know-it-all poseur, at least back it up with data.

https://chatgpt.com/share/6a4a7ea3-65e8-83e9-8871-5041d97bb9...

Download the .bmp file and open it with 'edit' on windows or 'nano' on linux. You should see text. The bytes that precede the text are padding and a header that roughly 'tells how to interpret the data', in the sense that it describes the dimensions of the 'image'.

There's python code that shows how the file was generated, note that it just writes a string to the file and then wraps it with a .bmp header so that it can be interpreted by image viewing programs.

If you open that image with an image viewer you will see an image with colorful pixels. Whether you send it as an image or as text, the only difference lies in what you declare the file to be. For example if you change the extension, operating systems will "open" it with the appropriate program by default, but the data is the same, there's no difference between ASCII text and the encoding you describe where each subpixel describes one character, it's the same thing.

Your idea about using 7 bits instead of 8bits is more space efficient, but you'd need to decompress it so that the text is byte aligned. In terms of costs, you are not charged by the bandwidth consumed in transit, but by the computational cost, which wouldn't change as the text would need to be expanded to 8bits per char, (and then converted to the same token anyways)

Again don't want to be mean but these are 1st year comp sci topics, not at all related to LLMs. Hence why I recommend studying fundamentals more than papers that look like bleeding edge LLM research.

Post reply on HN