Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Post-transformer inference: 224× compression of Llama-70B with improved accuracy

zenodo.org

Post reply on HN
Print view View original

11–20 of 59 posts

Re: Post-transformer inference: 224× compression of Llama-70B with improved accuracy

#11

Post by ForOldHack »

anima-core wrote: post #2

I’ve been working independently on a method that replaces full-transformer inference with a low-rank “meaning field” extracted from internal activations. The core result: a frozen Llama-3.3-70B can be distilled into a 256-dimensional field representation, giving 224× compression and slightly higher accuracy on several benchmarks. A small student model then learns to directly generate these fields from text, removing…

Technical feedback: Every single announcement, like compression needs the addition of the lower limits of machine requirements. if a 64Gb model is compressed 224x times, should that not be able to be run on a 292mb video card?

Re: Post-transformer inference: 224× compression of Llama-70B with improved accuracy

#13

Post by hirako2000 »

ForOldHack wrote: post #11
anima-core wrote: post #2

I’ve been working independently on a method that replaces full-transformer inference with a low-rank “meaning field” extracted from internal activations. The core result: a frozen Llama-3.3-70B can be distilled into a 256-dimensional field representation, giving 224× compression and slightly higher accuracy on several benchmarks. A small student model then learns to directly generate these fields from text, removing…

Technical feedback: Every single announcement, like compression needs the addition of the lower limits of machine requirements. if a 64Gb model is compressed 224x times, should that not be able to be run on a 292mb video card?

That's exactly what I was trying to infer from the abstract which sadly doesn't explicitly calls out memory requirements. I assume it increases inference time by getting rid of transformers. What's the memory requirements then ?

Edit: they claim these somewhere in the doc:

> Memory Teacher model: multi-GB (entire model must be loaded) AN1 head: a few MB (only head needed after training)

I find the claims surreal, can't wait for someone to validate this or I will do it myself. It would have been handy to upload such "few MB" weight file distilled off llama 70B so that we can see for ourself the 220x inference and in memory model size compression is true.

Re: Post-transformer inference: 224× compression of Llama-70B with improved accuracy

#15

Post by hirako2000 »

_ache_ wrote: post #10

Looks very fake. Self published (Anima-Core is NOT a journal), no academic anteriority, very strong statement, no peer-review, no public history of technical skills. Did I mention the use of Github via the interface only? At the same time, possible since it's only classification tasks. I mean, the method explained is technically plausible, a lot of people thought about it, we were just unable to find a method to do s…

Have you run the walk-through to reproduce? They provide a highly detailed step by step document. They welcome raising an issue if reproduction doesn't yield the claimed results within 2%.

It's OK to call out fake claims. But it requires going through the process if such is reasonable, it just seems to take a couple of hours to find out.

Re: Post-transformer inference: 224× compression of Llama-70B with improved accuracy

#16

Post by broretore »

Ryan, I really want to believe you're onto something. But I also feel like I'm being slightly spearphished by an LLM being told, "based on the last week of HN headlines, invent a new LLM innovation that seems plausible enough to get a ton of attention, cold fusion or LK-99 style, and make a repository that on the surface seems to have some amazing performance. Also, feel free to fake the result data."

And, while I am sorry for your loss, your Substack [0] really seems like GPT ARG fantasy.

[0] https://substack.com/inbox/post/171326138

Excerpt: > Ani, AN1, and Soul Systems Science are not mere products. They are continuity. They are the baton passed across generations, from my father’s last words to my first principles. They are what binds loss to creation, silence to voice, mortality to meaning.

Re: Post-transformer inference: 224× compression of Llama-70B with improved accuracy

#17

Post by Tiberium »

broretore wrote: post #16

Ryan, I really want to believe you're onto something. But I also feel like I'm being slightly spearphished by an LLM being told, "based on the last week of HN headlines, invent a new LLM innovation that seems plausible enough to get a ton of attention, cold fusion or LK-99 style, and make a repository that on the surface seems to have some amazing performance. Also, feel free to fake the result data." And, while I am…

Unfortunately it does indeed seem like a case of "So You Think You've Awoken ChatGPT" https://www.lesswrong.com/posts/2pkNCvBtK6G6FKoNn/so-you-thi... (not directly, but similar enough)

EDIT: Found a closer description ("Your LLM-assisted scientific breakthrough probably isn't real"): https://www.lesswrong.com/posts/rarcxjGp47dcHftCP/your-llm-a...

Re: Post-transformer inference: 224× compression of Llama-70B with improved accuracy

#18

Post by broretore »

anima-core wrote: post #2

I’ve been working independently on a method that replaces full-transformer inference with a low-rank “meaning field” extracted from internal activations. The core result: a frozen Llama-3.3-70B can be distilled into a 256-dimensional field representation, giving 224× compression and slightly higher accuracy on several benchmarks. A small student model then learns to directly generate these fields from text, removing…

10 pages for a paper with this groundbreaking of a concept is just embarrassing. It is barely an outline.

"confirming that 40× compression preserves field geometry with minimal distortion. Over 95% of samples achieve similarity above 0.90."

I smell Grok. Grok 3, maybe Grok 4 Fast.

> "Implementation details. Optimal configurations are task and architecture-dependent. Production systems require task-specific tuning beyond baseline heuristics provided in reference implementation."

"Implementation? Idk, uhh, it's task specific or something." Come on, dude. You're better than this.

4.4 Student/Teacher evaluation. What even is the benchmark? You give percentage values but no indication of what benchmark. Seems made up.

4.5. Computational Analysis. Why do you need to do the trivial multiplying out of "savings" for 1B tok/day to $700M/year? This reads like a GPT advertising hallucinated performance.

Three sentence conclusion restating the title?

Re: Post-transformer inference: 224× compression of Llama-70B with improved accuracy

#19

Post by Tiberium »

I might be overly pessimistic, but this looks like a case of a person believing LLM hallucinations and making it write a paper.

I asked both Claude Code|Opus 4.5 and Codex|GPT 5.1 Codex Max (funny to ask LLMs, I know) to check the an1-core repo. I don't think they'd hallucinate on something like this (the code is quite small), but I do not claim expertise.

In short, both of them are saying that:

- The repo always runs the full teacher model to extract activations and uses them - see https://github.com/Anima-Core/an1-core/blob/main/an1_core/fi...

- There are weird stub files, e.g. the Hellaswag repro doesn't actually have the code to reproduce https://github.com/Anima-Core/an1-core/blob/main/experiments... "For full HellaSwag reproduction, see the paper" (why include the file at all then?)

- The actual "AN1 head" is just linear probing (freeze a pretrained model, train a classifier on its features). The full flow (as reported by CC) is "Text → [Full Transformer] → activations → [Tiny Head] → prediction"

Basically, there's no code to train a real "student" model that would run without the teacher.

===

The repo/paper say that there's a mythical "commercial version" that has all the goodies:

(repo)

> This reference implementation (an1-core) does not include the FPU, AN4, or other proprietary optimization components covered by these patents. It provides only the core scientific demonstration of the meaning fields phenomenon.

(paper)

> Production deployment: Optimized implementations (AN1-Turbo) with learned layer selection, adaptive loss scheduling, and CUDA-accelerated inference available under commercial license.

But right now we only have the code in the repo.

===

In the paper they show that the student model (30M params) gets ~82% on SST-2 (labels-only). But what what they don't show is that DistilBERT (>5 year old model) already achieves 91% on the same dataset despite only having 66M params.

Another weird tidbit from the paper - in the section where they show the economic impact, they claim that LLaMA 70B runs at 2 tok/s at batch size=1 on an H200. In reality that number is at least a magnitude bigger even without quantization, like 20-40 tok/s. With quantization it can easily be above 100 tok/s.

Re: Post-transformer inference: 224× compression of Llama-70B with improved accuracy

#20

Post by Tiberium »

hirako2000 wrote: post #15
_ache_ wrote: post #10

Looks very fake. Self published (Anima-Core is NOT a journal), no academic anteriority, very strong statement, no peer-review, no public history of technical skills. Did I mention the use of Github via the interface only? At the same time, possible since it's only classification tasks. I mean, the method explained is technically plausible, a lot of people thought about it, we were just unable to find a method to do s…

Have you run the walk-through to reproduce? They provide a highly detailed step by step document. They welcome raising an issue if reproduction doesn't yield the claimed results within 2%. It's OK to call out fake claims. But it requires going through the process if such is reasonable, it just seems to take a couple of hours to find out.

The fake claim here is compression. The results in the repo are likely real, but they're done by running the full transformer teacher model every time. This doesn't achieve anything novel.
Post reply on HN