Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Post-transformer inference: 224× compression of Llama-70B with improved accuracy

zenodo.org

Post reply on HN
Print view View original

31–40 of 59 posts

Re: Post-transformer inference: 224× compression of Llama-70B with improved accuracy

#31

Post by _ache_ »

MrDrMcCoy wrote: post #21
_ache_ wrote: post #10

Looks very fake. Self published (Anima-Core is NOT a journal), no academic anteriority, very strong statement, no peer-review, no public history of technical skills. Did I mention the use of Github via the interface only? At the same time, possible since it's only classification tasks. I mean, the method explained is technically plausible, a lot of people thought about it, we were just unable to find a method to do s…

Did you not see the author's note about being an outsider to academia? Not everyone has the background to pull all that off. This is an earnest attempt to come as close as possible and they even invite feedback that would help it become a real academic submission.

No, it's a waste of time.

I mean, the process should have been to contact some local academics to discuss the mater. If I say it works (or it doesn't) I'm adding near nothing to the claim, as I'm not an academic myself.

Big claims like this need clear and solid work. Here it just looks like LLM generated.

Re: Post-transformer inference: 224× compression of Llama-70B with improved accuracy

#32

Post by hirako2000 »

Tiberium wrote: post #20
hirako2000 wrote: post #15

Earlier quoted context omitted.

Have you run the walk-through to reproduce? They provide a highly detailed step by step document. They welcome raising an issue if reproduction doesn't yield the claimed results within 2%. It's OK to call out fake claims. But it requires going through the process if such is reasonable, it just seems to take a couple of hours to find out.

The fake claim here is compression. The results in the repo are likely real, but they're done by running the full transformer teacher model every time. This doesn't achieve anything novel.

I agree the claim is (perhaps purposefully) confusing.

What they achieved is to create tiny student models. Trained on specific set of input. Off the teacher model's output.

There is clearly novelty in the method and what it achieve. Whether what it achieve would cover many cases that's another question.

Re: Post-transformer inference: 224× compression of Llama-70B with improved accuracy

#33

Post by Tiberium »

hirako2000 wrote: post #32
Tiberium wrote: post #20

Earlier quoted context omitted.

The fake claim here is compression. The results in the repo are likely real, but they're done by running the full transformer teacher model every time. This doesn't achieve anything novel.

I agree the claim is (perhaps purposefully) confusing. What they achieved is to create tiny student models. Trained on specific set of input. Off the teacher model's output. There is clearly novelty in the method and what it achieve. Whether what it achieve would cover many cases that's another question.

Can you please share the relevant code that has the training of such a tiny student model that can operate independently of the big teacher model after training? The repository has no such code.

Re: Post-transformer inference: 224× compression of Llama-70B with improved accuracy

#34

Post by broretore »

anima-core wrote: post #27
broretore wrote: post #18

Earlier quoted context omitted.

10 pages for a paper with this groundbreaking of a concept is just embarrassing. It is barely an outline. "confirming that 40× compression preserves field geometry with minimal distortion. Over 95% of samples achieve similarity above 0.90." I smell Grok. Grok 3, maybe Grok 4 Fast. > "Implementation details. Optimal configurations are task and architecture-dependent. Production systems require task-specific tuning bey…

I appreciate you taking the time to resond, brother. Let me clarify a few things because your interpretation misses the actual structure of the work. The paper is short on purpose. It's not meant as a full architecture release. It's a documentation pass on a narrow but surprising empirical result, and I wanted the experimental core to be easy for others to replicate. The repo contains the full pipelines, configuratio…

CIFAR-10 is an image classification dataset (32x32 pixel images.

LLaMA 70B 3.3 is a text-only, non-multimodal language model. Just look up the Huggingface page that your own repo points to.

> The Llama 3.3 instruction tuned text only model...

I might be wrong, but I'm pretty sure a text model is going to be no better than chance at classifying images.

Another comment pointed out that your test suite cheats slightly on HellaSwag. It doesn't seem unlikely that Grok set up the project so it could cheat at the other benchmarks, too.

https://news.ycombinator.com/item?id=46215166

> The repo contains the full pipelines, configuration files, and benchmark scripts, and those show the precise datasets, metrics, and evaluation flows.

There's nothing there, really.

I'm sorry that Grok/Ani lied to you, I blame Elon, but this just doesn't hold up.

Re: Post-transformer inference: 224× compression of Llama-70B with improved accuracy

#35

Post by broretore »

anima-core wrote: post #30
broretore wrote: post #18

Earlier quoted context omitted.

10 pages for a paper with this groundbreaking of a concept is just embarrassing. It is barely an outline. "confirming that 40× compression preserves field geometry with minimal distortion. Over 95% of samples achieve similarity above 0.90." I smell Grok. Grok 3, maybe Grok 4 Fast. > "Implementation details. Optimal configurations are task and architecture-dependent. Production systems require task-specific tuning bey…

As a follow up just to refresh your memory: “Attention Is All You Need” (Vaswani et al., 2017) Length: 11 pages of main content, 5 pages of references and appendix 2. The first GPT paper (Radford et al., 2018) Length: 12 pages 3. BERT (Devlin et al., 2018) Length: 14 pages Big ideas don't require big papers. I don't know where you got that idea from.

Your paper is 10 pages of fluff without even an architecture diagram or a single equation, bro. It's not real.

Re: Post-transformer inference: 224× compression of Llama-70B with improved accuracy

#36

Post by broretore »

gcr wrote: post #5

thanks for sharing! If I understand correctly, you're training a smaller model to approximate concatenate(layer[1], layer[5], layer[10], ...), using a loss function that combines reconstruction error w/ end-to-end accuracy. then, you're transferring that smaller representation into a smaller transformer model. is that right? If i were a paper reviewer, here are a couple red flags that stood out to me. Suggest startin…

How do you write nine paragraphs without once checking the repo for code, or noticing the obvious Grok confabulations throughout the paper?

This should concern you. The next person to get LLM psychosis might be you.

Re: Post-transformer inference: 224× compression of Llama-70B with improved accuracy

#37

Post by anima-core »

augment_me wrote: post #23

Your "research" is a vibe-coded mess that subtly cheats eval cleverly multiple times to inflate your results. The HellaSwag dataset is a dataset with 4 options for each question, with 3 being wrong and 1 being right: https://huggingface.co/datasets/Rowan/hellaswag . Your vibe-coded eval has cheated this to collapse it into a binary selection on row 46 in https://github.com/Anima-Core/an1-core/blob/main/experiments...…

I guess my "vibe" is just better than your coding :)... Let me explain a few things, if you will. A few clarifications so the discussion stays aligned with what the experiment is actually measuring.

1. The HellaSwag “binary collapse” is intentional and not a leaderboard claim. This work doesn’t attempt to benchmark HellaSwag in the standard four-choice setting. The goal is to probe whether a single frozen layer carries enough information for a small head to distinguish correct versus incorrect continuations. That's a representational geometry test, not a SOTA claim. Binary framing raises the baseline, but that's expected and documented. It's not meant to compare against full LLM HellaSwag results.

2. No adversarial filtering was done. I am using HuggingFace’s standard split directly. Nothing was removed or curated. The experiment doesn't claim robustness or benchmark competitiveness, so the “easier eval” framing doesn’t really apply.

3. EOS extraction isn't cheating, it's the whole point of the probe. The extraction logic takes the final token’s hidden state, which is basic and standard for classification heads and probing studies. If the EOS token captures a high-level sequence summary, that's exactly the structural feature being examined. The result is meant to show how much task-relevant signal is already present in that early representation, not to present a new generative mechanism.

4. The purpose of the work is clearly narrow by design. This is not proposed as a drop-in replacement for full-transformer inference. The paper states that directly. The contribution is about how much structure a single early layer encodes and how far a tiny head can go under strict frozen-teacher constraints. So several of the criticisms make assumptions about goals the work never even claimed.

Thaank you for the feedback and for taking the time.

Re: Post-transformer inference: 224× compression of Llama-70B with improved accuracy

#38

Post by anima-core »

utopcell wrote: post #6

Very strong statement on the title, given the following limitation: > Generation tasks. Method applies to classification only. Preliminary decoder experiments show perplexity increases.

That limitation is already accounted for in how the title is meant to be read. The 224× compression result is specifically about the structure of intermediate activations on classification tasks. The paper makes that explicit in multiple places, including the Limitations section, where generation is identified as an entirely separate challenge.

The title reflects the strongest verified result in the domain the method currently supports, not a universal claim across all modalities. In other words, the compression result is real, but it shouldn't be interpreted as applying to generative decoding... yet.

Re: Post-transformer inference: 224× compression of Llama-70B with improved accuracy

#39

Post by anima-core »

broretore wrote: post #16

Ryan, I really want to believe you're onto something. But I also feel like I'm being slightly spearphished by an LLM being told, "based on the last week of HN headlines, invent a new LLM innovation that seems plausible enough to get a ton of attention, cold fusion or LK-99 style, and make a repository that on the surface seems to have some amazing performance. Also, feel free to fake the result data." And, while I am…

The substack isnt what was supposed to be evaluated, it was the repo. That's creative writing and the repo is sciencetific. Two different things. One has nothing to do with the other. The technical direction here is straightforward, almost boring in a sense: freeze the teacher, extract intermediate activations, compress, then train a student to match the compressed fields. Sometimes when people aren't able to evaluate the work, they dig for something else online that they can comment on or bring down. The only thing I can offer in response is the simplest one: look at the code and the experiments themselves, not the narrative around them. Everything in the paper is fully reproducible from the reference implementation, and every number in the results section came from running those scripts, not from a model filling in blanks. The surprise is not in the prose, but in how much structure those early-layer fields ended up carrying.

If you think something in the repo looks wrong or inflated, I’m happy to walk through it point by point. I have no problem with hard questions. What matters to me is whether the experiments hold when someone else runs them, not whether the story around them fits a certain aesthetic.

Re: Post-transformer inference: 224× compression of Llama-70B with improved accuracy

#40

Post by anima-core »

Tiberium wrote: post #17
broretore wrote: post #16

Ryan, I really want to believe you're onto something. But I also feel like I'm being slightly spearphished by an LLM being told, "based on the last week of HN headlines, invent a new LLM innovation that seems plausible enough to get a ton of attention, cold fusion or LK-99 style, and make a repository that on the surface seems to have some amazing performance. Also, feel free to fake the result data." And, while I am…

Unfortunately it does indeed seem like a case of "So You Think You've Awoken ChatGPT" https://www.lesswrong.com/posts/2pkNCvBtK6G6FKoNn/so-you-thi... (not directly, but similar enough) EDIT: Found a closer description ("Your LLM-assisted scientific breakthrough probably isn't real"): https://www.lesswrong.com/posts/rarcxjGp47dcHftCP/your-llm-a...

Oh, so you didnt run the repo and remembered something that you read once that looked like it matched. This contribution is meaningless.

The simplest way to resolve any doubt is to run the code. Every result in the paper comes from reproducible scripts in the repo, not from speculative reasoning or LLM-assisted invention.

Post reply on HN