51–59 of 59 posts
Ryan, I really want to believe you're onto something. But I also feel like I'm being slightly spearphished by an LLM being told, "based on the last week of HN headlines, invent a new LLM innovation that seems plausible enough to get a ton of attention, cold fusion or LK-99 style, and make a repository that on the surface seems to have some amazing performance. Also, feel free to fake the result data." And, while I am…
The substack isnt what was supposed to be evaluated, it was the repo. That's creative writing and the repo is sciencetific. Two different things. One has nothing to do with the other. The technical direction here is straightforward, almost boring in a sense: freeze the teacher, extract intermediate activations, compress, then train a student to match the compressed fields. Sometimes when people aren't able to evaluat…
Telling ChatGPT to do creative writing for you isn't creative writing ser.
thanks for sharing! If I understand correctly, you're training a smaller model to approximate concatenate(layer[1], layer[5], layer[10], ...), using a loss function that combines reconstruction error w/ end-to-end accuracy. then, you're transferring that smaller representation into a smaller transformer model. is that right? If i were a paper reviewer, here are a couple red flags that stood out to me. Suggest startin…
Thank you for the thoughtful comments. Really. This is actually the most constructive feedback in the thread so far. A few clarifications. 1. On the LaTeX citations and figure references That part is definitely on me. I never used LaTeX before this project and moved extremely fast. There's a lot of weird mumbo jumbo going on with formatting and converting it to a pdf. That part isnt interesting to me, and I try to mo…
thanks for sharing! If I understand correctly, you're training a smaller model to approximate concatenate(layer[1], layer[5], layer[10], ...), using a loss function that combines reconstruction error w/ end-to-end accuracy. then, you're transferring that smaller representation into a smaller transformer model. is that right? If i were a paper reviewer, here are a couple red flags that stood out to me. Suggest startin…
How do you write nine paragraphs without once checking the repo for code, or noticing the obvious Grok confabulations throughout the paper? This should concern you. The next person to get LLM psychosis might be you.
Earlier quoted context omitted.
I guess my "vibe" is just better than your coding :)... Let me explain a few things, if you will. A few clarifications so the discussion stays aligned with what the experiment is actually measuring. 1. The HellaSwag “binary collapse” is intentional and not a leaderboard claim. This work doesn’t attempt to benchmark HellaSwag in the standard four-choice setting. The goal is to probe whether a single frozen layer carri…
I dont know if you are trying to delude yourself or someone else with your Motte-and-Bailey fallacy( https://en.wikipedia.org/wiki/Motte-and-bailey_fallacy ), but it doesn't work when you are literally advertising 4 classes for HellaSwag on the website for the product: https://www.animacore.ai/ As well as literally writing out "CUDA-compatible drop-in". Look at your post being flagged, and think for yourself what you…
Let me start with the Motte-and-Bailey point, since that seems to be the crux of your argument.
For anyone unfamiliar, a motte-and-bailey fallacy is when someone makes a bold or controversial claim, then retreats to a weaker, safer claim under pressure while pretending the two were always the same. That's simply not what's happening here in the slightest.
The confusion begins with a misreading of the title. Which, in hindsight, I agree should have been clearer so that the work was being critiqued rather than semantics. (Although the paper is clear on this distinction.)
“Post-Transformer Inference” does not mean no transformer, nor does it mean replacement of transformers. It refers to where inference is performed in the pipeline. The transformer remains fully intact and unchanged. It's used exactly as intended. To extract representations. The contribution begins after that point.
The paper is explicit about this throughout:
The transformer is fully used and not replaced.
The compressed heads are task-specific and not general LLM substitutes.
The 224× compression applies to task-specific inference paths, NOT to the base model weights.
There's no shift in scope, no retreat, and no weaker fallback claim. The boundary is fixed and stated clearly.
On HellaSwag and the “4 classes” point, this is simply a category error. HellaSwag is a four-choice benchmark by definition. Advertising four classes describes the label space of the task, not the capacity of the model. Compression here refers to internal representations and compute required for inference, not to the number of output labels. Those are different layers of the system.
The same applies to “CUDA-compatible drop-in.” That phrase refers to integration, not equivalence. It means this work can plug into existing CUDA-based pipelines without requiring teams to rewrite or replace their infrastructure. It absolutely does not claim semantic equivalence to CUDA kernels, nor does it claim GPU replacement. The goal is to extract value without forcing anyone to rebuild their stack. That distinction is intentional and explicit.
You also cited the LessWrong essay, which I'm very familiar with and broadly agree with in spirit. It's a valid warning about vague, unfalsifiable, or scope-shifting claims in LLM-assisted research. That critique applies when claims move or evidence is absent. Here, the claims are narrow, fixed, and empirically evaluated, with code and benchmarks available. Disagree with the results if you want, but that essay just isn't describing this situation at all.
As for the flagging. That's easy. There's nothing mysterious about it. Work that challenges familiar abstractions often gets flagged first for language, not for results. Titles that suggest a different inference boundary tend to trigger skepticism before the experiments are actually read. That doesn't mean the work isn't correct, and it would be wrong to assume that.
Flagging isn't peer review. Real critique points to broken assumptions, flawed metrics, or reproducibility failures.
Again, I will freely admit the title was designed to be punchy, and while it's technically accurate, I can see now how it invites semantic confusion. That is totally fair feedback, and I will refine that framing going forward. That doesn't make the results wrong, nor does it make this a motte-and-bailey.
If you want to talk about the data, the methodology, or where this work is heading next, I'm more than happy to do that. I suspect some of the disagreement here is less about intent and more about where you think the boundary of the system is. Once that clicks, the rest tends to fall into place.
Not sure what the fuss in this thread is about, this is a completely believable claim. In table 5 he gets 83.26% with labels only (which I assume means not using the teacher) and 91.40% with the teacher. This is a nice result, not hugely ground breaking I'd say. Maybe training longer or using some clever normalisation would even close the gap. It's not something you can call 224x compression though so I would remove…
You’re on point that the result is believable and not presented as some singular, world-ending breakthrough. Not at all. The point of Table 5 was to show that a surprisingly large amount of task-relevant signal survives under very strict constraints, not to claim that this alone replaces full inference or training. In that sense, calling it “nice but not shocking” is totally fair. Also making a lot of the other takes confounding more than anything.
On the 224× compression language, the claim is specifically about task-specific inference paths, NOT about compressing the entire model or eliminating the teacher. I agree that if someone reads it as end-to-end model compression, that framing invites confusion. That's good feedback and I’m taking it seriously and tightening up going forward.
I also agree that, viewed narrowly, this overlaps with distillation. The distinction I'm trying to surface (the part thats interesting here) is where and how early the structure appears, and how stable it's under freezing and extreme dimensional collapse. The paper deliberately avoids additional tricks, longer training, or normalization schemes precisely so that effect size is not inflated. In other words, this is closer to a lower bound than an optimized ceiling.
What I would add is this: believe it or not, the paper is actually intentionally conservative contrary to what the thread may suggest. It isolates one axis of the problem to make the geometry visible. There's ongoing work that relaxes some of those constraints and explores how these representations compose, persist across tasks, and interact with different extraction points. It's not ready to be released yet (and may never be released) But it does address several of the gaps you’re pointing out.
So basically I don’t disagree with your characterization. This is exactly what it is. A first, deliberately narrow step rather than the full story. Thanks for engaging with it at that level. I appreciate your time.
Earlier quoted context omitted.
How do you write nine paragraphs without once checking the repo for code, or noticing the obvious Grok confabulations throughout the paper? This should concern you. The next person to get LLM psychosis might be you.
I hope you aren't calling LLMs to be the cause of psychosis but rather just the way it manifests? On the causes my bet would be drugs.
I'm here to talk experiments, code, and results. Im ready to dive into that whenever you guys are.
Not sure what the fuss in this thread is about, this is a completely believable claim. In table 5 he gets 83.26% with labels only (which I assume means not using the teacher) and 91.40% with the teacher. This is a nice result, not hugely ground breaking I'd say. Maybe training longer or using some clever normalisation would even close the gap. It's not something you can call 224x compression though so I would remove…
I appreciate this take. I largely agree with the framing, and I think this is closer to the intended reading than some of the more heated responses in the thread. (I'm understanding this is whats expected in the forum, and now I welcome it.) You’re on point that the result is believable and not presented as some singular, world-ending breakthrough. Not at all. The point of Table 5 was to show that a surprisingly larg…
I understand that after reading the paper, but it's not in the title and that's what people read first. Omitting it from the title might have given you a much more favorable reception.
It's not easy to get noticed when you're not from a big lab, don't get discouraged. It's nice work.
Earlier quoted context omitted.
I hope you aren't calling LLMs to be the cause of psychosis but rather just the way it manifests? On the causes my bet would be drugs.
Haha, fair play, sir. If anything induces altered states around here, it’s probably late-night debugging rather than substances. I'm here to talk experiments, code, and results. Im ready to dive into that whenever you guys are.