> Claude Telenovela
Nice, hadn't seen this one before.
LLMs Are Complicated Now
61–70 of 86 posts
Re: LLMs Are Complicated Now
#62It's the bitter-lesson to feature-engineering lifecycle. When a technique or technology is new people are making massive gains by just applying it to some use case, or gathering more data for training, or giving it more resources. As time goes on those "bitter lesson" gains start to hit the shallow part of the logistic curve and companies have to start investing more and more effort into engineering for each small, i…
I assume the choice of phrase "bitter lesson" is intentional irony (since the original concept is that you get better results by just scaling up and not trying to be clever with domain-specific knowledge)?
I assume that the bitterness of the bitter lesson is not for engineers but for subject matter experts. I can only imagine how it would feel to discover that your decades of hard-earned expertise don't amount to a whole lot when it comes to domain-specific ML modeling, compared to simply throwing more compute at the problem.
Re: LLMs Are Complicated Now
#63It's the bitter-lesson to feature-engineering lifecycle. When a technique or technology is new people are making massive gains by just applying it to some use case, or gathering more data for training, or giving it more resources. As time goes on those "bitter lesson" gains start to hit the shallow part of the logistic curve and companies have to start investing more and more effort into engineering for each small, i…
I assume the choice of phrase "bitter lesson" is intentional irony (since the original concept is that you get better results by just scaling up and not trying to be clever with domain-specific knowledge)?
Maybe the ultimate bitter lesson is that entropy always wins in the end.
Re: LLMs Are Complicated Now
#64Earlier quoted context omitted.
I assume the choice of phrase "bitter lesson" is intentional irony (since the original concept is that you get better results by just scaling up and not trying to be clever with domain-specific knowledge)?
Maybe the ultimate bitter lesson is that entropy always wins in the end.
[dead]
Re: LLMs Are Complicated Now
#65Earlier quoted context omitted.
Just like em-dashes, some people have always done these though. Why are they penalized with immediate AI slop witch hunts? The LLMs didn't come up with these tics out of thin air.
> ust like em-dashes, some people have always done these though. Everytime someone claims that they have always written like this I grab a pre-2022 post of theirs and five both to a few SOTA chatbots and ask "did the same writer author both these texts". Thus far I have never gotten a "likely" response. If the author truly did not use an AI to write something, then it is more likely that theybhave spent so much time…
I've used pseudo-em dashes going back to the 90s. In that I'll use hyphen (-) a lot. So for me, an actual em dash would be a tell, but if I just have a normal hyper in between two thoughts, it's not.
And yes, I agree that most people who light up on AI tell scans are indeed using AI. That's not my point.
Re: LLMs Are Complicated Now
#66Re: LLMs Are Complicated Now
#67It's the bitter-lesson to feature-engineering lifecycle. When a technique or technology is new people are making massive gains by just applying it to some use case, or gathering more data for training, or giving it more resources. As time goes on those "bitter lesson" gains start to hit the shallow part of the logistic curve and companies have to start investing more and more effort into engineering for each small, i…
I assume the choice of phrase "bitter lesson" is intentional irony (since the original concept is that you get better results by just scaling up and not trying to be clever with domain-specific knowledge)?
My read of GP is that there is a cycle economy between "bitter lesson" style domain-general scaling and domain-specific adaptation once the scaling plateaus for the latest tech.
Re: LLMs Are Complicated Now
#68Why didn't this author compare Llama 3 with GLM 5.2 (released 1 week ago) which is a more standard attention based LLM? To compare 2 separate families of LLMs and then pointing out that they are different is not a surprising result and detracts from the point the author is trying to make. https://sebastianraschka.com/llm-architecture-gallery/?compa... If you look at it, the diagrams are very similar, but the main dif…
> Why didn't this author compare Llama 3 with GLM 5.2 (released 1 week ago) which is a more standard attention based LLM? To compare 2 separate families of LLMs and then pointing out that they are different is not a surprising result and detracts from the point the author is trying to make. The entire point of the comparison is that LLMs look vastly different today than before. Comparing more similar LLMs would detra…
It is misleading the reader since most current LLMs look the same as before. It is cherry picking an example to make a point when it's not necessary at all to make the argument he is trying to make.
Re: LLMs Are Complicated Now
#69Earlier quoted context omitted.
[[citation needed]] I am a professional writer and have been for over 30 years. (I do not use any form of LLM ever.) This means I read a lot . This also means that I have 30+ years of experience of readers not understanding what I wrote, or not getting further than the title, or not getting the main message, or inverting it in their heads, or inserting their own message and then complaining when I diverge, and an end…
I’m not sure if it is written by an LLM, but anything being called “load-bearing” (formatted that way and all) sets off my alarm bells
Hopefully you don't work in construction!
Re: LLMs Are Complicated Now
#70Why didn't this author compare Llama 3 with GLM 5.2 (released 1 week ago) which is a more standard attention based LLM? To compare 2 separate families of LLMs and then pointing out that they are different is not a surprising result and detracts from the point the author is trying to make. https://sebastianraschka.com/llm-architecture-gallery/?compa... If you look at it, the diagrams are very similar, but the main dif…
The author is correct, the model architecture is now much more complicated. You can see this if you use llama.cpp and follow the project. The earlier models were always fully implemented. Yet with more contributors, as of today tons of latest models only have partial implementation. DeepSeekv3.2 isn't fully implemented, same with KimiK2.6, GLM5.2+, DeepSeekv4 has no implementation, MiniMaxM3 not supported yet, Hy3-pr…
The architecture is not much more complicated. And llama.cpp not implementing something is more likely an issue of its business model and financial incentives than it being due to raw complexity.