Is This the Dawn of the Tokenpocalypse?
61–70 of 91 posts
Re: Is This the Dawn of the Tokenpocalypse?
#62I think it is easy to make some ragebait doomer articles with eye catching headlines. There are a lot of people who are ready to eat up AI catastrophism because something about apocalyptic predictions and catastrophism seems to attract certain kind of doomer-pilled people. Here are my concrete predictions 1. Token costs will come down and performance will go up 2. Everyone will spend even more on LLMs not less - the…
1. Both costs (going down) and performance (going up) likely are, or will shortly, approach asymptotic limits.
2. CFOs are seeing the token spend on the bottom line, and are not happy. CFOs don't care about 'the next big thing', they just count beans, and they are coming up short. CFOs tend to be the grown-ups in the C-suite, they will shut things down if they need to.
3. See 1. and 2.
Re: Is This the Dawn of the Tokenpocalypse?
#63I see at least two potential positives in this: - The frontier AI companies have realized they won't be able to count on gaining ground and earning more in the future through sheer moat. They have to start earning right now. - The playing field on the market got a whole lot more even as a result. Now everyone is competing on cost and quality - while there are still a lot of competition. AI suppliers can't easily get…
the moat always has been and will be compute and we are fast approaching limits which will be hard to overcome - electricity, chips
Don't you think that we will eventually get more specialized hardware that will greatly improve efficiency? Running neural networks on GPU:s seem like quite wasteful?
Re: Is This the Dawn of the Tokenpocalypse?
#64Earlier quoted context omitted.
the moat always has been and will be compute and we are fast approaching limits which will be hard to overcome - electricity, chips
Don't you think that we will eventually get more specialized hardware that will greatly improve efficiency? Running neural networks on GPU:s seem like quite wasteful?
we will, but data centers will compete for that hardware with retail consumers
Re: Is This the Dawn of the Tokenpocalypse?
#65Earlier quoted context omitted.
Why? What physical limitation will dictate that we can't have 1B tokens for cheap?
Thermodynamics is a harsh mistress
We are very, very far from thermodynamic limits. Lots of people have done the math, and current-gen systems use ~1000000000x more power than the Landauer limit, and ~100000x more power than ideal digital implementation on existing CMOS.
Currently, most AI systems work so that there is a large pool of memory on one side, compute on other side, and a very fat pipe between them. 90%+ of all energy goes into moving data from one side to the other, and selecting the specific element you wish to use from the large pool of ram. The energy cost of holding that data in memory and reading it from the memory cells, and the energy cost of doing the actual computation with low-precision FP are both trivial in comparison.
The systems are built this way because this is the most flexible architecture, and can be used for many different kinds of workloads. But the workload of a transformer in no way requires this flexibility. All the data is fairly local to the execution units that consume it. If you design a system as full PIM, where each ALU is associated and located with the small storage pool that contains only the elements used by that alu, and then tile that out to implement the full model, you cut out most of the energy cost of moving data. The cost is you need much more silicon to implement a working system, but the benefit is not just improved energy-efficiency, but also token speed and silicon efficiency.
The industry is moving towards such designs, with many startups working towards it with different approaches, Nvidia's recent aquisition* of Groq, etc. There is a well-understood path towards ~1000x higher token speeds at ~1000x better energy efficiency, that requires no new innovations, just investment of money into specialization.
There are even more gains if you move the weights into ROM, but that would require you to specialize not just for a specific type of model, but also for a specific set of model weights, ala Taalas.
I find the AI discourse is diseased because on one side you get people breathlessly overestimating the current state of the industry and progress that's going to happen in the next ~2 years, and on the other side people assume that the technology as is is what it will always be and completely ignore that the industry is aware of and actively working towards many ways to improve hardware, it's just that complex leading edge silicon chips take years to take from idea to working products, and transformer inference was only very recently proven to be a market large enough to specialize for.
Re: Is This the Dawn of the Tokenpocalypse?
#66For me it depends on who you listen to If you're following a bunch of people who are from LLM labs, you're going to be more incentivised to tokenmaxx because it's in the Lab's best interest tonget you to behave that way. Practically, many companies aren't labs with endless runway. Companies hopefully follow a PnL model. And when you look at things with that lens, many of the times the LLM use case falls apart. You're…
I'll take the contrary position and say that I think the "tokenmaxxing" we've previously seen was useful (but shouldn't continue indefinitely). My TLDR position is that TokenMaxxing was a way to force discovery of Product Market Fit. The push by companies to incorporate AI into everything is (depending on the company) either hype and cargo-culting or it was an attempt by management to (1) try and discover if/what new…
Usually it’s the seller whose responsibility it is to find PMF, not the buyer.
Re: Is This the Dawn of the Tokenpocalypse?
#67The linked Reddit thread is quite hilarious. Earlier this year my company hired a new CEO and his first company address was solely to tell everyone to use AI or they’d lose their job and become unemployable in general. I knew right there and then that he was a moron. There’s something about American companies where the best and brightest rarely show up in senior management. It seems to be populated by some weird clas…
> Surely, out there in the wilderness, there is a company staffed by intelligent, skilled people. Right?
of course there are but you don't hear about them.
Re: Is This the Dawn of the Tokenpocalypse?
#68Earlier quoted context omitted.
The idea probably was to pour billions into technologies powering these LLMs, and gain a moat. It then turns out that this isn't as hard a problem as expected, it's just very expensive. So as long as you have money, you can be an AI company, the money is the moat (unless you take a shortcut, like DeepSeek) and money is running out. I don't think you're missing anything, but I am surprised that the forces behind the A…
> They do need to start making money, but I don't think anyone has a plan as to how they are going to do this. Doesn't this just mean price increase ? What is not clear is how much the price needs to increase for AI companies to break even some time. 3x increase ? 10x increase ? Even more ? No one seems willing to give a clear number.
They already do it. Antropic hiked prices several times, Google just hiked prices in April/May by a lot (by lowering limits in plans a lot). It will continue regularly. I remember when 200$/m plan was first unveiled there were screams about insanity of it all. Today, if anyone complains about own LLM experience, the first question from the comments would be "are you "at least" paying 200$/m plan, for the poors?" like that is the baseline now and 1000$/m is a serious consideration. And looking at Google, they are slowly shifting whole features upwards in tiers. As does Antropic.
Re: Is This the Dawn of the Tokenpocalypse?
#69I think this is bigger news than many here care to admit. Most companies' AI "strategy" is very naive and depending on subscription based cost calculations. Token-based billing will force those companies to actually think deeper about costs versus benefits. And that's going to be a rude awakening. Other AI providers will follow suit sooner or later. Most probably did by means of intransparent compute cuts and Token throttling.
Re: Is This the Dawn of the Tokenpocalypse?
#70With no details, a bird told me of a project which estimated using several millions of tokens per day to automate a team work which got laid off. The operation is now a mess, there is no one willing to be considered liable and since the cheap model they used is about to be retired the company is going to see a 4x increase in price at least. I have the feeling that the age of 'i can't be blamed by AI stuff' will be a…
Glad I'm not the only one. Almost every factual thing with new opus is wrong (and it now even happens with 4.6?). I asked it about car stuff yesterday and it totally misrepresented how a car axle even looks like fundamentally. Today I talked about my CV and it was just plain wrong. I don't know what happened, it wasn't like this a few weeks back and I'm even considering cancelling claude alltogether. GPT 5.5 for codi…
By differences in the release dates between 4.7 and 4.8 it seems it was more likely an attempted bugfix
But 4.8 still underperforms on most tasks. I have things running where 4o-mini does it considerably better repeatably.
They might have tuned it for a particular reason and I would not doubt that the harness has been made worse.
Sometimes it teases me to think it does wrong things on purpose