421–430 of 536 posts
Earlier quoted context omitted.
The usage of LLMs is continuing to increase ~exponentially. I'm going to bet on that rather than some half-baked scenario analysis that only takes into account one scenario and assigns a 100% probability to it.
> The usage of LLMs is continuing to increase ~exponentially I would like a source for that statement. Additionally, I want to know by who? Because it certainly isn't end users. Inflating token usage doesn't make it any more economically viable if your user base, b2b or not, hasn't increased with it. On the contrary, that is a worse scenario for providers.
1. As a consultant pretty much every company I have worked with in the last 2 years are doing some kind of in-house "AI Revolution", I'm talking making "AI Taskforce" teams, having weekly internal "AI meetings" and pushing AI everywhere and to everyone. Small companies, SMEs and huge companies. From my observation it is mainly due to C-level being obsessed by the idea that AI will replace/uplift people and revenue will grow by either replacing people or launching features 10x quicker.
2. Did you see software job-boards recently? 9/10 (real) job listings are to do with AI. Either it is fully AI company (99% thin wrapper over Anthropic/OpenAI APIs) or some other SME that needs some AI implementations done. It is truly a breath of fresh air to work for companies that have nothing to do with AI.
The biggest laugh/cry for me are those thin wrappers that go down overnight - think all the "create your website" companies that are now completely useless since Ahtropic cut the middleman and created their own version of exactly that.
Earlier quoted context omitted.
AMD has built some consumer GPUs in the recent past with HBM - RX Vega and Radeon VII (although I assume not all "HBM" is created equal).
Isn't their APU also capable of doing HBM? There was an Intel AMD hybrid chip that used unified a while back too.
Earlier quoted context omitted.
> Free lists aren't expensive for most usage patterns. Whatever little CPU they waste is often worth more than the RAM they save. > For cases where they are we've got stuff like arena allocators. ... that work by using more RAM to save on CPU.
GC burns far more CPU cycles. Meanwhile I'm not sure where you got this idea about the value of CPU cycles relative to RAM. Most tasks stall on IO. Those that don't typically stall on either memory bandwidth or latency. Meanwhile CPU bound tasks typically don't perform allocations and if forced avoid the heap like the plague.
Far less for moving collectors. That's why they're used: to reduce the overhead of malloc/free based memory management. The whole point of moving collectors is that they can make the CPU cost of memory management arbitrarily low, even lower than stack allocation. In practice it's more complicated, but the principle stands.
The reason some programs "avoid the heap like the plague" is because their memory management is CPU-inefficient (as in the case of malloc/free allocators).
> Meanwhile I'm not sure where you got this idea about the value of CPU cycles relative to RAM
There is a fundamental relationship between CPU and RAM. As we learn in basic complexity theory, the power of what can be computed depends on how much memory an algorithm can use. On the flip side, using memory and managing memory requires CPU.
To get the most basic intuition, let's look at an extreme example. Consider a machine with 1 GB of free RAM and two programs that compute the same thing and consume 100% CPU for their duration. One uses 80MB of RAM and runs for 100s; the other uses 800MB of RAM and runs for 99s (perhaps thanks to a moving collector). Which is more efficient? It may seem that we need to compare the value of 1% CPU reduction vs a 10x increase in RAM consumption, but that's not necessary. The second program is more efficient. Why? Because when a program consumes 100% of the CPU, no other program can make use of any RAM, and so both programs effectively capture all 1GB, only the second program captures it for one second less.
This scales even to cases when the CPU consumption is less than 100% CPU, as the important thing to realise is that the two resources are coupled. The thing that needs to be optimised isn't CPU and RAM separately, but the RAM/CPU ratio. A program can be less efficient by using too little RAM if using more RAM can reduce its CPU consumption to get the right ratio (e.g. by using a moving collector) and vice versa.
Earlier quoted context omitted.
The Radeon VII came out in 2019 as a $700 consumer GPU with an 1TB/s HBM2 memory subsystem which is more than any consumer GPU you can get today, including the high-end ones afaik. At that point in time, there was a whole lineup of AMD GPUs with HBM going down into the midrange. If they could make this stuff and sell it to regular people a decade ago for very palatable prices, why do they come up with the idea that t…
I have been wondering this recently. It was the convention that if you wanted to keep costs down, try to keep the memory bus size down as low as possible. Still remember the awful Radeon 9200 SE - 64bit data bus that strangled an already slow GPU. Heck, I have a phone with a 16bit memory bus for instance. The high(ish) clock rate only makes up the difference slightly. But with general prices on all components going u…
Designing a part with a wide bus and putting the traces down on the board is what I would expect to be the easy part these days (surely).
But yield, yield comes for us all.
Earlier quoted context omitted.
Is that relevant? The grid in every country is getting ridiculously stressed by datacenters.
High-level, I would agree with you. One thing that blows me away: I think I read that Northern Virginia, USA has the highest data center density in the world. Mostly it is due to demand from US gov't, military, and spy agencies (like NSA). How did they do it? In mainstream media, I don't see any news about a stressed power grid in this area. I guess the US gov't carefully coordinated with local power providers to con…
That's because you don't live in Maryland.
Our energy bills are through the roof and our transmission company is talking about rolling blackouts in 2027.
https://www.thebanner.com/community/climate-environment/cont...
Earlier quoted context omitted.
> If AI makes software easier to create, that will drive the price down. Supposedly AI drives down the cost of producing software,not the "price". > How are software companies going to make enough revenue to pay for AI, when the amount of money being spent on AI is already multiples of the current total global expenditure on software? Currently, the cost of AI is between $20/month and around $200/month per developer.…
You need >64 GB of DRAM to run local models fast . You can run huge local models slowly with the weights stored on SSDs. Nowadays there are many computers that can have e.g. 2 PCIe 5.0 SSDs, which allow a reading throughput of 20 to 30 gigabyte per second, depending on the SSDs (or 1 PCIe 5.0 + 1 PCIe 4.0, for a throughput in the range 15-20 GB/s). There are still a lot of improvements that can be done to inference b…
Batching inferences doesn't necessarily help that much since as models get sparser the individual inferences are going to share fewer experts. It does always help wrt. shared routing layers, of course.
Earlier quoted context omitted.
So as I said, a self interested metric who also controls how many tokens it takes to get a desirable result from their models.
Users are willingly paying for larger volumes of tokens. You are layering your own unproven interpretation onto that. I would have arrived at an opposite interpretation given the available facts. Models are becoming more token efficient for the same task, such as ChatGPT 5.3 versus 5.2 which halved the token count, and capabilities show a log relationship with the number of tokens since o1 preview was revealed in Sep…
Earlier quoted context omitted.
GC burns far more CPU cycles. Meanwhile I'm not sure where you got this idea about the value of CPU cycles relative to RAM. Most tasks stall on IO. Those that don't typically stall on either memory bandwidth or latency. Meanwhile CPU bound tasks typically don't perform allocations and if forced avoid the heap like the plague.
> GC burns far more CPU cycles Far less for moving collectors. That's why they're used: to reduce the overhead of malloc/free based memory management. The whole point of moving collectors is that they can make the CPU cost of memory management arbitrarily low , even lower than stack allocation. In practice it's more complicated, but the principle stands. The reason some programs "avoid the heap like the plague" is be…
Earlier quoted context omitted.
then it must be the case you can't get one (for a fair price?)
Define "fair price" 5090s are certainly expensive compared to most other GPUs, but not expensive enough to be unobtanium for nearly any professional who could utilize one as part of their job