171–180 of 221 posts
Earlier quoted context omitted.
This is discussed in the lex Friedman episode. AMD’s own demo would kernel panic when run in a loop [1]. [1] https://youtube.com/watch?v=dNrTrx42DGQ&t=3218
Interesting. I wonder if focusing on GPUs and CPUs is something that requires two companies instead of one, whether the concentration of resources just leads to one arm of your company being much better than the other.
Earlier quoted context omitted.
> 4 times the bandwidth you would get from a PCIe Gen 7 x16 link So you have a full terabyte per second of bandwidth? What GPU is that? (The 64GB/s number is an x4 link. If you meant you have over four times that, then it sounds like CXL would be pretty competitive.)
The 3090 Ti and 4090 both have 1.01TB/sec memory bandwidth: https://www.techpowerup.com/gpu-specs/geforce-rtx-3090-ti.c3...
Earlier quoted context omitted.
The 4090 offers 82.58 teraflops of single-precision performance compared to the Radeon Pro VII's 13.06 teraflops.
On the other hand, for double precision a Radeon Pro VII is many times faster than a RTX 4090 (due to 1:2 vs. 1:64 FP64:FP32 ratio). Moreover, for workloads limited by the memory bandwidth, a Radeon Pro VII and a RTX 4090 will have about the same speed, regardless what kind of computations are performed. It is said that speed limitation by memory bandwidth happens frequently for ML/AI inferencing.
Earlier quoted context omitted.
Yeah but MLID says they are losing money on every one and have been winding down the internal development resources. That doesn't bode well for the future. I want to believe he's wrong, but on the parts of his show where I am in a position to verify, he generally checks out. Whatever the opposite of Gell-Mann Amnesia is, he's got it going for him.
Wait, are they losing money on every one in the sense that they haven't broken even on research and development yet? Or in the sense that they cost more to manufacture than they're sold at? Because one is much worse than the other.
https://news.ycombinator.com/item?id=42505496
That being said, the IP blocks are shared by their iGPUs, so the discrete GPUs do not need to recoup the costs of most of the R&D, as it would have been done anyway for the iGPUs.
Earlier quoted context omitted.
I've recently been poking around with Intel oneAPI and IPEX-LLM. While there are things that I find refreshing (like their ability to actually respond to bug reports in a timely manner, or at all) on a whole, support/maturity actually doesn't match the current state of ROCm. PyTorch requires it's own support kit separate from the oneAPI Toolkit (and runs slightly different versions of everything), the vLLM xpu suppor…
I've recently been trying to get IPEX working as well, apparently picking Ubuntu 24.04 was a mistake, because while things compile, everything fails at runtime. I've tried native, docker, different oneAPI versions, threw away a solid week of afternoons for nothing. SYCL with llama.cpp is great though, at least at FP16 since it supports nothing else but even Arc iGPUs easily give 2-4x performance compared to CPU infer…
I will only consider AMD GPUs for LLM when I can easily make my AMD GPU available within WSL and Docker on Windows. For now, it is as if AMD does not exist in this field for me.
$ rocminfo | grep -E "WSL|XTX"
WSL environment detected.
Marketing Name: AMD Radeon RX 7900 XTXEarlier quoted context omitted.
My testing has been w/ a Lunar Lake Core 258V chip (Xe2 - Arc 140V) on Arch Linux. It sounds like you've tried a lot of things already, but case it helps, my notes for installing llama.cpp and PyTorch: https://llm-tracker.info/howto/Intel-GPUs I have some benchmarks as well, and the IPEX-LLM backend performed a fair bit better than the SYCL llama.cpp backend for me (almost +50% pp512 and almost 2X tg128) so worth get…
Well that's funny, I think we already spoke on Reddit. I'm the guy who was testing the 125H recently. I guess there's like 5 of us who have intel hardware in total and we keep running into each other :P Honestly I think there's just something seriously broken with the way IPEX expects the GPU driver to be on 24.04 and there's nothing I can really do about it except wait for them to fix it if I want to keep using this…
Earlier quoted context omitted.
Yeah but MLID says they are losing money on every one and have been winding down the internal development resources. That doesn't bode well for the future. I want to believe he's wrong, but on the parts of his show where I am in a position to verify, he generally checks out. Whatever the opposite of Gell-Mann Amnesia is, he's got it going for him.
MLID on Intel is starting to become the same as UserBenchmark on AMD (except for the generally reputable sources)... he's beginning to sound like he simply wants Intel to fail, to my insider-info-lacking ears. For competition's sake I really hope that MLID has it wrong (at least the opining about the imminent failure of Intel's GPU division), and that the B series will encourage Intel to push farther to spark more co…
https://news.ycombinator.com/item?id=42505496
The margins might be describable as razor thin, but they are there. Whether it can recoup the R&D that they spent designing it is hard to say definitively since I do not have numbers for their R&D costs. However, their iGPUs share the same IP blocks, so the iGPUs should be able to recoup the R&D costs that they have in common with the discrete version. Presumably, Intel can recoup the costs specific to the discrete version if they sell enough discrete cards.
While this is not a great picture, it is not terrible either. As long as Intel keeps improving its graphics technology with each generation, profitability should gradually improve. Although I have no insider knowledge, I noticed a few things that they could change to improve their profitability in the next generation:
* Tom Petersen made a big deal about 16-lane SIMD in Battlemage being what games want rather than the 8-lane SIMD in Alchemist. However, that is not quite true since both Nvidia and AMD graphics use 32-lane SIMD. If the number of lanes really matter and I certainly can see how it would if game shaders have horizontal operations, then a switch to 32-lane SIMD should yield further improvements.
* Tom Petersen said in his interview with Hardware Unboxed that Intel reported the active transistor count for the B580 rather than the total transistor count. This is the contrary to others who report the total transistor count (as evidenced by their density figures being close to what TSMC claims the process can do). Tom Petersen also stated that they would not necessarily be forced by defects to turn dies into B570 cards. This suggests to me that they have substantial redundant logic in the GPU to prevent defects from rendering chips unusable, and that logic is intended to be disabled in production. GPUs are already highly redundant. They could drop much of the planned dark silicon and let defects force a larger percentage of the dies to be usable by only cutdown models.
Also, they really do have >40% dark silicon when doing density comparisons:
* ARC B580: 72.1M / mm²
* Nvidia 4070 Ti: 121.8M / mm²
* TSMC claim for 5nm: 138.2M / mm²
Note that we can rule out the cache lowering the density. The L1 + L2 cache on the 4070 Ti is 79872 KB while it is 59392 KB on the B580. We can also rule out IO logic as lowering the density, as the 4070 Ti has a 256-bit memory bus while the B580 has a 192-bit memory bus.
https://www.techpowerup.com/gpu-specs/arc-b580.c4244
https://www.techpowerup.com/gpu-specs/geforce-rtx-4070-ti.c3...
https://en.wikipedia.org/wiki/5_nm_process#Nodes
The hardware unboxed interview of Tom Petersen is here:
A used 3090 is $600-900, performs better than 7900, and is much more versatile because CUDA
Reality check for anyone considering this: I just got a used 3090 for $900 last month. It works great. I would not recommend buying one for $600, it probably either won’t arrive or will be broken. Someone will reply saying they got one for $600 and it works, that doesn’t mean it will happen if you do it. I’d say the market is realistically $900-1100, maybe $800 if you know the person or can watch the card running fir…
Earlier quoted context omitted.
Gigabytes per second? What is this, bandwidth for ants? My years old pleb tier non-HBM GPU has more than 4 times the bandwidth you would get from a PCIe Gen 7 x16 link, which doesn't even officially exist yet.
Yes CXL will soon benefit from PCIe Gen 7 x16 with expected 64GB/s in 2025 and the non-HBM bandwidth I/O alternative is increasing rapidly by the day. For most inferences of near real-time LLM it will be feasible. For majority of SME companies and other DIY users (humans or ants) with their localized LLM should not be any issues [1],[2]. In addition new techniques for more efficient LLM are being discover to reduce t…
https://github.com/ryao/llama3.c/blob/master/run.c
First, CXL is useless as far as I am concerned.
The smaller LLM stuff in 1 and 2 is overrated. LLMs get plenty of things wrong and while the capabilities of small LLMs is increasing, they just are never as good as the larger LLMs in my testing. To give an example, between a small LLM that gets things right 20% of the time and a large one that gets things right 40% of the time, you are never going to want to deal the small one. Even when they improvement you will just find new things that they are not able to do well. At least, that is my experience.
Finally, the 75% savings figure in 3 is misleading. It applies to the context, not the LLMs themselves. It is very likely that nobody will use it, since it is a form of lossy compression that will ruin the ability of the LLM to repeat what is in its memory.