171–180 of 213 posts
OK this was a bit funny: Top HW failure modes: * GPU falling off the bus I honestly thought "do they mean GPUs falling off a bus entering the data center" and then realized its actually the connectivity, as they mention in the next line GPUs falling off: In this case, GPUs are not detected by the host on PCIe.
OK this was a bit funny: Top HW failure modes: * GPU falling off the bus I honestly thought "do they mean GPUs falling off a bus entering the data center" and then realized its actually the connectivity, as they mention in the next line GPUs falling off: In this case, GPUs are not detected by the host on PCIe.
The bits on the bus go round and round! There is a lot of interesting yet unpublished work on 'data center' scale compute complexes. It was a rabbit hole I fell into several times while at Google.
[0] https://research.google/pubs/the-datacenter-as-a-computer-an...
> Since we did not have time to change the cooling infrastructure, we had to remain in an air-cooled environment. The mechanical and thermal designs had to change to accommodate this, and that triggered a validation cycle to support a large-scale deployment. > All of these hardware-related changes were challenging because we had to find a solution that fit within the existing resource constraints, with a very small d…
The last tech team to have no budget and time constraints to pursue their vision? I don’t know, the Xanadu team? Romero’s original Daikatana team?
Earlier quoted context omitted.
Much the same way you can have all the best gear and still fail - Google’s primary strength seems to be the DeepMind group. I’m not affiliated with Google, but IMHO the reason they will slowly die is because their engineering culture has taken a backseat due to their broken hiring practices. Bad hiring practices aren’t exclusive to them, but from all accounts it seems like their internal focus is on optimizing ad rev…
>ecause their engineering culture has taken a backseat due to their broken hiring practices. What company do you think has better hiring practices, and subsequently a higher talent pool? Meta is pretty similar to Google's (though with an emphasis on speed over creativity). Microsoft is certainly worse at hiring than the two aforementioned...
Earlier quoted context omitted.
With temperature set to 1, it recognizes the joke, but proceeds to explain what the "bus" is in computer terms, picks a problem this prompt could mean, and explains how to solve it. In ~20 tries it always gave me something along the lines of: The infamous "GPU falling off the bus" issue! This problem typically occurs when a graphics processing unit (GPU) is not properly seated or connected to its expansion slot, such…
+1, interesting findings! I like how it was able to infer the meaning from such a short phrase in a limited context.
Earlier quoted context omitted.
+1, interesting findings! I like how it was able to infer the meaning from such a short phrase in a limited context.
It's actually a very common phrase on forums, I think because it's an actual error that Linux will report: https://askubuntu.com/questions/868321/gpu-has-fallen-off-th... . I've also never heard of it, but it seems like it must appear a lot in the training data and probably about 0 times is referring to a bus on the road.
--
Earlier quoted context omitted.
They do sell shovels, you can get Google TPUs on Google Cloud.
Exactly and they are still about 1/18ths as good at training llms as a H100. Maybe they are less than 1/18ths the cost, so google technically have a marginally better unit cost but i doubt it when you consider the R&D cost. They are less bad at inference, but still much worse than even an A100.
"Deployed since 2020, TPU v4 outperforms TPU v3 by 2.1x and improves performance/Watt by 2.7x. ... For similar sized systems, it is ~4.3x--4.5x faster than the Graphcore IPU Bow and is 1.2x--1.7x faster and uses 1.3x--1.9x less power than the Nvidia A100. TPU v4s inside the energy-optimized warehouse scale computers of Google Cloud use ~2--6x less energy and produce ~20x less CO2e than contemporary DSAs in typical on-premise data centers."
Here is a link to the paper: https://dl.acm.org/doi/pdf/10.1145/3579371.3589350
Earlier quoted context omitted.
It's actually a very common phrase on forums, I think because it's an actual error that Linux will report: https://askubuntu.com/questions/868321/gpu-has-fallen-off-th... . I've also never heard of it, but it seems like it must appear a lot in the training data and probably about 0 times is referring to a bus on the road.
In my testing, both Llama 3 and its abliterated (uncensored) variant from[0] almost always remarked more or less directly that they see the joke in the phrase, so either they've seen the other meaning in training, or inferred it. -- [0] - https://news.ycombinator.com/item?id=40665721
OK this was a bit funny: Top HW failure modes: * GPU falling off the bus I honestly thought "do they mean GPUs falling off a bus entering the data center" and then realized its actually the connectivity, as they mention in the next line GPUs falling off: In this case, GPUs are not detected by the host on PCIe.
A GPU falling off the bus would be one mega flop
Earlier quoted context omitted.
Exactly and they are still about 1/18ths as good at training llms as a H100. Maybe they are less than 1/18ths the cost, so google technically have a marginally better unit cost but i doubt it when you consider the R&D cost. They are less bad at inference, but still much worse than even an A100.
If you're interested in a peer reviewed scientific comparison, Google writes retrospective papers after contemporary TPUs and GPUs are deployed versus speculation about future products. The most recent compares TPU v4 and A100. (TPU v5 and H100 is for a future paper). Here is a quote from the abstract: "Deployed since 2020, TPU v4 outperforms TPU v3 by 2.1x and improves performance/Watt by 2.7x. ... For similar sized…
Which sure makes the H100 sound both faster and more efficient (per unit of compute) than the TPU v4, given what was in your quote. I don't think your quote does anything to support the position that TPUs are noticeably better than Nvidia's offerings for this task.
Complicating this is that the TPU v5 generation has already come out, and the Nvidia B100 generation is imminent within a couple of months. (So, no, a comparison of TPUv5 to H100 isn't for a future paper... that future paper should be comparing TPUv5 to B100, not H100.)
[0]: https://developer.nvidia.com/blog/nvidia-hopper-architecture...