81–90 of 125 posts
Earlier quoted context omitted.
What's this all look like without an atmosphere?
Worse, heat dissipation is a major constraint for spacecraft and satellites because you can only radiate heat away as infrared photons.
I am interested in the more hardware management side of HPC (how problems are detected, diagnosed, mapped into actions such as reboot/reinstall/repairs, how these are scheduled and how that is optimized to provide the best level of service, how this is done if there are multiple objectives to optimize at once e.g. node availability vs overall throughput, how different topologies affect the above, how other constraint…
Meta also publishes a number of papers/blogs/OSS projects on their engineering site [2]
James Hamilton of AWS gives a talk most years on their infrastructure. Worth watching multiple years [3].
[1] https://youtu.be/69PrhWQorEM?si=u7vh_Um6SQNoyeFH
[2] https://engineering.fb.com/category/data-center-engineering/
Earlier quoted context omitted.
It always made me wonder why liquid cooling wasn't more of a thing for datacenters. Water has a massive amount of thermal capacity and can quickly and in bulk be cooled to optimal temperatures. You'd probably still need fans and AC to dissipate heat of non-liquid cooled parts, but for the big energy items like CPUs and GPUs/compute engines, you could ship out huge amounts of heat fairly quickly and directly. I guess…
Because it’s complex. Even more complex than “engineered” air. You need two circuits, and a CDU between them. Coolants needs maintaining. You add antifreeze, biocides, etc. Air is brute force. It cools everything it touches. Liquid cooling is serialized in a node. Two sockets? Second will be hotter. HBA not making good contact? It’ll overheat. You add extensive leak detection subsystems, the amount of coolant moving…
I've seen tests done on heavy PC loops (ie multi-GPU) both high-flow and low-flow, as well as on car engines, in different coolant flow configurations. The results from all of those are that the water doesn't rise meaningfully in temperature between components.
Unless I did my back-of-the-napkin math wrong, this seems reasonable. If you have a single 10mm ID pipe going through a 1U server and up to the next, then for a full 42U rack you have about 1.7kg of water going through the servers. If the flow rate is about 1s per server (so 42 seconds for the full rack) and each 1U server dumps 500W of energy into the water, there should be just a 3 degree C difference in the water temperature between the first and the last server.
Earlier quoted context omitted.
Because it’s complex. Even more complex than “engineered” air. You need two circuits, and a CDU between them. Coolants needs maintaining. You add antifreeze, biocides, etc. Air is brute force. It cools everything it touches. Liquid cooling is serialized in a node. Two sockets? Second will be hotter. HBA not making good contact? It’ll overheat. You add extensive leak detection subsystems, the amount of coolant moving…
> Liquid cooling is serialized in a node. Two sockets? I've seen tests done on heavy PC loops (ie multi-GPU) both high-flow and low-flow, as well as on car engines, in different coolant flow configurations. The results from all of those are that the water doesn't rise meaningfully in temperature between components. Unless I did my back-of-the-napkin math wrong, this seems reasonable. If you have a single 10mm ID pipe…
Earlier quoted context omitted.
Could you link me the YouTube videos/articles in question? It happens to be my research area and I'm interested in knowing how big companies such as meta deal with multi-GPU systems
I don't have them bookmarked anymore, but they may have been from this playlist: [0] [0] https://www.youtube.com/playlist?list=PLBnLThDtSXOw_kePWy3CS...
Earlier quoted context omitted.
> Liquid cooling is serialized in a node. Two sockets? I've seen tests done on heavy PC loops (ie multi-GPU) both high-flow and low-flow, as well as on car engines, in different coolant flow configurations. The results from all of those are that the water doesn't rise meaningfully in temperature between components. Unless I did my back-of-the-napkin math wrong, this seems reasonable. If you have a single 10mm ID pipe…
In our system every node gets inlet water at the same temperature via parallel piping, but when it’s in node, it goes through processors first, then RAM, then PCIe and disks. Delta T between two sockets is 5 degrees C, and the delta T between input and output is around 15-18 C depending on load.
> Delta T between two sockets is 5 degrees C
And secondly, ~5-10 degrees is what I see on my dual-socket workstation, and have been wondering about this delta ever since the first day I started monitoring the temperatures. At first, I thought that the heat sink wasn't installed properly but after reinstalling it the delta remained. Since I didn't notice any CPU throttling or whatsoever I figured it's "normal" and ignored it.
Earlier quoted context omitted.
In our system every node gets inlet water at the same temperature via parallel piping, but when it’s in node, it goes through processors first, then RAM, then PCIe and disks. Delta T between two sockets is 5 degrees C, and the delta T between input and output is around 15-18 C depending on load.
First, thanks for sharing these details, I find them fascinating because they are not so common to be read or heard about. > Delta T between two sockets is 5 degrees C And secondly, ~5-10 degrees is what I see on my dual-socket workstation, and have been wondering about this delta ever since the first day I started monitoring the temperatures. At first, I thought that the heat sink wasn't installed properly but after…
I mean, water travels from one socket to another, so one processor adds heat equal to 5 degrees C under nominal load. The second socket doesn’t complain much, but this is enormous amounts of heat transferred in a such quick pace.
Earlier quoted context omitted.
This seemed like a big topic when I was interviewing with Meta and nVidia some months ago. Meta had a few good YouTube videos about the problems of dealing with this many GPUs at scale.
Could you link me the YouTube videos/articles in question? It happens to be my research area and I'm interested in knowing how big companies such as meta deal with multi-GPU systems
[1]. https://techcommunity.microsoft.com/t5/microsoft-mechanics-b...
Earlier quoted context omitted.
What's this all look like without an atmosphere?
Worse, heat dissipation is a major constraint for spacecraft and satellites because you can only radiate heat away as infrared photons.
Earlier quoted context omitted.
> It always made me wonder why liquid cooling wasn't more of a thing for datacenters. Liquid cooling is almost a defacto-standard in data centers in the HPC world. The Top of the TOP500 machines are all liquid cooled. Not by choice, but due to physics constraints. There is a big gap in power density between the HPC world and the usual datacenter-commodity-hardware world. Commodity DS are designed with the assumption…
> we attempted to install a medium size HPC cluster in a well-known commerical Datacenter and network provider. The commercial of the DS almost fall from his chair when we announced the power requirements. Heh. We tried it too. They didn’t believe that a single node used their entire rack’s budget at first.