Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AMD EPYC Rome 2P Will Have 128-160 PCIe Gen4 Lanes and a Bonus

servethehome.com

Post reply on HN
Print view View original

51–60 of 66 posts

Re: AMD EPYC Rome 2P Will Have 128-160 PCIe Gen4 Lanes and a Bonus

#51

Post by kllrnohj »

velox_io wrote: post #49
kllrnohj wrote: post #48

Earlier quoted context omitted.

The chipset just multiplexes and it's fine. You're unlikely to actually use more than 16 lanes simultaneously in such a setup. But yes, Intel is being overly stingy with PCI-E lanes off of the CPU and leaning heavily on the chipset to compensate. I'm hoping Ryzen 3rd gen bumps the lane count a bit more, otherwise Threadripper's 64 lanes look mighty nice...

The new Ryzens are meant to launch with PCIe 4.0 (this summer, I'm due an upgrade!), so effectively... I'm surprised they don't come with more, especially many of the current ones come with 2-3 x16 slots, AND 2-3 M2s.. Both Intel and AMD have been adding more cores recently. Plus, if you're doing any GPU-based rendering, they're going to simultaneously move data back and forth from NVMe to the GPU's memory. But it gr…

> Plus, if you're doing any GPU-based rendering, they're going to simultaneously move data back and forth from NVMe to the GPU's memory.

Games definitely don't do this at all, which is the primary market for a discreet GPU in these consumer platforms.

When they do stream in assets they do so slowly & in a controlled, rationed amount to minimize impact on FPS. They are far from being PCI-E bandwidth limited. That's kind of why you see almost no FPS drop at all when restricting GPUs to x8 bandwidth, even.

If you're doing something more workstation-y or custom, that'd be when AMD & Intel would point you at the HEDT platforms which have more than 16-20 lanes.

Re: AMD EPYC Rome 2P Will Have 128-160 PCIe Gen4 Lanes and a Bonus

#52

Post by blattimwind »

PedroBatista wrote: post #42
blattimwind wrote: post #38

Earlier quoted context omitted.

While NVMe drives are becoming quite common in regular computers, sadly most software is ill-equipped to handle that sort of I/O bandwidth well. (I would blame OS interfaces for at least three fifths of that, though people who think using textual formats where the most fastest, least correctest parsers top out at 1.5-2 GB/s for bulk data also have their share of blame).

Software moves "slower" because hardware needs to exist 1st and needs to see a widespread adoption in order to justify the cost of development and continuous improvement, also only hardware companies have the expertise because, well, they are the ones who developed the thing. Doing software actually costs a lot, and no one wants to spend that kind of money past the "good-enough" threshold if they don't have to.

Software then moves a lot slower, because most user applications are pretty bad at using more than one core efficiently; and these CPUs have been mainstream for some 15 years now.

The hardware is certainly much more capable than what little use the software tends to make of it.

Re: AMD EPYC Rome 2P Will Have 128-160 PCIe Gen4 Lanes and a Bonus

#53

Post by velox_io »

justinclift wrote: post #17
kilo_bravo_3 wrote: post #14

Earlier quoted context omitted.

Off the top of my head this would be great for massive enterprise VDI installations and game streaming services (which is just VDI by another name), ML, AI applications, and NVMe storage. Personally, this will be good for me because everything I do is bandwidth-starved. I'm not a radar scientist but I am a systems engineer supporting radar scientists working on air- and space-based Synthetic Aperture Radar (SAR) syst…

Hmmm, how many radar pulses need to hit a target to generate video? Sounds potentially irradating... ;) eg: WARNING Don't attempt to image living creatures WARNING

Given the Inverse-square law, your phone and wifi network will probably give you more radiation..

I had some bones scans (plus a couple of CTs the same day, just a bit more ionising radiation) last year. They inject you with Technician 99m and YOU become the gamma/ x-ray source. It's a little concerning when you see the detail and spread of the radiation. If it wasn't a pure Gamma source (and 6hour half-life) it would be lethal!

I forget if it was 70,000 milli or microsieverts (think it was milli? 0.5% increased risk of cancer in my lifetime). Oh, and my bladder looked like a lightbulb.

Re: AMD EPYC Rome 2P Will Have 128-160 PCIe Gen4 Lanes and a Bonus

#54

Post by wmf »

velox_io wrote: post #44
Patrick-STH wrote: post #25

Earlier quoted context omitted.

Here is the easy example: A dual 100GbE NIC requires a PCIe Gen4 x16 slot to run both ports at full speed. Mellanox ConnectX-6 200GbE cards require either 2x Gen4 or 1x Gen5 slot for full dual port bandwidth. Most of the larger NVMEoF nodes we are seeing are using more than one 100GbE NIC. Enterprise SSDs will rapidly fill Gen4 x4 lanes later this year as we start to see them launch alongside more mainstream platform…

Hi Patrick, Before we had M2/U2 NVMe drives, there were quite a few manufacturers making drives that plug straight into the PCIe bus where they could easily access the full 16 lanes, and they still can... Intel seems to be the only ones making PCIe drives (Octane's are funny; they don't look fast only 2000 MB/s, although they do win on latency and random access). Anyway, there doesn't seem to be much demand for that…

There are still a decent number of x8 AIC SSDs on the market but enterprises don't want them because they're not hot swap.

Networking over PCIe exists but no one seems to care about it. https://semiaccurate.com/2014/09/23/look-avagos-expressfabri... Part of the problem may be that 96x8G PCIe switches look pretty small compared to 256x50G Ethernet switches and thus you'd need a lot more of them.

Re: AMD EPYC Rome 2P Will Have 128-160 PCIe Gen4 Lanes and a Bonus

#55

Post by velox_io »

blattimwind wrote: post #38
velox_io wrote: post #7

Earlier quoted context omitted.

Yeah, that's why most PCIe M2 top out at around 3500 MB/s, I doubt it will take long to saturate 7 GB/s (or double that in RAID0). Either way, it's a heck of a lot of bandwidth!

While NVMe drives are becoming quite common in regular computers, sadly most software is ill-equipped to handle that sort of I/O bandwidth well. (I would blame OS interfaces for at least three fifths of that, though people who think using textual formats where the most fastest, least correctest parsers top out at 1.5-2 GB/s for bulk data also have their share of blame).

I wouldn't say it's the bandwidth limit per se. A bigger problem is latency, cache misses have a bigger impact the faster clock speeds go.

Grace Hopper and her nanoseconds (30cm) of wire, doesn't sound so eccentric these days. https://en.wikipedia.org/wiki/Grace_Hopper#Anecdotes

Sadly, the extra bandwidth will come in handy as software developers become evermore slack.

I do agree about parsing text (like json!). It seems crazy to stick to text formats/ protocols when binary would be much faster (it's easy to convert binary to text for debugging purposes, as that's what happens anyway!), we're still not fully utilising http2. (Don't quote me) but I remember hearing somewhere that -10% of computer resources are spent converting between base10 and base2. Have a read of DotNet's Span (reference structs), by not copying values everywhere (and the relieving subsequent GC pressure) it's improved tasks like parsing text by an order of magnitude (often more) - That's something the compiler will be able to do without the developer's effort.

ASCII is underappreciated.

Re: AMD EPYC Rome 2P Will Have 128-160 PCIe Gen4 Lanes and a Bonus

#56

Post by floatboth »

kllrnohj wrote: post #51
velox_io wrote: post #49

Earlier quoted context omitted.

The new Ryzens are meant to launch with PCIe 4.0 (this summer, I'm due an upgrade!), so effectively... I'm surprised they don't come with more, especially many of the current ones come with 2-3 x16 slots, AND 2-3 M2s.. Both Intel and AMD have been adding more cores recently. Plus, if you're doing any GPU-based rendering, they're going to simultaneously move data back and forth from NVMe to the GPU's memory. But it gr…

> Plus, if you're doing any GPU-based rendering, they're going to simultaneously move data back and forth from NVMe to the GPU's memory. Games definitely don't do this at all, which is the primary market for a discreet GPU in these consumer platforms. When they do stream in assets they do so slowly & in a controlled, rationed amount to minimize impact on FPS. They are far from being PCI-E bandwidth limited. That's ki…

Are there any I/O heavy GPU workloads? Mining famously works with just one lane, offline rendering (e.g. Blender Cycles) I think also just uploads the whole scene once and then bounces the rays around…

Re: AMD EPYC Rome 2P Will Have 128-160 PCIe Gen4 Lanes and a Bonus

#57

Post by kllrnohj »

floatboth wrote: post #56
kllrnohj wrote: post #51

Earlier quoted context omitted.

> Plus, if you're doing any GPU-based rendering, they're going to simultaneously move data back and forth from NVMe to the GPU's memory. Games definitely don't do this at all, which is the primary market for a discreet GPU in these consumer platforms. When they do stream in assets they do so slowly & in a controlled, rationed amount to minimize impact on FPS. They are far from being PCI-E bandwidth limited. That's ki…

Are there any I/O heavy GPU workloads? Mining famously works with just one lane, offline rendering (e.g. Blender Cycles) I think also just uploads the whole scene once and then bounces the rays around…

> offline rendering (e.g. Blender Cycles) I think also just uploads the whole scene once and then bounces the rays around…

I'd imaging a workload like that or other HPC-compute workloads where the data set just doesn't fit in VRAM would certainly prefer more PCI-E lanes.

I think that'd usually be considered using the wrong hardware for the job but if you're just messing around as part of a hobby you're obviously not buying the $7000 Radeon Pro SSG, either.

Re: AMD EPYC Rome 2P Will Have 128-160 PCIe Gen4 Lanes and a Bonus

#58

Post by the8472 »

floatboth wrote: post #56
kllrnohj wrote: post #51

Earlier quoted context omitted.

> Plus, if you're doing any GPU-based rendering, they're going to simultaneously move data back and forth from NVMe to the GPU's memory. Games definitely don't do this at all, which is the primary market for a discreet GPU in these consumer platforms. When they do stream in assets they do so slowly & in a controlled, rationed amount to minimize impact on FPS. They are far from being PCI-E bandwidth limited. That's ki…

Are there any I/O heavy GPU workloads? Mining famously works with just one lane, offline rendering (e.g. Blender Cycles) I think also just uploads the whole scene once and then bounces the rays around…

There are GPU-accelerated database engines which can stream the database database from NVMe. I am not sure if they do direct device-to-device transfer which has only become supported recently or whether they still need to bounce through the main memory.

In raytracing complex scenes can exceed your VRAM so if you don't want to fall back to CPU tracing you need a renderer that can swap parts of the BVH in and out on demand.

Re: AMD EPYC Rome 2P Will Have 128-160 PCIe Gen4 Lanes and a Bonus

#59

Post by ip26 »

kilo_bravo_3 wrote: post #14
consp wrote: post #2

Can someone enlighten me with some information about application requiring the full 16x PCIe gen4 bandwidth per slot (or 32x gen3 for that matter)? I can imagine some HPC GPU solutions but other than that what requires the thoughput? There is obviously a market for this as both giants are building platforms.

Off the top of my head this would be great for massive enterprise VDI installations and game streaming services (which is just VDI by another name), ML, AI applications, and NVMe storage. Personally, this will be good for me because everything I do is bandwidth-starved. I'm not a radar scientist but I am a systems engineer supporting radar scientists working on air- and space-based Synthetic Aperture Radar (SAR) syst…

How does one contact you outside of HN?
Post reply on HN