Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GigaByte CXL memory expansion card with up to 512GB DRAM

gigabyte.com

Post reply on HN
Print view View original

41–50 of 67 posts

Re: GigaByte CXL memory expansion card with up to 512GB DRAM

#41

Post by to11mtm »

bobmcnamara wrote: post #35
kvemkon wrote: post #16

Earlier quoted context omitted.

I'd have rather a question why we had single (or already dual) core CPUs with dual-channel memory controller and now we have 16-core CPUs but still with only dual-channel RAM.

Intel and AMD I'd reckon. Apple went wide with their busses.

Well, Each Channel needs a lot of pins. I don't think all 288/262 pins need to go to the CPU, but a large number of them do, I'd wager; The old LGA 1366 (Tri-Channel) and LGA 1151 (Dual Channel) are probably as close as we can get to a simple reference point [0].

Apple FBOW, based on a quick and sloppy count of a reballing jig [1], has something on the order of 2500-2700 balls on an M2 CPU.

I think AMD's FP11 'socket' (it's really just a standard ball grid array) pinout is something on the order of 2000-2100 balls and that gets you four 64 Bit DDR channels (I think Apple works a bit different and uses 16 bit channels, thus the 'channel count' for an M2 is higher.)

Which is a roundabout way of saying, AMD and Intel probably can match the bandwidth but to do so likely would require moving to soldered CPUS which would be a huge paradigm shift for all the existing boardmakers/etc.

[0] - They do have other tradeoffs; namely that 1151 has built in PCIE, on the other hand the link to the PCH is AFAIR a good bit thinner than the QPI link on the 1366.

[1] - https://www.masterliuonline.com/products/a2179-a1932-cpu-reb... . I counted ~55 rows along the top and ~48 rows on the side...

Re: GigaByte CXL memory expansion card with up to 512GB DRAM

#43

Post by afr0ck »

immibis wrote: post #32
Twirrim wrote: post #7

CXL is going to be really interesting. On the positive side, you can scale out memory quite a lot, fill up PCI slots, even have memory external to your chassis. Memory tiering has a lot of potential. On the negative side, you've got latency costs to swallow up. You don't get distance from CPU for free (there's a reason the memory on your motherboard is as close as practical to the CPU) https://www.nextplatform.com/20…

What kind of motherboard, CPU, cables, switches, and end devices would I need to buy to have a CXL network?

CXL uses the PCIe physical layer, so you just need to buy hardware that understands the protocol, namely the CPU and the expansion boards. AMD Genoa (e.g. EPYC 9004) supports CXL 1.1 as well as Intel Saphire Rapids and all subsequent models do. For CXL memory expansion boards, you can get from Samsung or Marvell. I got a 128 GB model from Samsung with 25 GB/s read throughput.

Re: GigaByte CXL memory expansion card with up to 512GB DRAM

#44

Post by afr0ck »

tanelpoder wrote: post #8
mdaniel wrote: post #4

> Buy From One of the Regions Below > Egypt :-/ But, because I'm a good sport, I actually chased a couple of those links figuring that I could convert Egyptian Pound into USD but https://www.sigma-computer.com/en/search?q=CXL%20R5X4 > is "No results", and similar for the other ones that I could get to even load

Yeah I saw the same. I've been keeping an eye on the CXL world for ~5 years and so far it's 99% announcements, unveilings and great predictions. But the only CXL cards a consumer/small business can buy are some experimental-ish 64GB/128GB cards that you can actually buy today. Haven't seen any of my larger clients use it either. Both Intel Optane and DSSD storage efforts got discontinued after years of fanfare, from…

I think Meta has already rolled out some CXL hardware for memory tiering. Marvell, Samsung, Xconn and many others have built various memory chips and switching hardware up to CXL 3.0. All recent Intel and AMD CPUs support CXL.

Re: GigaByte CXL memory expansion card with up to 512GB DRAM

#46

Post by adgjlsfhk1 »

JonChesterfield wrote: post #33
wmf wrote: post #30

Earlier quoted context omitted.

It's only ~2x worse latency than main memory but 100x lower than SSD.

I'm finding ~50ns best case for pcie, ~10ns for system. Which is a lot closer than I expected.

no system ram is 10ns. That's closer to L2 cache.

Re: GigaByte CXL memory expansion card with up to 512GB DRAM

#47

Post by Cr8 »

JonChesterfield wrote: post #29

I don't get it. The point of (ddrN) memory is latency. If its on the far side of pcie latency is much worse than the system memory. In what sense is this better than ssd on the far side of pcie?

pcie devices can also do direct transfers to each other - if you have one of these and a gpu its relatively quick to move data between them without bouncing through main ram

Re: GigaByte CXL memory expansion card with up to 512GB DRAM

#48

Post by temp0826 »

Twirrim wrote: post #7

CXL is going to be really interesting. On the positive side, you can scale out memory quite a lot, fill up PCI slots, even have memory external to your chassis. Memory tiering has a lot of potential. On the negative side, you've got latency costs to swallow up. You don't get distance from CPU for free (there's a reason the memory on your motherboard is as close as practical to the CPU) https://www.nextplatform.com/20…

I have never had to go deep into NUMA configuration personally but couldn't it be leveraged here?

Re: GigaByte CXL memory expansion card with up to 512GB DRAM

#49

Post by samus »

tanelpoder wrote: post #10
Twirrim wrote: post #7

CXL is going to be really interesting. On the positive side, you can scale out memory quite a lot, fill up PCI slots, even have memory external to your chassis. Memory tiering has a lot of potential. On the negative side, you've got latency costs to swallow up. You don't get distance from CPU for free (there's a reason the memory on your motherboard is as close as practical to the CPU) https://www.nextplatform.com/20…

Yup, for best results you wouldn't just dump your existing pointer-chasing and linked-list data structures to CXL (like the Optane's transparent mode was, whatever it was called). But CXL-backed memory can use your CPU caches as usual and the PCIe 5.0 lane throughput is still good, assuming that the CXL controller/DRAM side doesn't become a bottleneck. So you could design your engines and data structures to account f…

DBMSs have been managing storage with different access times for decades and it should be pretty easy to adapt an existing engine. Or you could use it as a gigantic swap space. No clue whether additional kernel patches would be required for that.

Re: GigaByte CXL memory expansion card with up to 512GB DRAM

#50

Post by wmf »

temp0826 wrote: post #48
Twirrim wrote: post #7

CXL is going to be really interesting. On the positive side, you can scale out memory quite a lot, fill up PCI slots, even have memory external to your chassis. Memory tiering has a lot of potential. On the negative side, you've got latency costs to swallow up. You don't get distance from CPU for free (there's a reason the memory on your motherboard is as close as practical to the CPU) https://www.nextplatform.com/20…

I have never had to go deep into NUMA configuration personally but couldn't it be leveraged here?

Yes, if you want your app to be aware of CXL you can configure it as a separate NUMA node.
Post reply on HN