Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Intel's make-or-break 18A process node debuts for data center with 288-core Xeon

tomshardware.com

Post reply on HN
Print view View original

251–260 of 303 posts

Re: Intel's make-or-break 18A process node debuts for data center with 288-core Xeon

#251

Post by fc417fc802 »

consp wrote: post #244
fc417fc802 wrote: post #167

Earlier quoted context omitted.

Wrong level of abstraction. NUMA is an additional layer. If the program (script, whatever) was written with a monolithic CPU in mind then the big picture logic won't account for the new details. The kernel can't magically add information it doesn't have (although it does try its best). Given current trends I think we're eventually going to be forced to adopt new programming paradigms. At some point it will probably m…

Isn't high grade SSD storage pretty much a memory layer as well these days as the difference is no longer several orders of magnitude in access time and thoughput but only one or two (compared to tha last layer of memory)?

Sort of? Relative to 6 or more channels of RAM it's still quite abysmal but perhaps high bandwidth flash will change how things are done.

Re: Intel's make-or-break 18A process node debuts for data center with 288-core Xeon

#252

Post by nsteel »

andreadev wrote: post #157

I think everyone's focusing on the core count, but the packaging story is way more interesting here. This thing is 12 separate chiplets on 18A stacked on base dies made on Intel 3, connected to I/O tiles on Intel 7. Three different process nodes in one package, shipping at volume. That's nuts. And it's clearly an IFS play too. Intel Foundry needs a proof point — you can publish PDKs all day, but nothing sells foundry…

Agree entirely with your take. The packaging story is awesome, I wish there were more details on the stacking used on this one.

But I am at a loss to how Intel are really going to get any traction with IFS. How can anyone trust Intel as a long-term foundry partner. Even if they priced it more aggressively, the opportunity cost in picking a supplier who decides to quit next year would be catastrophic for many. The only way this works is if they practically give their services away to someone big, who can afford to take that risk and can also make it worth Intel's continued investment. Any ideas who that would be, I've got nothing.

Re: Intel's make-or-break 18A process node debuts for data center with 288-core Xeon

#253

Post by tiew9Vii »

carefree-bob wrote: post #58
stego-tech wrote: post #8

These sorts of core-density increases are how I win cloud debates in an org. * Identify the workloads that haven't scaled in a year. Your ERPs, your HRIS, your dev/stage/test environments, DBs, Microsoft estate, core infrastructure, etc. (EDIT, from zbentley: also identify any cross-system processing where data will transfer from the cloud back to your private estate to be excluded, so you don't get murdered with egr…

The main cost with on-prem is not the price of the gear but the price of acquiring talent to manage the gear. Most companies simply don't have the skillset internally to properly manage these servers, or even the internal talent to know whether they are hiring a good infrastructure engineer or not during the interview process. For those that do, your scaling example works against you. If today you can merge three ser…

> The main cost with on-prem is not the price of the gear but the price of acquiring talent to manage the gear. Most companies simply don't have the skillset internally to properly manage these servers

This comes up again and again. It was the original sales pitch from cloud vendors.

Often the very same companies repeating this messaging are recruiting and paying large teams of platform developers to manage their cloud…and pay for them to be on call.

Re: Intel's make-or-break 18A process node debuts for data center with 288-core Xeon

#254

Post by roryirvine »

Figs wrote: post #131
Aurornis wrote: post #114

Earlier quoted context omitted.

$120K isn't going to cover the fully loaded costs of an SRE who can set up and run that. Hiring 1 person to run the infrastructure means that 1 person is on-call 24/7 forever. If there's an issue with the server while they're sick or on vacation, you just stop and wait. If they take a new job, you need to find someone to take over or very quickly hire a replacement. There's a second bus factor: What happens when that…

> $120K isn't going to cover the fully loaded costs of an SRE who can set up and run that. > Hiring 1 person to run the infrastructure means that 1 person is on-call 24/7 forever. > If there's an issue with the server while they're sick or on vacation, you just stop and wait. Very much depends on what you're doing, of course, but "you just stop and wait" for sickness/vacation sometimes is actually good enough uptime…

So the entire business was happy to go offline for 2/3 weeks whenever their infra person fancied going off on their summer holiday?

By doing this, you're guaranteeing a bus factor of below 1. I can't think of any business that wouldn't see that as being a completely unacceptable risk.

Re: Intel's make-or-break 18A process node debuts for data center with 288-core Xeon

#255

Post by stinkbeetle »

zokier wrote: post #170
Retr0id wrote: post #141

Earlier quoted context omitted.

I had no idea we had socket counts so high, do you know where I could find a picture of one?

It's bit cheating because it's cluster based system: https://www.hpe.com/psnow/doc/a50004268enw So 4 sockets per chassis, up to 8 chassis in a complete system. Afaik OS sees it as single huge system, that is kinda their special sauce here.

It's not cheating or a cluster based system. All the biggest high end servers use multiple externally cabled systems (chassis, sled, drawer). The biggest ones even span multiple racks (aka frames). These days it is HP and IBM remaining in the game.

These all have real hardware coherency going over the external cables, same protocol. Here is a Power10 server picture, https://www.engineering.com/ibm-introduces-power-e1080-serve... the cables attach right to headers brought out of the chip package right off the phy, there's no ->PCI->ethernet-> or anything like that.

These HP systems are similar. These are actually descendants of SGI Altix / SGI Origin systems which HP acquired, and they still use some of the same terminology (NUMAlink for the interconnect fabric). HP did make their own distinct line of big iron systems when they had PA-RISC and later Itanium but ended up acquiring and going with SGI's technology for whatever reasons.

These HP/SGI systems are slightly different from IBM mini/mainframes because they use "commodity" CPUs from Intel that don't support glueless multi socket that large or have signaling that can get across boards, so these have their own chipset that has some special coherency directories and a bunch of NUMAlink PHYs.

SGI systems came from HPC so they were actually much bigger before that, the biggest ones were something around 1024 sockets, back when you only had 1 CPU per socket. The interconnect topology used to be some tree thing that had like 10 hops between the farthest nodes. It did run Linux and wasn't technically cheating, but you really had to program it like a cluster because resource contention would quickly kill you if there was much cacheline transfer between nodes. Quite amazing machines, but not suitable for "enterprise" so IIRC they have cut it down and gone with all-to-all interconnect. It would be interesting to know what they did with coherency protocol, the SGI systems used a full directory scheme which is simple and great at scaling to huge sizes but not the best for performance. IBM systems use extremely complex broadcast source snooping designs (highly scoped and filtered) to avoid full directory overhead. Would be interesting to know if HPE finally went that way with NUMAlink too.

Found this diagram from an old HP product which was an SGI derivative https://support.hpe.com/hpesc/public/docDisplay?docId=a00062... 2 QPI busses and 16 NUMAlink ports!

Aha, it's still a directory protocol. https://support.hpe.com/hpesc/public/docDisplay?docId=sd0000...

Cheating IMO would be an actual cluster of systems using software (firmware/hypervisor) to present a single system image using MMU and IB/ethernat adapters to provide coherency.

Re: Intel's make-or-break 18A process node debuts for data center with 288-core Xeon

#256

Post by stinkbeetle »

nixon_why69 wrote: post #223
zokier wrote: post #170

Earlier quoted context omitted.

It's bit cheating because it's cluster based system: https://www.hpe.com/psnow/doc/a50004268enw So 4 sockets per chassis, up to 8 chassis in a complete system. Afaik OS sees it as single huge system, that is kinda their special sauce here.

How the heck does the OS see it as a single system, is there some pcie or rdma black magic that allows the kernel to just address memory in a different chassis? Maybe CXL?

No it's actual hardware coherent memory across the system. At a high level it is the same way two cores/caches are connected within one chip, or the same way two sockets are connected on the same board. Just using cables instead of wires in the chip or on a board.

This system has SMP ASICs on the motherboards that talk to a couple of Intel processor sockets using their coherency protocol over QPI and they basically present themselves as a coherency agent and memory provider (similarly to the way that processors themselves have caches and DDR controllers). The Intel CPUs basically talk to them the same way they would another processor. But out the other side these ASICS connect to a bunch of others all doing the same thing, and they use their own coherency protocol among themselves.

Re: Intel's make-or-break 18A process node debuts for data center with 288-core Xeon

#257

Post by 0x0203 »

nsteel wrote: post #252
andreadev wrote: post #157

I think everyone's focusing on the core count, but the packaging story is way more interesting here. This thing is 12 separate chiplets on 18A stacked on base dies made on Intel 3, connected to I/O tiles on Intel 7. Three different process nodes in one package, shipping at volume. That's nuts. And it's clearly an IFS play too. Intel Foundry needs a proof point — you can publish PDKs all day, but nothing sells foundry…

Agree entirely with your take. The packaging story is awesome, I wish there were more details on the stacking used on this one. But I am at a loss to how Intel are really going to get any traction with IFS. How can anyone trust Intel as a long-term foundry partner. Even if they priced it more aggressively, the opportunity cost in picking a supplier who decides to quit next year would be catastrophic for many. The onl…

I suspect that timing might help Intel here, with so much of the better established foundries near fully allocated for the next two years, it may be more a question of availability than brand name risk. And for whatever problems Intel has, it's pretty unlikely they'd go completely under and disolve in less than a year. Good non completion clauses in the contracts can mitigate a good chunk of the remaining risk.

Not to mention potential customers who would prefer a US based foundry regardless. My guess is that there's a pretty large part of the market that would be perfectly fine with using Intel.

Re: Intel's make-or-break 18A process node debuts for data center with 288-core Xeon

#258

Post by misswaterfairy »

MrBuddyCasino wrote: post #250
stego-tech wrote: post #8

These sorts of core-density increases are how I win cloud debates in an org. * Identify the workloads that haven't scaled in a year. Your ERPs, your HRIS, your dev/stage/test environments, DBs, Microsoft estate, core infrastructure, etc. (EDIT, from zbentley: also identify any cross-system processing where data will transfer from the cloud back to your private estate to be excluded, so you don't get murdered with egr…

It seems a lot of people have forgotten how BigCorp IT used to work. - request some HW to run $service - the "IT dept" (really, self-interested gatekeeper) might give you something now, or in two weeks, or god help you if they need to order new hardware then its in two months, best case - there will be various weird rules on how the on-prem HW is run, who has access etc, hindering developer productivity even further…

> The cloud is a way around this political minefield.

Until the bills _really_ start skyrocketing...

Re: Intel's make-or-break 18A process node debuts for data center with 288-core Xeon

#259

Post by TomMasz »

50lo wrote: post #27

With packages like this (lots of cores, multi-chip packaging, lots of memory channels), the architecture is increasingly a small cluster on a package rather than a monolithic CPU. I wonder whether the next bottleneck becomes software scheduling rather than silicon - OS/runtimes weren’t really designed with hundreds of cores and complex interconnect topologies in mind.

I was wondering this, too. There's no mention of OS support, but I assume Intel is working with the usual suspects on it.

Re: Intel's make-or-break 18A process node debuts for data center with 288-core Xeon

#260

Post by cyanydeez »

topspin wrote: post #159
cyanydeez wrote: post #72

Earlier quoted context omitted.

is RAM a concern in these cluster applications, cause if prices stay up, how do you get them off the shelf if you also need TB of memory.

> how do you get them off the shelf if you also need TB of memory You make products for well capitalized wireless operators that can afford the prevailing cost of the hardware they need. For these operations, the increase in RAM prices is not a major factor in their plans: it's a marginal cost increase on some of the COTS components necessary for their wireless system. The specialized hardware they acquire in bulk is…

I do likr the idea that capitalism can always ignore the broader base of consumers and just raise prices. Eventually, therell only be one viagra pill bought by trillionaires at 1$ million dollars.

God bless capitalism.

Post reply on HN