Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Intel's make-or-break 18A process node debuts for data center with 288-core Xeon

tomshardware.com

Post reply on HN
Print view View original

161–170 of 303 posts

Re: Intel's make-or-break 18A process node debuts for data center with 288-core Xeon

#161

Post by sounds »

bearjaws wrote: post #92
stego-tech wrote: post #8

These sorts of core-density increases are how I win cloud debates in an org. * Identify the workloads that haven't scaled in a year. Your ERPs, your HRIS, your dev/stage/test environments, DBs, Microsoft estate, core infrastructure, etc. (EDIT, from zbentley: also identify any cross-system processing where data will transfer from the cloud back to your private estate to be excluded, so you don't get murdered with egr…

I just don't know if the human capital is there. At my job we use HyperV, and finding someone who actually knows HyperV is difficult and expensive. Throw in Cisco networking, storage appliances, etc to make it 99.99% uptime... Also that means you have just one person, you need at least two if you don't want gaps in staffing, more likely three. Then you still need all the cloud folks to run that. We have a hybrid setu…

> I just don't know if the human capital is there.

> At my job we use HyperV, and finding someone who actually knows HyperV is difficult and expensive...

Try offering significantly higher pay.

Re: Intel's make-or-break 18A process node debuts for data center with 288-core Xeon

#162

Post by charcircuit »

Aurornis wrote: post #114
tgrowazay wrote: post #107

Earlier quoted context omitted.

Self-hosted 8xH100 is ~$250k, depreciated across three years => $80k/year, with power and cooling => $90k/year (~$10/hour total). AWS charges $55/hour for EC2 p5.48xlarge instance, which goes down with 1 or 3 year commitments. With 1 year commitment, it costs ~$30/hour => $262k per year. 3-year commitment brings price down to $24/hour => $210k per year. This price does NOT include egress, and other fees. So, yeah, th…

$120K isn't going to cover the fully loaded costs of an SRE who can set up and run that. Hiring 1 person to run the infrastructure means that 1 person is on-call 24/7 forever. If there's an issue with the server while they're sick or on vacation, you just stop and wait. If they take a new job, you need to find someone to take over or very quickly hire a replacement. There's a second bus factor: What happens when that…

>If there's an issue with the server while they're sick or on vacation, you just stop and wait.

You can ask AI to troubleshoot and fix the issue.

Re: Intel's make-or-break 18A process node debuts for data center with 288-core Xeon

#163

Post by to11mtm »

bearjaws wrote: post #92
stego-tech wrote: post #8

These sorts of core-density increases are how I win cloud debates in an org. * Identify the workloads that haven't scaled in a year. Your ERPs, your HRIS, your dev/stage/test environments, DBs, Microsoft estate, core infrastructure, etc. (EDIT, from zbentley: also identify any cross-system processing where data will transfer from the cloud back to your private estate to be excluded, so you don't get murdered with egr…

I just don't know if the human capital is there. At my job we use HyperV, and finding someone who actually knows HyperV is difficult and expensive. Throw in Cisco networking, storage appliances, etc to make it 99.99% uptime... Also that means you have just one person, you need at least two if you don't want gaps in staffing, more likely three. Then you still need all the cloud folks to run that. We have a hybrid setu…

I think you're hitting on a general problem statement a lot of orgs run into, even ignoring the uptime figure...

All of the complexity of onprem, especially when you need to worry about failover/etc can get tricky, especially if you are in a wintel env like a lot of shops are.

i.e. lots of companies are doing sloppy 'just move the box to an EC2 instance' migrations because of how VMWare jacked their pricing up, and now suddenly EC2/EBS/etc costing is so cheap it's a no brain choice.

I think the knowledge base to set up a minimal cost solution is too tricky to find a benefit vs all the layers (as you almost touched on, all the licensing at every layer vs a cloud provider managing...)

That said, rug pulls are still a risk; I try to push for 'agnostic' workloads in architecture, if nothing else because I've seen too many cases where SaaS/PaaS/etc decide to jack up the price of a service that was cheap, and sure you could have done your own thing agnostically, but now you're there, and migrating away has a new cost.

IOW, I agree; I don't think the human capital is there as far as infra folks who know how to properly set up such environments, especially hitting the 'secure+productive' side of the triangle.

Re: Intel's make-or-break 18A process node debuts for data center with 288-core Xeon

#164

Post by Eridrus »

throwup238 wrote: post #75
carefree-bob wrote: post #58

Earlier quoted context omitted.

The main cost with on-prem is not the price of the gear but the price of acquiring talent to manage the gear. Most companies simply don't have the skillset internally to properly manage these servers, or even the internal talent to know whether they are hiring a good infrastructure engineer or not during the interview process. For those that do, your scaling example works against you. If today you can merge three ser…

> even the internal talent to know whether they are hiring a good infrastructure engineer or not during the interview process. This is really the core problem. Every time I’ve done the math on a sizable cloud vs on-prem deployment, there is so much money left on the table that the orgs can afford to pay FAANG-level salaries for several good SREs but never have we been able to find people to fill the roles or even kno…

I disagree with on-prem being ideal for GPU for most people.

If you're doing regular inference for a product with very flat throughput requirements (and you're doing on-prem already), on-prem GPUs can make a lot of sense.

But if you're doing a lot of training, you have very bursty requirements. And the H100s are specifically for training.

If you can have your H100 fleet If you have batch throughput you can run on the H100s when you're not training, you're probably closer to being able to wanting on-prem.

But the other thing to keep in mind is that AWS is not the only provider. It is a particularly expensive provider, and you can buy capacity from other neoclouds if you are cost-sensitive.

Re: Intel's make-or-break 18A process node debuts for data center with 288-core Xeon

#165

Post by PunchyHamster »

carefree-bob wrote: post #58
stego-tech wrote: post #8

These sorts of core-density increases are how I win cloud debates in an org. * Identify the workloads that haven't scaled in a year. Your ERPs, your HRIS, your dev/stage/test environments, DBs, Microsoft estate, core infrastructure, etc. (EDIT, from zbentley: also identify any cross-system processing where data will transfer from the cloud back to your private estate to be excluded, so you don't get murdered with egr…

The main cost with on-prem is not the price of the gear but the price of acquiring talent to manage the gear. Most companies simply don't have the skillset internally to properly manage these servers, or even the internal talent to know whether they are hiring a good infrastructure engineer or not during the interview process. For those that do, your scaling example works against you. If today you can merge three ser…

> The main cost with on-prem is not the price of the gear but the price of acquiring talent to manage the gear. Most companies simply don't have the skillset internally to properly manage these servers, or even the internal talent to know whether they are hiring a good infrastructure engineer or not during the interview process.

That's partially true; managing cloud also takes skill, most people forget that with end result being "well we saved on hiring sysadmins, but had to have more devops guys". Hell I manage mostly physical infrastructure (few racks, few hundred VMs) and good 80% of my work is completely unrelated to that, it's just the devops gluing stuff together and helping developers to set their stuff up, which isn't all that different than it would be in cloud.

> And remember, you want 24/7 monitoring, replication for disaster recovery, etc.

And remember, you need that for cloud too. Plenty of cloud disaster stories to see where they copy pasted some tutorial thinking that's enough then surprise.

There is also partial way of just getting some dedicated servers from say OVH and run infra on that, you cut out a bit of the hardware management from skillset and you don't have the CAPEX to deal with.

But yes, if it is less than at least a rack, it's probably not worth looking for onprem unless you have really specific use case that is much cheaper there (I mean less than usual half)

Re: Intel's make-or-break 18A process node debuts for data center with 288-core Xeon

#166

Post by dgxyz »

carefree-bob wrote: post #58
stego-tech wrote: post #8

These sorts of core-density increases are how I win cloud debates in an org. * Identify the workloads that haven't scaled in a year. Your ERPs, your HRIS, your dev/stage/test environments, DBs, Microsoft estate, core infrastructure, etc. (EDIT, from zbentley: also identify any cross-system processing where data will transfer from the cloud back to your private estate to be excluded, so you don't get murdered with egr…

The main cost with on-prem is not the price of the gear but the price of acquiring talent to manage the gear. Most companies simply don't have the skillset internally to properly manage these servers, or even the internal talent to know whether they are hiring a good infrastructure engineer or not during the interview process. For those that do, your scaling example works against you. If today you can merge three ser…

This is not the case. We had to double staff count going from three cages to AWS. And AWS was a lot more expensive. And now we're stuck.

On top of that no one really knows what the fuck they are doing in AWS anyway.

Re: Intel's make-or-break 18A process node debuts for data center with 288-core Xeon

#167

Post by fc417fc802 »

brcmthrowaway wrote: post #101
Agingcoder wrote: post #96

Earlier quoted context omitted.

Yes there are scheduling issues, Numa problems , etc caused by the cluster in a box form factor. We had a massive performance issue a few years ago that we fixed by mapping our processes to the numa zones topology . The default design of our software would otherwise effectively route all memory accesses to the same numa zone and performance went down the drain.

Intel contributes to Linux, how is this a problem?

Wrong level of abstraction. NUMA is an additional layer. If the program (script, whatever) was written with a monolithic CPU in mind then the big picture logic won't account for the new details. The kernel can't magically add information it doesn't have (although it does try its best).

Given current trends I think we're eventually going to be forced to adopt new programming paradigms. At some point it will probably make sense to treat on-die HBM distinctly from local RAM and that's in addition to the increasing number of NUMA nodes.

Re: Intel's make-or-break 18A process node debuts for data center with 288-core Xeon

#168

Post by to11mtm »

50lo wrote: post #27

With packages like this (lots of cores, multi-chip packaging, lots of memory channels), the architecture is increasingly a small cluster on a package rather than a monolithic CPU. I wonder whether the next bottleneck becomes software scheduling rather than silicon - OS/runtimes weren’t really designed with hundreds of cores and complex interconnect topologies in mind.

> OS/runtimes weren’t really designed with hundreds of cores and complex interconnect topologies in mind.

I mean....

IMO Erlang/Elixir is a not-terrible benchmark for how things should work in that state... Hell while not a runtime I'd argue Akka/Pekko on JVM Akka.Net on the .NET side would be able to do some good with it...[0] Similar for Go and channels (at least hypothetically...)

[0] - Of course, you can write good scaling code on JVM or CLR without these, but they at least give some decent guardrails for getting a good bit of the Erlang 'progress guaranteed' sauce.

Re: Intel's make-or-break 18A process node debuts for data center with 288-core Xeon

#169

Post by PunchyHamster »

everforward wrote: post #156
throwup238 wrote: post #75

Earlier quoted context omitted.

> even the internal talent to know whether they are hiring a good infrastructure engineer or not during the interview process. This is really the core problem. Every time I’ve done the math on a sizable cloud vs on-prem deployment, there is so much money left on the table that the orgs can afford to pay FAANG-level salaries for several good SREs but never have we been able to find people to fill the roles or even kno…

I work in SRE and the way you describe it would give me pause. The first is that SRE team size primarily scales with the number of applications and level of support. It does scale with hardware but sublinearly, where number of applications usually scales super linearly. It takes a ton less effort to manage 100 instances of a single app than 1 instance of 100 separate apps (presuming SRE has any support responsibiliti…

> The last is that I don’t know many SREs (maybe none at all) that are well-versed enough in all the hardware disciplines to manage a footprint the size we’re talking. If each SRE is 4 racks and a minimum team size is 4, that’s 16 racks. You’d need each SRE to be comfortable enough with networking, storage, operating system, compute scheduling (k8s, VMWare, etc) to manage each of those aspects for a 16 rack system. In reality, it’s probably 3 teams, each of them needs 4 members for oncall, so a floor of like 48 racks. Depending on how many applications you run on 48 racks, it might be more SREs that split into more specialized roles (a team for databases, a team for load balancers, etc).

That's vastly overstating it. You hit nail in the head in previous paragraphs, it's number of apps (or more generally speaking ,environments) that you manage, everything else is secondary.

And that is especially true with modern automation tools. Doubling rack count is big chunk of initial time spent moving hardware of course, but after that there is almost no difference in time spent maintaining them.

In general time per server spent will be smaller because the bigger you grow the more automation you will generally use and some tasks can be grouped together better.

Like, at previous job, server was installed manually, coz it was rare.

At my current job it's just "boot from network, pick the install option, enter the hostname, press enter". Doing whole rack (re)install would take you maybe an hour, everything else in install is automated, you write manifest for one type/role once, test it, and then it doesn't matter whether its' 2 or 20 servers.

If we grew server fleet say 5-fold, we'd hire... one extra person to a team of 3. If number of different application went 5-fold we'd probably had to triple the team size - because there is still some things that can be made more streamlined.

Tasks like "go replace failed drive" might be more common but we usually do it once a week (enough redundancy) for all servers that might've died, if we had 5x the number of servers the time would be nearly the same because getting there dominates the 30s that is needed to replace one.

Re: Intel's make-or-break 18A process node debuts for data center with 288-core Xeon

#170

Post by zokier »

Retr0id wrote: post #141
zokier wrote: post #84

Earlier quoted context omitted.

afaik the mainline limit is 4096 threads. HP sells server with 32 sockets x 60 cores/socket x 2 threads/core = 3840 threads, so we are pretty close to that limit.

I had no idea we had socket counts so high, do you know where I could find a picture of one?

It's bit cheating because it's cluster based system: https://www.hpe.com/psnow/doc/a50004268enw

So 4 sockets per chassis, up to 8 chassis in a complete system. Afaik OS sees it as single huge system, that is kinda their special sauce here.

Post reply on HN