AWS doesn't make sense for scientific computing • Hacker News Board

Re: AWS doesn't make sense for scientific computing

#211

Post by zmmmmm » Sat, Oct 08, 2022, 12:25 AM UTC

A missing elements for me is that with a lot of exploratory scientific work we (often half intentionally) have no idea what we are doing. We can easily run a giant job that uses 100x more compute than expected. Yes you can limit cloud compute resources if you are smart but its much better is the default is that you run out of compute and your job takes longer than you get a 100x bill back from our cloud provider. And if you limited your cloud resources to a fixed amount, didn't you just eliminate half the benefit of using cloud in the first place?

Then the problems of data management, transfer and egress are huge. Again the "no idea what we are doing" factor comes into play. If you have a really good idea up front what is going to happen you can plan out a strategy that minimises costs. But if you have no idea at all - genuinely, because this is science and we are doing new things - then you could end up blowing huge amounts of money on unnecessary egress and storage costs. And at the small end we can be talking about experiments run on a shoestring where a few thousand dollars is a big deal.

The way I see it, we need everything - powerful individual workstations / laptops for direct analysis, then a tier of fixed HPC style compute for this kind of work that is poorly matched to cloud, and then for specific projects where it makes sense (massive scaleout, exotic hardware needs - GPUs, FPGAs etc) you embrace cloud resources for that.

Re: AWS doesn't make sense for scientific computing

#212

Post by zmmmmm » Sat, Oct 08, 2022, 12:33 AM UTC

timeu wrote: post #155

As an HPC sysadmin for 3 research institutes (mostly life sciences & biology) I can't see how cloud HPC system could be any cheaper than an on-prem HPC system especially if I look at the resource efficiency (how much resources were requested vs how much were actually useed) of our users SLURM jobs. Often the users request 100s of GB but only use a fraction of it. In our on-prem HPC system this might decrease utilizat…

you touch on a good point in that because cloud compute requires pretty knowledgeable users in order not to waste massive amounts of money it effectively imposes a much higher level of competency requirement onto the users. You can view that different ways, one is that its a good thing if everyone learns to use compute better. But another is you are locking out a whole tier of scientific users from doing computation at all which is a pretty unfortunate thing - we may miss out on real and important scientific discoveries - even if they are horrifically bad at doing it efficiently.

Re: AWS doesn't make sense for scientific computing

#213

Post by absolutelynobo » Sat, Oct 08, 2022, 12:33 AM UTC

W-Stool wrote: post #32

walnutclosefarm wrote: post #24

Having had the responsibility of providing HPC for a literal buildings full of scientists, I can say that it may be true that you can get computation cheaper with owned hardware, than in a cloud. Certainly pay as you go, individual project at a time processing will look that way to the scientist. But I can also say with confidence that the contest is far closer than they think. Scientists who make this argument almos…

Let me echo this as someone who once was responsible for HPC computing in a research intensive public university. Most career academics have NO IDEA how much enterprise computing infrastructure costs. If a 1 terabyte USB hard drive is $40 at Costco we (university IT) must be getting a much better deal than that. Take this argument and apply it to any aspect of HPC computing and that's what you're fighting against. Th…

> Don't forget the AC terminal strips that power the whole thing, sourced from the local dollar store.

Love how you’ve fuzzed the root cause to make it seem like the “dollar store strip” is the problem and not that it was plugged into an overloaded outlet or run inside a closet at significantly elevated temperatures, leading to plastic melting and wires shorting.

Always helps to keep the “magic” a secret so the rubes have to keep us wizards employed, right?

Re: AWS doesn't make sense for scientific computing

#214

Post by absolutelynobo » Sat, Oct 08, 2022, 12:35 AM UTC

xani_ wrote: post #201

pbronez wrote: post #173

Earlier quoted context omitted.

This is my fear about my homelab lol Fire extinguisher nearby, smart temp sensors, but still...

No more or less chance that really anything else connected to power to catch on fire. Just make sure RCD works. Hell, some weeks ago my oven decided the lower heater's line is now connected to ground and blows fuses... Our 8 racks in DC only had single event of something blowing (power supply) and aside of smell and fuse blowing nothing really happened Servers are essentially metal boxes with a bit of glass-reinforce…

Cool how you use the acronym RCD and don’t expand it anywhere. Or any of the other homelab acronyms you used!

Re: AWS doesn't make sense for scientific computing

#215

Post by bushbaba » Sat, Oct 08, 2022, 12:35 AM UTC

awiesenhofer wrote: post #156

bushbaba wrote: post #29

Earlier quoted context omitted.

Checkout Apache Iceberg which makes it fairly trivial to get high throughput from S3 without much fine-tuning. Bursts from 0 to 50Gbps should be possible from S3 without much effort, just have object sizes that are in the NN+ MiB range. Personally, Lustre is a mess, it's expensive and even more pain to fine-tune.

From https://iceberg.apache.org > Iceberg is a high-performance format for huge analytic tables. How would that help speedup S3? Genuine question?

Iceberg doesn’t use the directory structure to represent what objects belong to which table. The change allows objects to have significant entropy in the object keys prefix. This avoids hot spotting s3 and allows for greater bursts to s3 QPs.

Re: AWS doesn't make sense for scientific computing

#216

Post by absolutelynobo » Sat, Oct 08, 2022, 12:36 AM UTC

gonzo41 wrote: post #190

systemvoltage wrote: post #172

Earlier quoted context omitted.

I am dealing with the exact opposite problem: "Oh you mean, we should leave the EC2 instance running 24/7 ??? No way, that would be too expensive"... to which I need to respond "No, it would be like $15/month. Trivial, stop worrying about costs in EC2 and S3, we're like 7 people here with 3 GB of data." I deal with Scientists that think AWS is some sort of a massively expensive enterprise thing. I can be, but not for…

Don't say the budget outloud near AWS. They'll find a way to help you spend it.

GP has already lost that battle.

Re: AWS doesn't make sense for scientific computing

#217

Post by nwilkens » Sat, Oct 08, 2022, 12:38 AM UTC

kortex wrote: post #209

nwilkens wrote: post #163

Earlier quoted context omitted.

I really like this description of how baremetal infrastructure should work, and this is where I think (shameless self promotion) Triton DataCenter[1] plays really well today on-prem. PXE booted lightweight compute nodes with a robust API, including operator portal, user portal, and cli. Keep an eye out for the work we are doing with Triton Linux + K8s. Very lightweight Triton Linux compute node + baremetal k8s deploy…

Thanks! This is pretty sleek! I'm going to have to dust off my homelab and play around with this. What is the stack written in? Looks like a lot of javascript and makefiles from the github side of things but idk if that's the whole kit and caboodle.

As you've seen, we're open source -- https://github.com/tritondatacenter. There are tons of repos.. Mostly Nodejs and some golang/rust!

Re: AWS doesn't make sense for scientific computing

#218

Post by geoduck14 » Sat, Oct 08, 2022, 12:40 AM UTC

bilbo0s wrote: post #80

eastbound wrote: post #55

Earlier quoted context omitted.

Don’t worry, we do incremental backups during weekdays and a full backup on Sunday. We use 2 tapes only, so one is always outside of the building. But you know, we saved money.

A bit off topic, but I gotta say you guys are a riot! If there was a comedy tour for IT/Programmer types, I'd pay to see you guys in it. Best thing about your stuff is that it's literally all funny precisely because it's all true.

I don't know why you are being down voted. I'd watch a Netflix special on this stuff

Re: AWS doesn't make sense for scientific computing

#219

Post by geoduck14 » Sat, Oct 08, 2022, 12:43 AM UTC

systemvoltage wrote: post #172

W-Stool wrote: post #32

Earlier quoted context omitted.

Let me echo this as someone who once was responsible for HPC computing in a research intensive public university. Most career academics have NO IDEA how much enterprise computing infrastructure costs. If a 1 terabyte USB hard drive is $40 at Costco we (university IT) must be getting a much better deal than that. Take this argument and apply it to any aspect of HPC computing and that's what you're fighting against. Th…

I am dealing with the exact opposite problem: "Oh you mean, we should leave the EC2 instance running 24/7 ??? No way, that would be too expensive"... to which I need to respond "No, it would be like $15/month. Trivial, stop worrying about costs in EC2 and S3, we're like 7 people here with 3 GB of data." I deal with Scientists that think AWS is some sort of a massively expensive enterprise thing. I can be, but not for…

We use a phrase around my job: the only attack we suffer from is "denial of wallet"

Re: AWS doesn't make sense for scientific computing

#220

Post by cdsx » Sat, Oct 08, 2022, 12:50 AM UTC

walnutclosefarm wrote: post #24

Having had the responsibility of providing HPC for a literal buildings full of scientists, I can say that it may be true that you can get computation cheaper with owned hardware, than in a cloud. Certainly pay as you go, individual project at a time processing will look that way to the scientist. But I can also say with confidence that the contest is far closer than they think. Scientists who make this argument almos…

Perspective from a computational biologist: Campus hosted HPC means the direct cost pressure is seen as IT staff and hardware related costs. Researchers are encouraged to use the available capacity. This is good.

Externally-hosted HPC means every single compute job is seen as something that directly costs money. This negatively affects the quality of scientific output (research playfulness / creativity / focus on the research / etc).