Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS doesn't make sense for scientific computing

noahlebovic.com

Post reply on HN
Print view View original

171–180 of 281 posts

Re: AWS doesn't make sense for scientific computing

#171

Post by lebovic »

adolph wrote: post #124
lebovic wrote: post #16

Earlier quoted context omitted.

For fast-moving researchers who are blocked by a queue, cloud computing still makes sense. I guess I wasn't clear enough in the last section about how I still use AWS for startup-scale computational biology. My scientific computing startup (trytoolchest.com) is 100% built on top of AWS. Most scientific computing still happens on supercomputers in slower moving academic or big co settings. That's the group for whom cl…

Another service that runs on AWS is CodeOcean. It looks like Toolchest is oriented toward facilitating execution of specific packages rather than organization and execution like CodeOcean. Is that a fair summary? https://codeocean.com/explore

Yep, that's right! Toolchest focuses on compute, deploying and optimizing popular scientific computing packages.

Re: AWS doesn't make sense for scientific computing

#172

Post by systemvoltage »

W-Stool wrote: post #32
walnutclosefarm wrote: post #24

Having had the responsibility of providing HPC for a literal buildings full of scientists, I can say that it may be true that you can get computation cheaper with owned hardware, than in a cloud. Certainly pay as you go, individual project at a time processing will look that way to the scientist. But I can also say with confidence that the contest is far closer than they think. Scientists who make this argument almos…

Let me echo this as someone who once was responsible for HPC computing in a research intensive public university. Most career academics have NO IDEA how much enterprise computing infrastructure costs. If a 1 terabyte USB hard drive is $40 at Costco we (university IT) must be getting a much better deal than that. Take this argument and apply it to any aspect of HPC computing and that's what you're fighting against. Th…

I am dealing with the exact opposite problem: "Oh you mean, we should leave the EC2 instance running 24/7??? No way, that would be too expensive"... to which I need to respond "No, it would be like $15/month. Trivial, stop worrying about costs in EC2 and S3, we're like 7 people here with 3 GB of data."

I deal with Scientists that think AWS is some sort of a massively expensive enterprise thing. I can be, but not for the use case they're going to be embarking on. Our budget is $7M spanning 4 years.

Re: AWS doesn't make sense for scientific computing

#173

Post by pbronez »

0xbadcafebee wrote: post #38
W-Stool wrote: post #32

Earlier quoted context omitted.

Let me echo this as someone who once was responsible for HPC computing in a research intensive public university. Most career academics have NO IDEA how much enterprise computing infrastructure costs. If a 1 terabyte USB hard drive is $40 at Costco we (university IT) must be getting a much better deal than that. Take this argument and apply it to any aspect of HPC computing and that's what you're fighting against. Th…

I remember the first time a server caught fire in the closet we kept the rack in. Backups were kept on a server right below the one on fire. But, y'know, we saved money.

This is my fear about my homelab lol

Fire extinguisher nearby, smart temp sensors, but still...

Re: AWS doesn't make sense for scientific computing

#174

Post by thamer »

betolink wrote: post #105

I see both sides of the argument, there is a reason why CERN is not processing their data using EC2 and lambdas.

The vast majority of researchers don't need anywhere close to the amount of resources that CERN needs. The fact that CERN doesn't use EC2 and lambdas shouldn't be taken as a lesson by anyone who's not operating at their scale.

This feels like a similar argument to the one made by people who use Kubernetes to ensure their web app with 100 visitors a day is web scale.

Re: AWS doesn't make sense for scientific computing

#175

Post by pbronez »

walnutclosefarm wrote: post #24

Having had the responsibility of providing HPC for a literal buildings full of scientists, I can say that it may be true that you can get computation cheaper with owned hardware, than in a cloud. Certainly pay as you go, individual project at a time processing will look that way to the scientist. But I can also say with confidence that the contest is far closer than they think. Scientists who make this argument almos…

The article estimated:

Running a modern AMD-based server that has 48 cores, at least 192 GB of RAM, and no included disk space costs:

    ~$2670.36/mo for a c5a.24xlarge AWS on-demand instance
    ~$1014.7/mo for a c5a.24xlarge AWS reserved instance on a three-year term, paid upfront
    ~$558.65/mo on OVH Cloud[1]
    ~$512.92/mo on Hetzner[2]
    ~$200/mo on your own infrastructure as a large institution[3]
Footnote [3] explains this cost estimate as:

"Assumes an AMD EPYC 7552 run at 100% load in Boston with high electricity prices of $0.23/kWh, for $33.24/mo in raw power. Hardware is amortized over five years, for an average monthly price of $67.08/mo. We assume that your large institution already has 24/7 security and public internet bandwidth, but multiply base hardware and power costs by 2x to account for other hardware, cooling, physical space, and a half-a-$120k-sysadmin amortized across 100 servers."

Re: AWS doesn't make sense for scientific computing

#176

Post by nharada »

I'd love to buy my own servers for small-scale (i.e. startup size or research lab size) projects, but it's very hard to be utilizing them 24x7. Does anyone know of open-source software or tools that allow multiple people to timeshare one of these? A big server full of A100s would be awesome, with the ability to reserve the server on specific days.

Re: AWS doesn't make sense for scientific computing

#177

Post by lebovic »

somesortofthing wrote: post #111

The author makes a convincing argument against doing this workload on on-demand instances, but what about spot instances? AWS explicitly calls out scientific computing as a major use case for scientific computing in its training/promotional materials. Given the advertised ~70-90% markdown on spot instance time, it seems like a great option compared to paying almost the same amount as the workstation but not having to…

Author here! Spot instance pricing is better than on-demand, but it doesn't include data transfer, and it's still more expensive than on-prem/Hetzner/etc. Data transfer costs exceed the cost of the instance itself if you're transferring many TB off AWS.

For of our more popular AWS instance types I use – a c5a.24xlarge, used for comparison in the post – the cheapest spot price over the past month in us-east-1 was $1.69. That's still $1233.70/mo: above on-prem, colo, or Hetzner pricing. Data transfer is still extremely expensive.

That said, for bursty loads that can't be smoothed with a queue, spot instances (or just normal EC2 instances) do make sense! I use them all the time for my computational biology company.

Re: AWS doesn't make sense for scientific computing

#178

Post by davidmr »

bluedino wrote: post #113
W-Stool wrote: post #32

Earlier quoted context omitted.

Let me echo this as someone who once was responsible for HPC computing in a research intensive public university. Most career academics have NO IDEA how much enterprise computing infrastructure costs. If a 1 terabyte USB hard drive is $40 at Costco we (university IT) must be getting a much better deal than that. Take this argument and apply it to any aspect of HPC computing and that's what you're fighting against. Th…

It's kind of funny around this time of year when some researchers have $10,000 in their budget they need to spend, and they want to 'gift' us with some GPU's.

That was definitely one of the weirdest things of working in academia IT: “hey. Can you buy me a workstation that’s as close to $6,328.45 as it is possible to get, and can you do it by 4pm?”

Re: AWS doesn't make sense for scientific computing

#179

Post by didip »

nharada wrote: post #176

I'd love to buy my own servers for small-scale (i.e. startup size or research lab size) projects, but it's very hard to be utilizing them 24x7. Does anyone know of open-source software or tools that allow multiple people to timeshare one of these? A big server full of A100s would be awesome, with the ability to reserve the server on specific days.

This is just the cloud with extra steps.

Re: AWS doesn't make sense for scientific computing

#180

Post by davidmr »

pclmulqdq wrote: post #36

Even as a big cloud detractor, I have to disagree with this. A lot of scientific computing doesn't need a persistent data center, since you are running a ton of simulations that only take a week or so, and scientific computing centers at big universities are a big expense that isn't always well-utilized. Also, when they are full, jobs can wait weeks to run. These computing centers have fairly high overhead, too, alth…

This is tangential to your point, but I’ll just mention that Azure has some properly specced out HPC gear: IB, FPGAs, the works. You used to be able to get time on a Cray XC with an Ares interconnect, but I never have occasion to use it, so I don’t know if you still can. They’ve been aggressively hiring top-notch HPC people for a while.
Post reply on HN