Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS doesn't make sense for scientific computing

noahlebovic.com

Post reply on HN
Print view View original

181–190 of 281 posts

Re: AWS doesn't make sense for scientific computing

#181

Post by davidmr »

lostmsu wrote: post #19
KaiserPro wrote: post #18

Its much more complex than described. The author is making a brilliant argument for getting a secondhand workstation and shoving under their desk. If you are doing multi machine batch style processing, then you won't be using ondemand, you'd use the spot pricing. The missing argument in that part is storage costs. Managing a high speed, highly available synchronous file system that can do a sustained 50gb/sec is hard…

Isn't 50GB sec like 5 NVMe Gen 5 SSDs + 1 or 2 for redundancy? Actually, you are right. Consumer SSDs I've seen only do about 1.5GB/s sustained.

Not in the context the person you responded to meant it. Yes, you can very easily get 50GB/s from a few NVMe devices on a single box. Getting 50GB/s on a POSIX-ish filesystem exported to 1000 servers is very possible and common, but orders of magnitude more complicated. 500GB/s is tougher still. 5TB/s is real tough, but real fun.

Re: AWS doesn't make sense for scientific computing

#182

Post by capableweb »

systemvoltage wrote: post #172
W-Stool wrote: post #32

Earlier quoted context omitted.

Let me echo this as someone who once was responsible for HPC computing in a research intensive public university. Most career academics have NO IDEA how much enterprise computing infrastructure costs. If a 1 terabyte USB hard drive is $40 at Costco we (university IT) must be getting a much better deal than that. Take this argument and apply it to any aspect of HPC computing and that's what you're fighting against. Th…

I am dealing with the exact opposite problem: "Oh you mean, we should leave the EC2 instance running 24/7 ??? No way, that would be too expensive"... to which I need to respond "No, it would be like $15/month. Trivial, stop worrying about costs in EC2 and S3, we're like 7 people here with 3 GB of data." I deal with Scientists that think AWS is some sort of a massively expensive enterprise thing. I can be, but not for…

> think AWS is some sort of a massively expensive enterprise thing

Compared to using dedicated instances with way cheaper bandwidth, storage and compute power, it might as well be.

Cloud makes sense when you have to scale up/down very quickly, or you'd be losing money fast. But most don't suffer from this problem.

Re: AWS doesn't make sense for scientific computing

#183

Post by davidmr »

avereveard wrote: post #170
e63f67dd-065b wrote: post #167

Earlier quoted context omitted.

> hardware running 100% won't last five years Five year is a pretty typical amortisation schedule for HPC hardware. During my sysadmin days, of CPU, memory, cooling, power, storage, and networking, the only things that broke were hard disks and a few cooling fans. Disks were replaced by just grabbing a space and slotting it in, and fans were replaced by, well, swapping them out. Modern CPUs and memory last a very lon…

if you expect downtime in the 5 year to replace fan and whatnot, you're not getting 100% of your money/perf back - and I didn't see that in the article. if you have spares, spares need to be in the cost, and value lost to downtime stay minimal. but you have to include spares in the expenses. if you don't have spares, 1-2 day downtime is going to be a decent hit to value.

I’m not sure I understand what you mean. I’ve run HPC clusters for a long time now, and node failures are just a fact of life. If 3 or 4 nodes of your 500 node cluster are down for a few days while you wait for RMA parts to arrive, you haven’t lost much value. Your cluster is still functioning at nearly peak capacity.

You have a handful of nodes that the cluster can’t function without (scheduler, fileservers, etc), but you buy spares and 24x7 contracts for those nodes.

Did I misunderstand your comment?

Re: AWS doesn't make sense for scientific computing

#184

Post by aflag »

avereveard wrote: post #159

> Hardware is amortized over five years hardware running 100% won't last five years if hardware is not needed to be running 100% at full steam for five years, you can turn down instances on the cloud and you don't pay anything in 2 years you'll be stuck with the same hardware, while on the cloud you follow cpu evolution as it arrives to the provider all in all the comparison is too high level to be useful

I've worked with a yarn cluster with around 200 nodes which ran non stop for well over 5 years and still kicking. There were a handful of failures and replacements, but I'd 95% of the cluster was fine 7 years in.

Re: AWS doesn't make sense for scientific computing

#185

Post by rovr138 »

pbronez wrote: post #173
0xbadcafebee wrote: post #38

Earlier quoted context omitted.

I remember the first time a server caught fire in the closet we kept the rack in. Backups were kept on a server right below the one on fire. But, y'know, we saved money.

This is my fear about my homelab lol Fire extinguisher nearby, smart temp sensors, but still...

oh, nice idea with temp sensor.

I have extinguishers all over the house, but hadn't considered a temperature sensor set to send alerts.

Do you have any recommendations?

Re: AWS doesn't make sense for scientific computing

#186

Post by insane_dreamer »

walnutclosefarm wrote: post #24

Having had the responsibility of providing HPC for a literal buildings full of scientists, I can say that it may be true that you can get computation cheaper with owned hardware, than in a cloud. Certainly pay as you go, individual project at a time processing will look that way to the scientist. But I can also say with confidence that the contest is far closer than they think. Scientists who make this argument almos…

Plus the supposed savings of in-house hardware only materialize if you have sufficiently managed and queued load to keep your servers running at 100% 24/7. The advantage of AWS/other is to be able to acquire the necessary amount of compute power for the duration that you need it.

For a large university it probably makes sense to have and manage their own compute infrastructure (cheap post-doc labor, ftw!) but for smaller outfits, AWS can make a lot of sense for scientific computing (said as someone who uses AWS for scientific computing), especially if you have fluctuating loads.

What works best IMO (and what we do) is have a minimum-to-moderate amount of compute resources in house that can satisfy the processing jobs most commonly run (and where you haven't had to overinvest in hardware), and then switch to AWS/other for heavier loads that run for a finite period.

Another problem with in-house hardware is that you spent all that money on Nvidia V100's a few years ago and now there's the A100 that blows it away, but you can't just switch and take advantage of it without another huge capital investment.

Re: AWS doesn't make sense for scientific computing

#187

Post by W-Stool »

pbronez wrote: post #173
0xbadcafebee wrote: post #38

Earlier quoted context omitted.

I remember the first time a server caught fire in the closet we kept the rack in. Backups were kept on a server right below the one on fire. But, y'know, we saved money.

This is my fear about my homelab lol Fire extinguisher nearby, smart temp sensors, but still...

What are you using for a homelab priced temperature sensor?

Re: AWS doesn't make sense for scientific computing

#188

Post by jpeloquin »

nharada wrote: post #176

I'd love to buy my own servers for small-scale (i.e. startup size or research lab size) projects, but it's very hard to be utilizing them 24x7. Does anyone know of open-source software or tools that allow multiple people to timeshare one of these? A big server full of A100s would be awesome, with the ability to reserve the server on specific days.

> the ability to reserve the server on specific days

In an environment where there are not too many users and everyone is cooperative, using Google Calendar to reserve time slots works very well and is very low maintenance. Technical restrictions are needed only when the users can't be trusted to stay out of each other's way.

Re: AWS doesn't make sense for scientific computing

#189

Post by lebovic »

lowbloodsugar wrote: post #153

>Even 2.5x over building your own infrastructure is significant for a $50M/yr supercomputer. Can’t imagine you are paying public prices on any cloud provider if you have a $50M/yr budget. In addition, if, as the article states, the scientists are ok to wait some considerable time for results, then one can run most, if not all, on spot instances, and that can save 10x right there. If you don’t have $50M/yr there are c…

Author here. I agree that pricing is highly negotiable for any large cloud provider, and there are even (capped) egress fee waivers that you can negotiate as a part of your contract. There's also a place for using AWS; I used it for a smaller DNA sequencing facility, and I use it for my computational biology startup.

That said, I'll repeat something that I commented somewhere else: most of scientific computing (by % of compute) happens in a context that still doesn't make sense in AWS. There's often a physical machine within the organization that's creating data (e.g. a DNA sequencer, particle accelerator, etc), and a well-maintained HPC cluster that analyzes that data.

Spot instances are still pretty expensive for a steady queue (2x of Hetzer monthly costs, for reference), and you still have to pay AWS data transfer egress costs – which are at least 30x more expensive than a colo or on-prem, if you're saturating a 1 Gbps link. Data transfer to optimize for spot instance pricing becomes prohibitive when your job has 100 TB of raw data.

Re: AWS doesn't make sense for scientific computing

#190

Post by gonzo41 »

systemvoltage wrote: post #172
W-Stool wrote: post #32

Earlier quoted context omitted.

Let me echo this as someone who once was responsible for HPC computing in a research intensive public university. Most career academics have NO IDEA how much enterprise computing infrastructure costs. If a 1 terabyte USB hard drive is $40 at Costco we (university IT) must be getting a much better deal than that. Take this argument and apply it to any aspect of HPC computing and that's what you're fighting against. Th…

I am dealing with the exact opposite problem: "Oh you mean, we should leave the EC2 instance running 24/7 ??? No way, that would be too expensive"... to which I need to respond "No, it would be like $15/month. Trivial, stop worrying about costs in EC2 and S3, we're like 7 people here with 3 GB of data." I deal with Scientists that think AWS is some sort of a massively expensive enterprise thing. I can be, but not for…

Don't say the budget outloud near AWS. They'll find a way to help you spend it.
Post reply on HN