The author makes a convincing argument against doing this workload on on-demand instances, but what about spot instances? AWS explicitly calls out scientific computing as a major use case for scientific computing in its training/promotional materials. Given the advertised ~70-90% markdown on spot instance time, it seems like a great option compared to paying almost the same amount as the workstation but not having to pay to buy, maintain, or replace the hardware.
AWS doesn't make sense for scientific computing
111–120 of 281 posts
Re: AWS doesn't make sense for scientific computing
#112Earlier quoted context omitted.
Generally, scientists aren't blocked while they are waiting on a computational queue. The results of a computation are needed eventually, but there is lots of other work that can be done that doesn't depend on a specific calculation.
It's good to learn how not to be blocked on long-running calculations. On the other hand, if transitioning to a bursty cloud model means you can do your full run in hours instead of weeks, that has real impact on how many iterations you can do and often does appreciably affect velocity.
It can, if you have the technical ability to write code that can leverage the scale-out that most bursty-cloud solutions entail. Coding for clustering can be pretty challenging, and I would generally recommend a user target a single large system with job that takes a week over trying to adapt that job to a clustered solution of 100 smaller systems that can complete it in 8 hours.
Re: AWS doesn't make sense for scientific computing
#113Having had the responsibility of providing HPC for a literal buildings full of scientists, I can say that it may be true that you can get computation cheaper with owned hardware, than in a cloud. Certainly pay as you go, individual project at a time processing will look that way to the scientist. But I can also say with confidence that the contest is far closer than they think. Scientists who make this argument almos…
Let me echo this as someone who once was responsible for HPC computing in a research intensive public university. Most career academics have NO IDEA how much enterprise computing infrastructure costs. If a 1 terabyte USB hard drive is $40 at Costco we (university IT) must be getting a much better deal than that. Take this argument and apply it to any aspect of HPC computing and that's what you're fighting against. Th…
It's kind of funny around this time of year when some researchers have $10,000 in their budget they need to spend, and they want to 'gift' us with some GPU's.
Re: AWS doesn't make sense for scientific computing
#114Why does no one consider colocation services anymore? And why do people only know Hetzner, OVH and Linode as alternatives to the big cloud providers? There are so many good and inexpensive server hosting providers, some with decades of experience.
Any particular you could recommend for GPU?
datacrunch.io has some 80G A100s
Re: AWS doesn't make sense for scientific computing
#115I think this post is identifying scientific computing with simulation studies and legacy workflows, to a fault. Scientific computing includes those things, but it also includes interactive analysis of very large datasets as well as workflows designed around cloud computing. Interactive analysis of large datasets (e.g. genome & exome sequencing studies with 100s of 1000s of samples) is well suited to low-latency, serv…
I've followed Hail and applaud the Broad Institute's work wrt establishing better bioinformatics software and toolkits so I hope this doesn't come as rude, but I can't imagine an instance in a real industry or academic workflow where you need 300ms feedback from an experiment to "maintain flow" considering how long experiments on data that large (especially exome sequencing!) take overall? My (likely lacking) imagination aside I guess what I'm really saying is that I don't know what's preventing the usecase you've described from being performed locally considering there'd be even less latency?
Re: AWS doesn't make sense for scientific computing
#116Earlier quoted context omitted.
I think it really depends on the task. Where HIPAA violation is a real threat, the equation changes. And just for CYA purposes those projects can get pushed to a cloud. Which does not necessarily involve any attempts to make them any more secure, but this is a different topic. That said, many scientists are operating on premise hardware like this: some servers in a shared rack and an el-cheapo storage solutions with…
Thankfully, only a small part of the academic research enterprise involves human subjects, HIPAA, and all that. Neither fruit flies nor quarks have privacy rights.
Research involving human subjects (psychology, cognitive neuroscience, behavioral economics, etc.) requires institutional review board approval and informed consent, etc. but mostly doesn't involve HIPAA either.
Re: AWS doesn't make sense for scientific computing
#117I imagine what makes this especially hard is you have (at least) three parties in play here:
- the people doing the research
- the institution's IT services group
- the administrator who writes the checks
And in my experience, "actual knowledge of what must be done and what it will or could cost" can vary greatly across these three groups; frequently in very unintuitive ways.
Re: AWS doesn't make sense for scientific computing
#118This has been my exact field of work for a few years now; in general I have found that: When people claim it is 10x more expensive to use public cloud, they have no earthly idea what it actually costs to run a HPC service, a data centre, or do any of the associated maintenance. When the claim is 3x more expensive in the cloud, they do know those things but are making a bad faith comparison because their job involves…
This is oversimplifying things a bit.
It can categorically be stated that for a year's worth of CPU compute, local will always be less than Amazon. Of course, putting percentages on it doesn't work - there are just too many variables.
There are many admins out there who have no idea what an Alpha is who'll swear that if you're not buying Dell or HP hardware at a premium with expensive support contracts, you're doing things wrong and you're not a real admin. Visit Reddit's /r/sysadmin if you want to see the kind of people I'm talking about.
The point is that if people insist on the most expensive, least efficient type of servers such as Dell Xeons with ridiculous service contracts, the savings over Amazon won't be large.
It's a cumulative problem, because trying to cool and house less efficient hardware requires more power and that hardware ultimately has less tolerance for non-datacenter cooling.
Rethink things. You can have AMD Threadripper / EPYC systems in larger rooms that require less overall cooling, that have better temperature tolerance, that're more reliable in aggregate, which cost less and for which you can easily keep around spare parts which would give better turnaround and availability than support contracts from Dell / HP. Suddenly your compute costs are halved, because of pricing, efficiency, overall power, real estate considerations...
So percentages don't work, but the bottom line is that when you're doing lots of compute, over time it's always cheaper locally, even if you do things the "traditional" expensive and inefficient way, so arguing percentages with so many variables doesn't make any sense - it's still cheaper, no matter what.
Re: AWS doesn't make sense for scientific computing
#119This rings true for me. I have a federal grant that prohibits me from using its funds for capital acquisitions: i.e. servers. But I can spend it on AWS at massive cost for minimal added utility for my use case. Even though it would be a far better use of taxpayer funds to buy the servers, I have to rent them instead.
> prohibits me from using its funds for capital acquisitions What is a legitimate reason for this restriction?
I can think of a few ways to abuse it while still spinning it as "for research". The obvious one is to buy a $9999 gaming machine with several of whatever the fanciest GPU on the market is at the time, and say you're doing machine learning.
So my guess is it's an overly broad patch for that sort of thing.
Re: AWS doesn't make sense for scientific computing
#120We have 500-node cluster at a chemical company, and we've been experimenting with "hybrid-cloud". This allows jobs to use servers with resources we just don't have, or couldn't add fast enough.
Storage is a huge issue for us. We have a petabyte of local storage from big name vendor that's bursting at the seams, and expensive to upgrade. A lot of our users leave big files laying around for a large time. Every few months we have to hound everyone to delete old stuff.
The other thing that you get with the cloud is there's way more accountability for who's using how much resources. Right now we just let people have access and roam free. Cloud HPC is 5-10x more in cost and the beancounters would shut shit down real quick if the actual costs were divvied up.
We also still have a legacy datacenter so in a similar vein, it's hard to say how much not having to deal with physical hardware/networking/power/bandwidth would be worth. Our work is maybe 1% of what that team does.