Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS doesn't make sense for scientific computing

noahlebovic.com

Post reply on HN
Print view View original

21–30 of 281 posts

Re: AWS doesn't make sense for scientific computing

#21

Post by julienchastang »

I’ve also been skeptical of the commercial cloud for scientific computing workflows. I don’t think this cost benefit analysis mentions it, but the commercial cloud makes even less sense when you take into account brick and mortar considerations. In other words, if your company/institution has already paid for the machine rooms, sys admins, networks, the physical buildings, the commercial cloud is even less appealing. This is especially true with “persistent services” for example data servers that are always on because they handle real-time data, for example.

Another aspect of scientific computing on the commercial cloud that’s a pain if you work in academia is procurement or paying for the cloud. Academic groups are much more comfortable with the grant model. They often operate on shoe-string budgets and are simply not comfortable entering a credit card number. You can also get commercial cloud grants, but they often lack long-term, multiyear continuity.

Re: AWS doesn't make sense for scientific computing

#22

Post by lostmsu »

adgjlsfhk1 wrote: post #20
idiot900 wrote: post #2

This rings true for me. I have a federal grant that prohibits me from using its funds for capital acquisitions: i.e. servers. But I can spend it on AWS at massive cost for minimal added utility for my use case. Even though it would be a far better use of taxpayer funds to buy the servers, I have to rent them instead.

Can you get your university to buy some servers for unrelated reasons and have them rent them to you?

Doesn't have to be a university either. Depending on the amount of compute needed any capable IT guy can do it for you from their garage with a contract.

Re: AWS doesn't make sense for scientific computing

#24

Post by walnutclosefarm »

Having had the responsibility of providing HPC for a literal buildings full of scientists, I can say that it may be true that you can get computation cheaper with owned hardware, than in a cloud. Certainly pay as you go, individual project at a time processing will look that way to the scientist. But I can also say with confidence that the contest is far closer than they think. Scientists who make this argument almost invariably leave major costs out of their calculation - assuming they can put their servers in a closet,maintain them themselves, do all the security infrastructure, provide redundancy and still get to shared compute when they have an overflow need. When the closet starts to smoke because they stuffed it with too many cheaply sourced, hot-running cores and GPUs, or gets hacked by one of their postdocs resulting in an institutional HIPAA violation, well, that's not their fault.

Put like for like in a well managed data center against negotiated and planned cloud services, and the former may still win, but it won't be dramatically cheaper, and figured over depreciable lifetime and including opportunity cost, may cost more. It takes work to figure out which is true.

Re: AWS doesn't make sense for scientific computing

#25

Post by lostmsu »

zatarc wrote: post #11

Why does no one consider colocation services anymore? And why do people only know Hetzner, OVH and Linode as alternatives to the big cloud providers? There are so many good and inexpensive server hosting providers, some with decades of experience.

Any particular you could recommend for GPU?

Re: AWS doesn't make sense for scientific computing

#26

Post by giantrobot »

idiot900 wrote: post #2

This rings true for me. I have a federal grant that prohibits me from using its funds for capital acquisitions: i.e. servers. But I can spend it on AWS at massive cost for minimal added utility for my use case. Even though it would be a far better use of taxpayer funds to buy the servers, I have to rent them instead.

I'm not saying AWS is automatically the best option but the question isn't just servers. It's servers, networking hardware, HVAC, a facility to put them all in, and at least a couple people to run and maintain it all. The TCO of some servers is way higher than the cost of the hardware.

Re: AWS doesn't make sense for scientific computing

#27

Post by jefftk »

secabeen wrote: post #10
aschleck wrote: post #7

This is sort of a confusing article because it assumes the premise of "you have a fixed hardware profile" and then argues within that context ("Most scientific computing runs on queues. These queues can be months long for the biggest supercomputers".) Of course if you're getting 100% utilization then you'll find better raw pricing (and this article conveniently leaves out staffing costs), but this model misses one of…

Generally, scientists aren't blocked while they are waiting on a computational queue. The results of a computation are needed eventually, but there is lots of other work that can be done that doesn't depend on a specific calculation.

It's good to learn how not to be blocked on long-running calculations.

On the other hand, if transitioning to a bursty cloud model means you can do your full run in hours instead of weeks, that has real impact on how many iterations you can do and often does appreciably affect velocity.

Re: AWS doesn't make sense for scientific computing

#28

Post by chrisseaton »

adgjlsfhk1 wrote: post #20
idiot900 wrote: post #2

This rings true for me. I have a federal grant that prohibits me from using its funds for capital acquisitions: i.e. servers. But I can spend it on AWS at massive cost for minimal added utility for my use case. Even though it would be a far better use of taxpayer funds to buy the servers, I have to rent them instead.

Can you get your university to buy some servers for unrelated reasons and have them rent them to you?

Well that’s just rebuilding AWS badly. I’ve used academic-managed time-sharing setups and have some horror stories.

Re: AWS doesn't make sense for scientific computing

#29

Post by bushbaba »

KaiserPro wrote: post #18

Its much more complex than described. The author is making a brilliant argument for getting a secondhand workstation and shoving under their desk. If you are doing multi machine batch style processing, then you won't be using ondemand, you'd use the spot pricing. The missing argument in that part is storage costs. Managing a high speed, highly available synchronous file system that can do a sustained 50gb/sec is hard…

Checkout Apache Iceberg which makes it fairly trivial to get high throughput from S3 without much fine-tuning. Bursts from 0 to 50Gbps should be possible from S3 without much effort, just have object sizes that are in the NN+ MiB range. Personally, Lustre is a mess, it's expensive and even more pain to fine-tune.

Re: AWS doesn't make sense for scientific computing

#30

Post by rpep »

I think there are some things this misses about the scientific ecosystem in Universities/etc. that can make the cloud more attractive than it first appears:

* If you want to run really big jobs e.g. with multiple multi-GPU nodes, this might not even be possible depending on your institution or your access. Most research-intensive Universities have a cluster but they’re not normally big machines. For regional and national machines, you usually have to bid for access for specific projects, and you might not be successful.

* You have control of exactly what hardware and OS you want on your nodes. Often you’re using an out of date RHEL version and despite spack and easybuild gaining ground, all too often you’re given a compiler and some old versions of libraries and that’s it.

* For many computationally intensive studies, your data transfer actually isn’t that large. For e.g. you can often do the post-processing on-node and then only get aggregate statistics about simulation runs out.

Post reply on HN