AWS doesn't make sense for scientific computing • Hacker News Board

Re: AWS doesn't make sense for scientific computing

#121

Post by harunurhan » Fri, Oct 07, 2022, 7:08 PM UTC

betolink wrote: post #105

I see both sides of the argument, there is a reason why CERN is not processing their data using EC2 and lambdas.

The cost isn't the only reason

- CERN started planning its computing grid before AWS was launched.

- It's pretty complicated (politics, mission, vision) for CERN to use external proprietary software/hardware for its main functions (they have even started to MS Office like products.)

- [cost] CERN is quite different than a small team researchers doing few years research. the scale is enormous and very long lived, like for decades continue

- and more...

HPC and scientific computing aside, I would have loved to be able to use AWS when I worked there, internal infra for running web apps and services wasn't nearly good & reliable, neither had a wide catalog of services offered.

Re: AWS doesn't make sense for scientific computing

#122

Post by latchkey » Fri, Oct 07, 2022, 7:14 PM UTC

lebovic wrote: post #91

latchkey wrote: post #61

I read this as a thinly veiled advertisement for the authors service, toolchest.

Toolchest actually runs scientific computing on AWS! I'm just frustrated by what we can build, because most scientific compute can't effectively shift to AWS

As others have noted, there are many other providers out there. I think your essay would have had more value if it didn't end with an advertisement.

Re: AWS doesn't make sense for scientific computing

#123

Post by betolink » Fri, Oct 07, 2022, 7:15 PM UTC

harunurhan wrote: post #121

betolink wrote: post #105

I see both sides of the argument, there is a reason why CERN is not processing their data using EC2 and lambdas.

The cost isn't the only reason - CERN started planning its computing grid before AWS was launched. - It's pretty complicated (politics, mission, vision) for CERN to use external proprietary software/hardware for its main functions (they have even started to MS Office like products.) - [cost] CERN is quite different than a small team researchers doing few years research. the scale is enormous and very long lived, like…

I think the spirit of the article is to put the cloud in perspective of the organization size and the workload type. There is a sweet spot where the cloud is the only option that makes sense, definitely with variable loads and capacity to basically scale on demand as big as our budget, there is no match for that. However... there are organizations with certain type of workloads that could afford to put infrastructure in place and even with the costs of staffing, energy etc they will save millions in the long run. NASA, CERN etc are some. This is not limited to HPC, the cloud at scale is not cheap either see: https://a16z.com/2021/05/27/cost-of-cloud-paradox-market-cap...

Re: AWS doesn't make sense for scientific computing

#124

Post by adolph » Fri, Oct 07, 2022, 7:16 PM UTC

lebovic wrote: post #16

aschleck wrote: post #7

This is sort of a confusing article because it assumes the premise of "you have a fixed hardware profile" and then argues within that context ("Most scientific computing runs on queues. These queues can be months long for the biggest supercomputers".) Of course if you're getting 100% utilization then you'll find better raw pricing (and this article conveniently leaves out staffing costs), but this model misses one of…

For fast-moving researchers who are blocked by a queue, cloud computing still makes sense. I guess I wasn't clear enough in the last section about how I still use AWS for startup-scale computational biology. My scientific computing startup (trytoolchest.com) is 100% built on top of AWS. Most scientific computing still happens on supercomputers in slower moving academic or big co settings. That's the group for whom cl…

Another service that runs on AWS is CodeOcean. It looks like Toolchest is oriented toward facilitating execution of specific packages rather than organization and execution like CodeOcean. Is that a fair summary?

https://codeocean.com/explore

Re: AWS doesn't make sense for scientific computing

#125

Post by Fomite » Fri, Oct 07, 2022, 7:18 PM UTC

One of the aspects not touched on for this is PII/confidential data/HIPAA data, etc.

For that, whether it makes sense or not, a lot of universities are moving to AWS, and the infrastructure cost of AWS for what would be a pretty modest server are still considerably less than the cost of complying with the policies and regulations involved in that.

Recently at my institution I asked about housing it on premise, and the answer was that IT supports AWS, and if I wanted to do something else, supporting that - as well as the responsibility for a breach - would rest entirely on my shoulders. Not doing that.

Re: AWS doesn't make sense for scientific computing

#126

Post by charcircuit » Fri, Oct 07, 2022, 7:20 PM UTC

dmicah wrote: post #116

secabeen wrote: post #109

Earlier quoted context omitted.

Thankfully, only a small part of the academic research enterprise involves human subjects, HIPAA, and all that. Neither fruit flies nor quarks have privacy rights.

Research involving human subjects (psychology, cognitive neuroscience, behavioral economics, etc.) requires institutional review board approval and informed consent, etc. but mostly doesn't involve HIPAA either.

That is not a law.

Re: AWS doesn't make sense for scientific computing

#127

Post by adolph » Fri, Oct 07, 2022, 7:21 PM UTC

bluedino wrote: post #120

We have 500-node cluster at a chemical company, and we've been experimenting with "hybrid-cloud". This allows jobs to use servers with resources we just don't have, or couldn't add fast enough. Storage is a huge issue for us. We have a petabyte of local storage from big name vendor that's bursting at the seams, and expensive to upgrade. A lot of our users leave big files laying around for a large time. Every few mont…

I can relate to these problems. Cloud brings positive accountability that is difficult to justify onprem. I have some hope that higher level tools for project/data/experiment management (as opposed to a bash prompt and a path) will bring some accountability without stifling flexibility.

Re: AWS doesn't make sense for scientific computing

#128

Post by Fomite » Fri, Oct 07, 2022, 7:21 PM UTC

billythemaniam wrote: post #86

crazygringo wrote: post #47

Earlier quoted context omitted.

> And it works just fine for them. Until it doesn't because there's a fire or huge power surge or whatever. That's the point -- there's a lot of risk they're not taking into account, and by focusing on the "it works just fine for them", you're cherry picking the ones that didn't suffer disaster.

The counterpoint to that point is that a significant percentage of scientific computing doesn't care about any of that. They are unlikely to have enough hardware to cause a fire and they don't care about outages or even data loss in many cases. As others have said, it depends on the specifics of the research. In the cases where that stuff matters, the cloud would be better option.

This. If my lab-level server failed tomorrow, I'd be annoyed, order another one, and start the simulations again.

Re: AWS doesn't make sense for scientific computing

#129

Post by Fomite » Fri, Oct 07, 2022, 7:23 PM UTC

dmicah wrote: post #116

secabeen wrote: post #109

Earlier quoted context omitted.

Thankfully, only a small part of the academic research enterprise involves human subjects, HIPAA, and all that. Neither fruit flies nor quarks have privacy rights.

Research involving human subjects (psychology, cognitive neuroscience, behavioral economics, etc.) requires institutional review board approval and informed consent, etc. but mostly doesn't involve HIPAA either.

And many, many institutions are over cautious. My own university, for example, has no data classification between "It would be totally okay if anyone in the university has access" and "Regulated data", so "I mean, it's health information, and it's governed by our data use agreement with the provider..." gets it kicked to the same level as full-fat HIPAA data.

Re: AWS doesn't make sense for scientific computing

#130

Post by CreRecombinase » Fri, Oct 07, 2022, 7:23 PM UTC

These MPI-based scientific computing applications make up a bulk of the compute hours on hpc clusters, but there is a crazy long tail of scientists who have workloads that can’t (or shouldn’t) run on their personal computers. The other option is HPC. This sucks for a ton of reasons, but I think the biggest one is that it’s more or less impossible to set up a persistent service of any kind. So no databases; if you want spark, be ready to spin it up from nothing every day (also no HDFS unless you spin that up in your SLURM job too). This makes getting work done harder but it also means that it makes integrating existing work so much harder because everyone’s workflow involves reinventing everything, and everyone does it in subtly incompatible ways; there are no natural (common) abstraction layers because there are no services.