Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS doesn't make sense for scientific computing

noahlebovic.com

Post reply on HN
Print view View original

261–270 of 281 posts

Re: AWS doesn't make sense for scientific computing

#261

Post by hpcjoe »

anonymousDan wrote: post #246
hpcjoe wrote: post #239

Earlier quoted context omitted.

That's the Sentinel system. I worked on it when I was at Cray, and we did some covid stuff[1][2] with a researcher at UAH. We accelerated a docking code using some cool tech I created (in Perl, so there!) and some mods my teammates did to the queuing system. The work won some award at SC20[3] (fka Supercomputing conference). I had considered submitting for the Gordon Bell prize, which had been specifically requesting…

So in Azure it's possible to get access to an infiniband cluster somehow? Bare metal?

I don't speak for them (never have), but I believe it to be possible. MSFT do a number of things right (and a few really badly wrong), but you can generally spin up a decent bare metal system there. IO is going to be an issue with any cloud, it will cost for real performance. Between that and networking, clouds could potentially throw in the compute for free ...

Reminds me of a quip I made back in my SGI-Cray (1st time) days. A Cray supercomputer (back then) was a bunch of static ram that was sold, along with a free computer ... Not really true, but it gave a sense of the costs involved.

This said, Azure had (last I checked) real Mellanox networking kit for RDMA access. At Cray we placed a cluster in Azure for an end user (who shall rename nameless), and used several of Mellanox's largest switch frames for 100G Infiniband across > 1k nodes, each with many V100 GPUs. Unit would have been mid single digits on the top500 list that year.

AWS is doing their own thing network wise. Not nearly as good from a performance (latency or bandwidth) as the Mellanox kit. I don't know if Google Cloud is doing anything beyond TCP.

You can do bare metal at most/all of these. You can do some version of NVMe/local disk at all of them. Some/most let you spin up a parallel file system (network charges, so beware), either their own Lustre flavor, or one of BeeGFS, Weka, etc.

Re: AWS doesn't make sense for scientific computing

#262

Post by perihelions »

intelVISA wrote: post #191
boldlybold wrote: post #34

Earlier quoted context omitted.

Lots of places (Hetzner for example) will rent you servers at 10-25% the cost of AWS if you want dedicated hardware, without the ability to autoscale. You can even set up a K8s cluster there if the overhead is worth it.

Fond memories of Hetzner asking for my driving license as ID for renting a $2 VPS. Lost a customer for life with that nonsense.

Also lost a customer for life here. In my case they asked to let an AI scan my face through a webcam.

Nah.

Re: AWS doesn't make sense for scientific computing

#264

Post by chrchang523 »

pclmulqdq wrote: post #44
bee_rider wrote: post #42

Is genomic code typically distributed-memory parallel? I'm under the impression that it is more like batch processing, not a ton of node-to-node communication but you want lots of bandwidth and storage. If you are doing a big distributed-memory numerical simulation, on the other hand, you probably want infiniband I guess. AWS seems like an OK fit for the former, maybe not great for the latter...

The fastest way to do a lot of genomics stuff is with FPGA accelerators, which also aren't used by most of the other tenants in a multi-tenant scientific computing center. The cloud is perfect for that kind of work.

This is a bit overstated. Yes, FPGA accelerators can be used effectively in some common genomics workflows. However, my experience is that the right software running on regular Intel/AMD/ARM processors is very competitive with FPGA-using solutions.

Re: AWS doesn't make sense for scientific computing

#265

Post by halper »

absolutelynobo wrote: post #214
xani_ wrote: post #201

Earlier quoted context omitted.

No more or less chance that really anything else connected to power to catch on fire. Just make sure RCD works. Hell, some weeks ago my oven decided the lower heater's line is now connected to ground and blows fuses... Our 8 racks in DC only had single event of something blowing (power supply) and aside of smell and fuse blowing nothing really happened Servers are essentially metal boxes with a bit of glass-reinforce…

Cool how you use the acronym RCD and don’t expand it anywhere. Or any of the other homelab acronyms you used!

"Residual current device". It detects current leaking to ground; essentially it prevents you from killing someone by throwing a toaster into the bathtub.

Re: AWS doesn't make sense for scientific computing

#266

Post by esalman »

pclmulqdq wrote: post #36

Even as a big cloud detractor, I have to disagree with this. A lot of scientific computing doesn't need a persistent data center, since you are running a ton of simulations that only take a week or so, and scientific computing centers at big universities are a big expense that isn't always well-utilized. Also, when they are full, jobs can wait weeks to run. These computing centers have fairly high overhead, too, alth…

Neither grant agencies nor universities are ready to pay for commercial compute out of grant money. They'd rather have you run analysis on your work laptop/desktop they already provided. Even some of the folks who manage the HPC are unwilling to help researchers (many of whom are not programmers) to use the HPC, lest they mess up and damage the hardware. Source: I work at a tri-institutional collaboration research center in Georgia.

Re: AWS doesn't make sense for scientific computing

#267

Post by esalman »

Fomite wrote: post #132
secabeen wrote: post #112

Earlier quoted context omitted.

It can, if you have the technical ability to write code that can leverage the scale-out that most bursty-cloud solutions entail. Coding for clustering can be pretty challenging, and I would generally recommend a user target a single large system with job that takes a week over trying to adapt that job to a clustered solution of 100 smaller systems that can complete it in 8 hours.

This is a big part of it. In my lab, I have a lot of grad students who are computational scientists, not computer scientists. The time it will take them to optimize code far exceeds a quick-and-dirty job array on Slurm and then going back to working on the introduction of the paper, or catching up on the literature, or any one of a dozen other things.

Grad student here, I can attest to that.

Re: AWS doesn't make sense for scientific computing

#268

Post by esalman »

didip wrote: post #160

No way. I vehemently disagree. When a company reached a certain mass, hardware cost is a factor that is considered but not a big factor. The bigger problems are lost opportunity costs and unnecessary churns. Businesses lose a lot when the product launch is delayed by a year simply because the hardware arrived late or have too many defects (Ask your hardware fulfillment people how many defective RAM and SSD they got p…

Research labs are not businesses.

Re: AWS doesn't make sense for scientific computing

#269

Post by fuzzfactor »

whatever1 wrote: post #144
walnutclosefarm wrote: post #24

Having had the responsibility of providing HPC for a literal buildings full of scientists, I can say that it may be true that you can get computation cheaper with owned hardware, than in a cloud. Certainly pay as you go, individual project at a time processing will look that way to the scientist. But I can also say with confidence that the contest is far closer than they think. Scientists who make this argument almos…

Nah for us it was the department IT guy who set up once everything (a full cluster of 50 r720s) and works like a dream. Properly provisioned linux machines need no maintenance. You drive them until there is a hardware failure.

Sounds like you've got the kind of outstanding IT guy that was motivated to make the electronics run like the scientists wanted more so than anything.

At the other end of the spectrum there's labs [0] where the scientists need to carefully study and develop increasing skills at operating the electronics the way IT wants it done, and even worse of a distraction when there's a moving target keeping up with the IT electronics approach changing faster than the progress most labs make in their own scientific field.

What labs need more of is your kind of IT operator who can bring that option (your extreme end of the spectrum in favor of lab workers) within reach for when it is the most appropriate choice.

When labs fail to retain such adequate talent, they can rule out that option going forward, and that's one less tool in the toolbox.

[0] Including many which have good records of breakthrough progress before becoming computerized to begin with.

Re: AWS doesn't make sense for scientific computing

#270

Post by chrchang523 »

jiggawatts wrote: post #250
lebovic wrote: post #241

Earlier quoted context omitted.

Author here. It's clear that you genuinely believe my intent here is to be deceptive, so I think your comment deserves a thoughtful response. For context, I work on a computational biology infrastructure company that only uses cloud computing; my incentive is for more scientific computing to be on the cloud. Responses to your points below. Azure does have some better HPC infrastructure than AWS, so maybe some of my r…

> Azure does have some better HPC infrastructure than AWS That's somewhat surprising to hear, I just assumed AWS has equivalent products. My customer is 80% AWS and 20% Azure, so it's a useful data point to know that some HPC workloads are better off in Azure. > large CPU/RAM instances have availability issues, especially for spot instances. I've had two different ~128 vCPU instances running for days and days in my l…

"An example is the Smith-Waterman algorithm for genomics processing"

Sure, they can advertise that.

But are the vector units on Intel/AMD/ARM processors really not good enough?

"Debunking the 100X GPU vs. CPU Myth" is a somewhat controversial paper, but it lines up with my experience with common genomics workflows.

Post reply on HN