AWS doesn't make sense for scientific computing

I almost got a Tenure Track position at a Data Science Faculty in Virginia and I think them not having a HPC was the single issue that blew this move (from both sides). During interviews, I was asking the dean how they set up their HPC - turned out, they hadn't. I then asked a Professor in the next review round how they teach their students without a HPC:

> "I buy all resources on AWS - it's painfull because I have to contact AWS almost monthly for accidental over-billing, but we don't have a solution".

All of this made me really sceptical, since coming from a big University in Germany, we have unlimited HPC resources for free. I have 16 VMs, the biggest one 125 GB Memory, I can set those up or move around how I want. No space limitations - in need 10 TB of space for 3 month? Open a service ticket, 3 hours later it's available. Ports need to be opened worldwide to the web? No problem. Need a Jupyter Hub Cluster on Kubernetes? Here you go. This has really improved my work (quality, performance, and convenience) so much.

I was once coordinator of a research project where we had 30k EUR left and didn't know what to do with it. I contacted our HPC and asked if they want the money - answer: "30k really isn't worth the effort, we don't know what to do with it atm."

RCD is not a homelab acronym, its a type of circuit breaker.

So in Azure it's possible to get access to an infiniband cluster somehow? Bare metal?

I second this, would also ring my alarm bells in an interview. Back as a master student I went from a top 10 research university that didn't have a strong HPC infra at the time (ETH Zurich) to a top 100 one that did (Tokyo Tech), at least as a guest. Difference was staggering - all students just got handed a login on their big (top 100) cluster to be able to work with the actual hardware, even if limited to a couple of nodes. This much increased the likelihood that someone would be able to come up with an interesting project just together with their academic supervisor (no huge group projects with tons of extra funding needed).

> Azure does have some better HPC infrastructure than AWS

That's somewhat surprising to hear, I just assumed AWS has equivalent products. My customer is 80% AWS and 20% Azure, so it's a useful data point to know that some HPC workloads are better off in Azure.

> large CPU/RAM instances have availability issues, especially for spot instances.

I've had two different ~128 vCPU instances running for days and days in my lab environment, but that's probably because my region tends not to have a lot of HPC workloads that would compete for spot instances. I've noticed that "popular" sizes in Azure such as D4, D8, E4, and E8 are pre-empted regularly, but the "special" sizes like HPC not so much.

> surprising large charge for sufficiently large files.

Both Azure and AWS use this as a "roach motel" to encourage vendors and partners to co-locate in the same cloud. It's unfortunate that they charge on the order of $100 per terabyte, but it is what it is. However, bioinformatics files compress well, and compared to getting something out of an on-prem traditional network, the egress fees are a bargain.

My customer has a rural site with a "WAN" link. Their non-cloud option is to buy a NAS, replicate it to another NAS in a data center, and then build a permanent "file sharing solution". This is going to cost tens of thousands of dollars. They might share a few terabytes annually, which makes cloud egress fees look practically free in comparison.

> A new Illumina NovaSeq sequencer (about the size of a copy machine) with dual S4 flow cells produces 6Tb every couple days.

The scientists I talked to raised this, and to be honest I'm also concerned, especially as some of the groups I deal with are in rural areas "far from the cloud." (They analyse samples from cattle ranchers to try and prevent foot and mouth disease.)

Let's say the machine generates 6 terabytes in 2 days, so 3 TB daily. Assuming that's the uncompressed data, it'll be about 1 TB after compression. The location I'm thinking of has a 500 Mbps link, but that can transfer that data volume in just 4 hours: https://www.wolframalpha.com/input?i=%28+1+TB+%29+%2F+500+Mb...

One trick I discovered recently is that both the s3cmd and azcopy tools can take pipeline input. Combine that with a parallel compression tool like 'pigz' that can output to the pipeline, and you can have a workflow where the "raw" input files are compressed and streamed to the cloud storage at the same time. That alone can cut hours off the transfer time!

> If you have 100 TB of data, that increases the cost of re-running data – especially if it's in cold storage or archived.

Not necessarily. Azure Cold storage access has no special charges associated with it. It's a bit slower, but streaming reads were quite fast in my experience. Archive tier of course has some additional costs, but it's not a drama in most cases. For example, "high priority" retrieval costs extra, but normal priority appears to be free.

> what software are they using in bioinformatics that uses a GPU

From: https://developer.nvidia.com/blog/nvidia-hopper-architecture...

"An example is the Smith-Waterman algorithm for genomics processing".

Admittedly, GPU usage for genomics is still rare, but it is becoming more common.

> If you want instant scalability, the math is different.

In my example, think of 10-20 scientists doing semi-regular gene sequencing workloads and running related analyses. Sometimes needing a single machine with 2TB of memory, other times running 10,000 trivial jobs.

In principle, the flexibility of the cloud is nearly optimal for a scenario like this.