AWS doesn't make sense for scientific computing • Hacker News Board

Re: AWS doesn't make sense for scientific computing

#271

Post by jupp0r » Sat, Oct 08, 2022, 7:13 PM UTC

shadycuz wrote: post #245

jupp0r wrote: post #240

Having worked in the high performance computing field and in cloud hosted commercial applications, I can agree with the article but for entirely different reasons. The reason why some scientific computing shouldn't be done on AWS has to do with networking and latency between compute nodes. Supercomputers often use specialized networking hardware to get single digit microsecond latencies for data transfer between comp…

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/placemen...

This is using IP over ethernet in a tree, which has an order of magnitude more latency and less bandwidth than what you would see in a supercomputer, even if all your instances end up in the same rack.

Re: AWS doesn't make sense for scientific computing

#272

Post by lowbloodsugar » Sat, Oct 08, 2022, 7:25 PM UTC

lebovic wrote: post #189

lowbloodsugar wrote: post #153

>Even 2.5x over building your own infrastructure is significant for a $50M/yr supercomputer. Can’t imagine you are paying public prices on any cloud provider if you have a $50M/yr budget. In addition, if, as the article states, the scientists are ok to wait some considerable time for results, then one can run most, if not all, on spot instances, and that can save 10x right there. If you don’t have $50M/yr there are c…

Author here. I agree that pricing is highly negotiable for any large cloud provider, and there are even (capped) egress fee waivers that you can negotiate as a part of your contract. There's also a place for using AWS; I used it for a smaller DNA sequencing facility, and I use it for my computational biology startup. That said, I'll repeat something that I commented somewhere else: most of scientific computing (by %…

> 100 TB of raw data

Surely the raw data is input, so ingress costs, which is free?

The problem is if you have large amounts of intermediate data, and you want to transfer that somewhere else to continue analysis. Then it's "expensive". So the logical conclusion is to do all the work on the cloud, so you never have egress costs. That causes anxiety, sure. That said, 100TB costs $1000 to egress, assuming that your $50M/yr covers a Verizon 10GB/s line.

These opinions are my own and not those of my employer or former employer.

Re: AWS doesn't make sense for scientific computing

#273

Post by Moissanite » Sat, Oct 08, 2022, 9:51 PM UTC

davidmr wrote: post #180

pclmulqdq wrote: post #36

Even as a big cloud detractor, I have to disagree with this. A lot of scientific computing doesn't need a persistent data center, since you are running a ton of simulations that only take a week or so, and scientific computing centers at big universities are a big expense that isn't always well-utilized. Also, when they are full, jobs can wait weeks to run. These computing centers have fairly high overhead, too, alth…

This is tangential to your point, but I’ll just mention that Azure has some properly specced out HPC gear: IB, FPGAs, the works. You used to be able to get time on a Cray XC with an Ares interconnect, but I never have occasion to use it, so I don’t know if you still can. They’ve been aggressively hiring top-notch HPC people for a while.

The Azure FPGAs are a bit tangential from a customer perspective; they are just the equivalent of the AWS Nitro smart-NIC. Azure IB is interesting in that I originally expected it to be a killer feature, but for customers I work with it just isn't enough to overcome the multitude of downsides of having to use Azure for everything else. In the end, hardly any commercially relevant codes absolutely need IB, and work well enough with the low-latency ethernet both AWS and GCP offer.

Re: AWS doesn't make sense for scientific computing

#274

Post by Moissanite » Sat, Oct 08, 2022, 9:55 PM UTC

hpcjoe wrote: post #239

davidmr wrote: post #180

Earlier quoted context omitted.

This is tangential to your point, but I’ll just mention that Azure has some properly specced out HPC gear: IB, FPGAs, the works. You used to be able to get time on a Cray XC with an Ares interconnect, but I never have occasion to use it, so I don’t know if you still can. They’ve been aggressively hiring top-notch HPC people for a while.

That's the Sentinel system. I worked on it when I was at Cray, and we did some covid stuff[1][2] with a researcher at UAH. We accelerated a docking code using some cool tech I created (in Perl, so there!) and some mods my teammates did to the queuing system. The work won some award at SC20[3] (fka Supercomputing conference). I had considered submitting for the Gordon Bell prize, which had been specifically requesting…

The Azure Met Office win left me very conflicted. As someone who is relatively positive about cloud adoption for science it was good to see some forward thinking. On the other hand, what I've heard about how the procurement was run plus my taxpayer-based views on where critical national infrastructure should be housed makes me rather less happy about the outcome.

Re: AWS doesn't make sense for scientific computing

#275

Post by getcrunk » Sat, Oct 08, 2022, 10:09 PM UTC

funstuff007 wrote: post #205

W-Stool wrote: post #32

Earlier quoted context omitted.

Let me echo this as someone who once was responsible for HPC computing in a research intensive public university. Most career academics have NO IDEA how much enterprise computing infrastructure costs. If a 1 terabyte USB hard drive is $40 at Costco we (university IT) must be getting a much better deal than that. Take this argument and apply it to any aspect of HPC computing and that's what you're fighting against. Th…

I cannot wait for Ampere servers to become widespread. Most of the power and cooling issues will go away.

sarcasm?

Re: AWS doesn't make sense for scientific computing

#276

Post by funstuff007 » Sat, Oct 08, 2022, 11:28 PM UTC

walnutclosefarm wrote: post #258

funstuff007 wrote: post #234

Earlier quoted context omitted.

I appreciate your perspective, but it seems the security team should be watching for reverse proxies, tunnels, and other firewall anomalies for on-prem hardware just as a normal course of biz. And if a PI installs a self-managed server, that really should not gum up the works. All that being said, I have never worked at a place (or in a dept) whose threat profile made APT a real thing.

Obviously we do all those things. But you NEVER want an outside, and particularly APT owned machine inside your network. They can be well hidden and still do very real damage. You're fortunate if APTs don't consider you a worthy target. They are no joke, and in most cases are playing a long game, more interested in penetration, persistent presence, and quiet theft of information, than in doing anything you'd notice -…

Is an APT-owned server a significantly different risk than an APT-owned desktop? They are both inside your house.

Re: AWS doesn't make sense for scientific computing

#277

Post by aflag » Sun, Oct 09, 2022, 9:55 PM UTC

avereveard wrote: post #170

e63f67dd-065b wrote: post #167

Earlier quoted context omitted.

> hardware running 100% won't last five years Five year is a pretty typical amortisation schedule for HPC hardware. During my sysadmin days, of CPU, memory, cooling, power, storage, and networking, the only things that broke were hard disks and a few cooling fans. Disks were replaced by just grabbing a space and slotting it in, and fans were replaced by, well, swapping them out. Modern CPUs and memory last a very lon…

if you expect downtime in the 5 year to replace fan and whatnot, you're not getting 100% of your money/perf back - and I didn't see that in the article. if you have spares, spares need to be in the cost, and value lost to downtime stay minimal. but you have to include spares in the expenses. if you don't have spares, 1-2 day downtime is going to be a decent hit to value.

I think spot instances on amazon at least don't do partial hours, do they? So, you'll also have some wasted cycles there. Probably enough to compensate for your on-prem downtime.

Re: AWS doesn't make sense for scientific computing

#278

Post by KaiserPro » Mon, Oct 10, 2022, 10:08 AM UTC

bushbaba wrote: post #29

KaiserPro wrote: post #18

Its much more complex than described. The author is making a brilliant argument for getting a secondhand workstation and shoving under their desk. If you are doing multi machine batch style processing, then you won't be using ondemand, you'd use the spot pricing. The missing argument in that part is storage costs. Managing a high speed, highly available synchronous file system that can do a sustained 50gb/sec is hard…

Checkout Apache Iceberg which makes it fairly trivial to get high throughput from S3 without much fine-tuning. Bursts from 0 to 50Gbps should be possible from S3 without much effort, just have object sizes that are in the NN+ MiB range. Personally, Lustre is a mess, it's expensive and even more pain to fine-tune.

> Lustre is a mess, it's expensive and even more pain to fine-tune.

Its a huge raid-0, so long as your entire team understands that, you'll be ok. Its a lot better than in 2008, but now that AWS have a managed service, I'd just use that. (my heart is always in GPFS land...)

Re: AWS doesn't make sense for scientific computing

#279

Post by mattpallissard » Mon, Oct 10, 2022, 3:51 PM UTC

walnutclosefarm wrote: post #24

Having had the responsibility of providing HPC for a literal buildings full of scientists, I can say that it may be true that you can get computation cheaper with owned hardware, than in a cloud. Certainly pay as you go, individual project at a time processing will look that way to the scientist. But I can also say with confidence that the contest is far closer than they think. Scientists who make this argument almos…

I also managed HPC data centers and I agree with you. I feel like the term data center is a key word there. There is a point of scaling where it's just cheaper to manage a data center with a dedicated in house team. I think that holds true in other industries as well.

As far as HPC goes specifically, we could get some of the financial numbers to make sense in the cloud (cpu intensive jobs), but couldn't make it work for others (data intensive jobs shipping PB's around on the reg)

That and HPC has a lot of grant funding. It can be quite advantageous for an org to use an on prem data center almost like a slush fund. Can keep key projects running that would otherwise die when they are having a rough funding year.

Re: AWS doesn't make sense for scientific computing

#280

Post by kjs3 » Mon, Oct 10, 2022, 4:53 PM UTC

davidmr wrote: post #178

bluedino wrote: post #113

Earlier quoted context omitted.

It's kind of funny around this time of year when some researchers have $10,000 in their budget they need to spend, and they want to 'gift' us with some GPU's.

That was definitely one of the weirdest things of working in academia IT: “hey. Can you buy me a workstation that’s as close to $6,328.45 as it is possible to get, and can you do it by 4pm?”

Same thing happens in the government sector here (US). If you don't spend all of the budget you requested last year, you might not get it next year. There is an entire ecosystem of bottom-feeder GSA companies that apparently exist to spend year-end money that would otherwise go to 'waste'.