AWS doesn't make sense for scientific computing • Hacker News Board

Re: AWS doesn't make sense for scientific computing

#191

Post by intelVISA » Fri, Oct 07, 2022, 10:18 PM UTC

boldlybold wrote: post #34

idiot900 wrote: post #2

This rings true for me. I have a federal grant that prohibits me from using its funds for capital acquisitions: i.e. servers. But I can spend it on AWS at massive cost for minimal added utility for my use case. Even though it would be a far better use of taxpayer funds to buy the servers, I have to rent them instead.

Lots of places (Hetzner for example) will rent you servers at 10-25% the cost of AWS if you want dedicated hardware, without the ability to autoscale. You can even set up a K8s cluster there if the overhead is worth it.

Fond memories of Hetzner asking for my driving license as ID for renting a $2 VPS. Lost a customer for life with that nonsense.

Re: AWS doesn't make sense for scientific computing

#192

Post by jpeloquin » Fri, Oct 07, 2022, 10:18 PM UTC

adamsb6 wrote: post #85

I've never worked in this space, but I'm curious about the need for massive egress. What's driving the need to bring all that data back to the institution? Could whatever actions have to be performed on the data also be performed in AWS? Also while briefly looking into this I found that AWS has an egress waiver for researchers and educational instiutions: https://aws.amazon.com/blogs/publicsector/data-egress-waiver..…

Regarding the waiver—"The maximum discount is 15 percent of total monthly spending on AWS services". Was very excited at first.

As for leaving data in AWS, data is often (not always) revisited repeatedly for years after the fact. If new questions are raised about the results it's often much easier to check the output than rerun the analysis. And cloud storage is not cheap. But yes it sometimes makes sense to egress only summary statistics and discard the raw data.

Re: AWS doesn't make sense for scientific computing

#193

Post by treeman79 » Fri, Oct 07, 2022, 10:20 PM UTC

eastbound wrote: post #55

0xbadcafebee wrote: post #38

Earlier quoted context omitted.

I remember the first time a server caught fire in the closet we kept the rack in. Backups were kept on a server right below the one on fire. But, y'know, we saved money.

Don’t worry, we do incremental backups during weekdays and a full backup on Sunday. We use 2 tapes only, so one is always outside of the building. But you know, we saved money.

We had a million dollars worth of hardware installed in a closet. It had a portable AC hooked up that needed it’s water bin changed every so often.

Well I was in the middle of that. When the Director decided to show off the new security doors. So he closed the room up. Then found out that new security doors didn’t work. I find out as I’m coming back to turn AC back on. Room will get hot really fast.

We get office Security to unlock door. He says he doesn’t have authority. His supervisor will be by later in the day.

Completely deadpan, and in front of several VPs of a forth one 50.

I turn to guy to my right who lived nearby. “Go home and get your chainsaw”

We were quickly let in. Also got fast approval to install proper cooling.

Re: AWS doesn't make sense for scientific computing

#194

Post by systemvoltage » Fri, Oct 07, 2022, 10:31 PM UTC

gonzo41 wrote: post #190

systemvoltage wrote: post #172

Earlier quoted context omitted.

I am dealing with the exact opposite problem: "Oh you mean, we should leave the EC2 instance running 24/7 ??? No way, that would be too expensive"... to which I need to respond "No, it would be like $15/month. Trivial, stop worrying about costs in EC2 and S3, we're like 7 people here with 3 GB of data." I deal with Scientists that think AWS is some sort of a massively expensive enterprise thing. I can be, but not for…

Don't say the budget outloud near AWS. They'll find a way to help you spend it.

Hahaha, may be I need to just go into the AWS ether and start yakking big words like "Elastic Kubernetes Service" to confuse the scientists and get my aws fix. These people are too stingy. I want some shit running in AWS, what good is this admin IAM role.

Re: AWS doesn't make sense for scientific computing

#195

Post by [deleted] » Fri, Oct 07, 2022, 10:37 PM UTC

latchkey wrote: post #122

lebovic wrote: post #91

Earlier quoted context omitted.

Toolchest actually runs scientific computing on AWS! I'm just frustrated by what we can build, because most scientific compute can't effectively shift to AWS

As others have noted, there are many other providers out there. I think your essay would have had more value if it didn't end with an advertisement.

[deleted]

Re: AWS doesn't make sense for scientific computing

#196

Post by prpl » Fri, Oct 07, 2022, 10:46 PM UTC

walnutclosefarm wrote: post #24

Having had the responsibility of providing HPC for a literal buildings full of scientists, I can say that it may be true that you can get computation cheaper with owned hardware, than in a cloud. Certainly pay as you go, individual project at a time processing will look that way to the scientist. But I can also say with confidence that the contest is far closer than they think. Scientists who make this argument almos…

The things that tend to be "cheap" on campuses:

Power (especially if there is some kind of significant scientific facility on premise), space (especially in reused buildings), manpower (undergrads, grad students, post docs, professional post graduates), running old/reused hardware, etc...

You can get away with those at large research universities. Some of that you can get away with at national lab sorts of places (not going to find as much free/cheap labor, surplus hardware). If you start going down in scale/prestige, etc... none of that holds true.

Running a bunch of hardware from the surplus store in a closet somewhere with Lasko fans taped to the door is cheap. To some extent, the university system encourages such subsidies.

In any case, once you get to actually building a datacenter, if you have to factor in power, if you have a 4 hardware refresh cycle, professional staffing, etc... unless you are in one of those low CoL college towns - cloud is probably more more than 1.5 to 3x more expensive for compute (spot, etc...). Storage on prem is much cheaper - erasure coded storage systems are cheap to buy and run, and everybody wants their own high performance file system.

One continuing cloud obstacle though - researchers don't want to spend their time figuring out how to get their code friendly to preemptible VMs - which is the cost effective way to run on cloud.

Another real issue with sticking to on-prem HPC is talent acquisition and staff development. When you don't care about those things so much, it's easy to say it's cheap to run on-prem, but often the pay is crap for the required expertise, and ignoring cloud doesn't help your staff either.

Re: AWS doesn't make sense for scientific computing

#197

Post by onetimeusename » Fri, Oct 07, 2022, 10:47 PM UTC

walnutclosefarm wrote: post #24

Having had the responsibility of providing HPC for a literal buildings full of scientists, I can say that it may be true that you can get computation cheaper with owned hardware, than in a cloud. Certainly pay as you go, individual project at a time processing will look that way to the scientist. But I can also say with confidence that the contest is far closer than they think. Scientists who make this argument almos…

Is a postdoc hacking a cluster something you have seen before? I am genuinely curious because I worked on a cluster owned by my university as an undergrad and everyone was kind of assumed to be trusted. If you had shell access on the main node you could run any job you wanted on the cluster. You could enhance security I just wonder about this threat model, that's an interesting one. I am sure it happens to be clear.

Re: AWS doesn't make sense for scientific computing

#198

Post by xani_ » Fri, Oct 07, 2022, 10:51 PM UTC

It always was for load that doesn't allow for autoscaling to save you money; the savings were always from convenience of not having to do ops and pay for ops.

Then again a part of ops cost you save is paid again in dev salary that have to deal with AWS stuff instead of just throwing a blob of binaries and letting ops worry about the rest.

Re: AWS doesn't make sense for scientific computing

#199

Post by xani_ » Fri, Oct 07, 2022, 10:57 PM UTC

W-Stool wrote: post #187

pbronez wrote: post #173

Earlier quoted context omitted.

This is my fear about my homelab lol Fire extinguisher nearby, smart temp sensors, but still...

What are you using for a homelab priced temperature sensor?

Homelab-priced sensor is the temp sensor in your server, it's free! Actual servers have a bunch, usually have one at intake, "random old PC" servers can use motherboard temp as rough proxy for environment temp.

Hell, even in DC you can look at temperatures and see in front of which server technican was standing just by those sensors.

Second cheapest would be USB-to-1wire module + some DS18B20 1-sire sensors. Easy hobby job to make. They also come with unique ID which means if you put it in TSDB by that ID it doesn't matter where you plug those sensors.

Re: AWS doesn't make sense for scientific computing

#200

Post by timmclean » Fri, Oct 07, 2022, 11:01 PM UTC

lebovic wrote: post #177

somesortofthing wrote: post #111

The author makes a convincing argument against doing this workload on on-demand instances, but what about spot instances? AWS explicitly calls out scientific computing as a major use case for scientific computing in its training/promotional materials. Given the advertised ~70-90% markdown on spot instance time, it seems like a great option compared to paying almost the same amount as the workstation but not having to…

Author here! Spot instance pricing is better than on-demand, but it doesn't include data transfer, and it's still more expensive than on-prem/Hetzner/etc. Data transfer costs exceed the cost of the instance itself if you're transferring many TB off AWS. For of our more popular AWS instance types I use – a c5a.24xlarge, used for comparison in the post – the cheapest spot price over the past month in us-east-1 was $1.6…

FWIW, spot prices for c5a.24xlarge in us-east-2b and us-east-2c seem to have been under $0.92/hr for most of the last 3 months. So, assuming some flexibility on the choice of region, that would adjust your estimate to $0.92 / $1.69 * $1233.70/mo = $671.60/mo, which looks a lot more reasonable. Hopefully I did that math right. Data egress prices are definitely still ridiculous, I agree.