Amazon FSx for Lustre

#1

Post by mcrute » Wed, Nov 28, 2018, 4:33 PM UTC

Re: Amazon FSx for Lustre

#2

Post by davidmr » Wed, Nov 28, 2018, 5:06 PM UTC

This was inevitable, and I’m sure the engineering from Amazon’s side is impressive because Lustre is an absolute beast to run well at scale, but I’m not sure how great an idea it is for most people.

Coming from an academia HPC background and then moving into the private sector, I’ve mostly come to believe that parallel filesystems (especially POSIX-compliant ones) are rarely the right solution outside of MPI simulations. Like NFS, it makes it extremely easy and attractive to implement anti-patterns like using the filesystem for IPC or generating a bazillion files and then needing to reduce them to move to the next stage of the pipeline. In my experience, it’s rare that people don’t regret doing that sort of stuff in the long run.

That said, I’m sure the AWS team knows their customers and what they’re doing better than I do!

Re: Amazon FSx for Lustre

#3

Post by damnhungry » Wed, Nov 28, 2018, 5:12 PM UTC

I don't mean to be cynical but for the last few days some one is bombarding with lot of amazon news

Re: Amazon FSx for Lustre

#4

Post by SSilver2k2 » Wed, Nov 28, 2018, 5:13 PM UTC

damnhungry wrote: post #3

I don't mean to be cynical but for the last few days some one is bombarding with lot of amazon news

This week is the AWS re:Invent conference, so every day new services are being introduced.

Re: Amazon FSx for Lustre

#5

Post by colmmacc » Wed, Nov 28, 2018, 5:13 PM UTC

damnhungry wrote: post #3

I don't mean to be cynical but for the last few days some one is bombarding with lot of amazon news

AWS re:Invent, AWS's annual conference, is currently happening in Las Vegas. There are quite a number of headline announcements each day to go along with each keynote (there are 4 total).

Re: Amazon FSx for Lustre

#6

Post by fold_left » Wed, Nov 28, 2018, 5:15 PM UTC

finally, something that can handle node_modules! :)

Re: Amazon FSx for Lustre

#7

Post by damnhungry » Wed, Nov 28, 2018, 5:15 PM UTC

SSilver2k2 wrote: post #4

damnhungry wrote: post #3

I don't mean to be cynical but for the last few days some one is bombarding with lot of amazon news

This week is the AWS re:Invent conference, so every day new services are being introduced.

okay, my apologies :)

Re: Amazon FSx for Lustre

#8

Post by pinewurst » Wed, Nov 28, 2018, 5:41 PM UTC

That would have to be the worst job in the world - keeping Lustre going as an Amazon service, with management that utterly lacks understanding and sympathy.

Re: Amazon FSx for Lustre

#9

Post by davidmr » Wed, Nov 28, 2018, 5:45 PM UTC

pinewurst wrote: post #8

That would have to be the worst job in the world - keeping Lustre going as an Amazon service, with management that utterly lacks understanding and sympathy.

They’d have to pay me a lot of money to do it, that’s for sure. I’d love to see the disaster recovery plans. Every major Lustre site I’m aware of has had a data loss “incident” at some point in their history. It’s possible AWS has it all figured out with background backups and block device replication and whatnot, but I’m skeptical.

Re: Amazon FSx for Lustre

#10

Post by mbreese » Wed, Nov 28, 2018, 5:56 PM UTC

davidmr wrote: post #9

pinewurst wrote: post #8

That would have to be the worst job in the world - keeping Lustre going as an Amazon service, with management that utterly lacks understanding and sympathy.

They’d have to pay me a lot of money to do it, that’s for sure. I’d love to see the disaster recovery plans. Every major Lustre site I’m aware of has had a data loss “incident” at some point in their history. It’s possible AWS has it all figured out with background backups and block device replication and whatnot, but I’m skeptical.

Given that they call the non-S3 linked version 'ephemeral', I'm not sure there is a plan. I think S3 is the plan.