To store small files (10 to 1000 KB) uploaded by users in a web app, what is the best tool, S3 or EFS?
Amazon Elastic File System – Production-Ready in Three Regions
31–40 of 122 posts
Re: Amazon Elastic File System – Production-Ready in Three Regions
#32This is really cool but also kind of a shame. I feel like it will encourage bad behavior. I can see people using this to deploy code by deploying onto the shared volume so that all their instances get an "instant update". Which is great when it works, but lord help you when EFS goes down and every app server you have is hung on a broken NFS connection.
I have to admit that is the first use case that I thought of when I read this. No more messing around with AMIs and launch config whenever an update is deployed. You bring up a good point on the EFS being a point of failure though. What use case do you think the EFS is good for? It seems like even worse for data storage since IO can easily become a bottleneck.
> What use case do you think the EFS is good for
Out of band big data/data pipeline processing. Something where you want an easy way to sync your data but can handle extended downtimes.
Re: Amazon Elastic File System – Production-Ready in Three Regions
#33This is really cool but also kind of a shame. I feel like it will encourage bad behavior. I can see people using this to deploy code by deploying onto the shared volume so that all their instances get an "instant update". Which is great when it works, but lord help you when EFS goes down and every app server you have is hung on a broken NFS connection.
but thats the same with any component. how is that different to SNS et al? NFS is actually multi homed, so there is no real reason why it can't be HA/clustered, apart from the block store and the underlying file system.
> how is that different to SNS et al?
It's not really, and you should write your software to account for SNS outages, etc.
That being said, this is presented as NFS. NFS has a nasty habit of freezing your system if it breaks. If SNS breaks, you get errors and timeouts in your logs, but can keep going. If NFS breaks, you pretty much just sit around waiting for it to come back.
Re: Amazon Elastic File System – Production-Ready in Three Regions
#34Finally! My client was unable to use AWS for lacking of real POSIX fs.
Re: Amazon Elastic File System – Production-Ready in Three Regions
#35To store small files (10 to 1000 KB) uploaded by users in a web app, what is the best tool, S3 or EFS?
A benefit (usually) of S3 is that since it's operating over HTTP, you don't need any additional things to make the files accessible. In case of EFS you would need a HTTP server running. Maintaining a HTTP work is extra work, extra configuration but might give you additional benefits.
Thinking of small file size, EFS would probably work a bit faster (since you'd have some HTTP overhead) but I doubt that it would be significant, unless users upload hundreds of files at once.
EFS advantage in this case would be that you'd no longer need a S3 library to do it, since it's POSIX compliant. Given that most languages have simple, solid libraries for interacting with it, it's not a huge difference but still :)
All in all, I'd go with S3 unless there are some special requirements that are hard to satisfy using current S3 features.
Re: Amazon Elastic File System – Production-Ready in Three Regions
#36To store small files (10 to 1000 KB) uploaded by users in a web app, what is the best tool, S3 or EFS?
Object storage (i.e. s3+cloudfront) is still the correct way to store immutable files. that said, I know this will be used when the options are to rewrite filesystem access to an abstract (i.e. fs/s3/azure/etc) or just use nfs. People already use s3fs-fuse[1] which is dubious at best.
Re: Amazon Elastic File System – Production-Ready in Three Regions
#37Earlier quoted context omitted.
Isn't the whole point of AWS that you can blame AWS engineers for downtime instead of your own (in this case) code deployment thingy?
I'm not sure if you're being sarcastic or not, but no, it isn't. It's to provide building blocks for infrastructure. Down time is still on you. Only a lazy engineer blames their tools for their failures.
Not sarcastic: aws is not a tool, it's a service. You're exchanging dollars for SLA. The SLA is built in: you're sometimes (rarely) down, and someone else gets called about it.
Re: Amazon Elastic File System – Production-Ready in Three Regions
#38Earlier quoted context omitted.
Isn't the whole point of AWS that you can blame AWS engineers for downtime instead of your own (in this case) code deployment thingy?
I'm not sure if you're being sarcastic or not, but no, it isn't. It's to provide building blocks for infrastructure. Down time is still on you. Only a lazy engineer blames their tools for their failures.
[deleted]
Re: Amazon Elastic File System – Production-Ready in Three Regions
#39Earlier quoted context omitted.
Maybe but unfortunately the database doesn't support "backup store" that we can use directly S3, it just writes the data to specified directory in a filesystem. We should either fork the database or implement "S3 backed" filesystem and we picked the latter one. Unfortunately the performance FUSE is not that good for read and writing data to S3 is somehow expensive.
If you don't mind me asking, what DB is this?
We're implementing Clickhouse (http://clickhouse.yandex) to our open-source analytics platform Rakam. (https://github.com/rakam-io/rakam)
Re: Amazon Elastic File System – Production-Ready in Three Regions
#40Earlier quoted context omitted.
I'm not sure if you're being sarcastic or not, but no, it isn't. It's to provide building blocks for infrastructure. Down time is still on you. Only a lazy engineer blames their tools for their failures.
Not sarcastic: aws is not a tool, it's a service. You're exchanging dollars for SLA. The SLA is built in: you're sometimes (rarely) down, and someone else gets called about it.
Your customers will blame you: you don't want it to happen. Even if amazon pay you some damage you may lose trust or your whole business in the meantime.