61–70 of 139 posts
S3 isn't a file system? There's a reason it's called buckets. There's no 'renaming' of any file in S3. I don't think AWS had S3 as file system in mind. There's EBS for that.
Don't you have to spin up an EC2 instance to use that?
S3 isn't a file system? There's a reason it's called buckets. There's no 'renaming' of any file in S3. I don't think AWS had S3 as file system in mind. There's EBS for that.
S3 isn't a file system? There's a reason it's called buckets. There's no 'renaming' of any file in S3. I don't think AWS had S3 as file system in mind. There's EBS for that.
>There's EBS for that Don't you have to spin up an EC2 instance to use that?
S3 isn't a file system? There's a reason it's called buckets. There's no 'renaming' of any file in S3. I don't think AWS had S3 as file system in mind. There's EBS for that.
EBS isn't a file system, it's lower level than that (it's. block store, hence the name; you bring your own filesystem.) EFS and FSx are filesystems.
High-availability durable filesystem is a difficult problem to solve. It usually starts with NFS, which is a big huge single point of failure. Depending on the nature of the application this might be good enough. But if it's not, you'll typically want cross-datacenter replication so if one rack goes down you don't lose all your data. So then you're looking at something like Glusterfs/MooseFS/Ceph. But the latencies i…
> It usually starts with NFS, which is a big huge single point of failure. NFS is just the protocol. Whether it's a single point of failure depends on the server-side implementation. In Amazon EFS it is not. (disclaimer: I'm a PM-T on the EFS team)
Reading/writing SMALL files are SUPER slow on things like S3, Google Drive, Backblaze. Also using a lot of threads does only help a little bit but it's nowhere near reading/writing speeds of e.g. a single 600MB file. Is HopsFS helping in this area?
I have to disagree here. If you look at benchmarks on internet, yes, it will look like S3 is dead slow. But that is a client problem, not an S3 problem. For instance, boto3 (s3transfer) is an awful implementation that was so overengineered with a reimplementation of futures, task stealing, etc, that the download throughput is pathetic. Most often it will make you top below 200MB/s.
But S3 itself scales very well if you know how to use it, and skip boto3.
From my experience and benchmarks, each S3 connection will deliver up to 80MB/s, and with range requests you can easily have multiple parallel blocks downloaded.
I wrote a simple library that does this called s3pd (https://github.com/NewbiZ/s3pd). It's not perfect and is process based instead of coroutines, but that will give you an idea.
For reference, using s3pd I can saturate any EC2 network interface that I found (tested up to 10Gb/s interfaces, with download speed >1GB/s).
Boto is really doing bad press to S3.
I'm the author. Let me know if you have any questions.
What is the access protocol ? What tools am I using to access the POSIX presentation ?
High-availability durable filesystem is a difficult problem to solve. It usually starts with NFS, which is a big huge single point of failure. Depending on the nature of the application this might be good enough. But if it's not, you'll typically want cross-datacenter replication so if one rack goes down you don't lose all your data. So then you're looking at something like Glusterfs/MooseFS/Ceph. But the latencies i…
> EdgeFS is one I was looking at recently Do you have any additional info? EdgeFS's github[1] doesn't work; does repo access require a Nexenta sales call? We're also looking into asynchronously replicated FSs, I think built-in caching + tiering is slightly nicer than cron + rsync; would love to know what other solutions you looked into. [1] https://github.com/Nexenta/edgefs
Things like cron+rsync fail in boring ways.
Fancy things fail in fascinating ways.
Reading/writing SMALL files are SUPER slow on things like S3, Google Drive, Backblaze. Also using a lot of threads does only help a little bit but it's nowhere near reading/writing speeds of e.g. a single 600MB file. Is HopsFS helping in this area?
My experience was the opposite. Small files work acceptably quickly, the cost can't be beat, and the reliability probably beats out anything else I've seen. We saw latencies of ~50ms to write out small files. Slower than an SSD, yes, but not really that slow.
You do have to make sure you're not doing things like re-establishing the connection to get that, though. If you have to do a TLS handshake… it won't be fast. Also, in you're in Python, boto's API encourages, IMO, people to call `get_bucket`, which will do an existence check on the bucket (and thus, double the latency due to the extra round-trip); usually, you have out-of-band knowledge that the bucket will exist, and can skip the check. (If you're wrong, the subsequent object GET/PUT will error anyways.)