111–120 of 139 posts
S3 isn’t a file system. It’s a key-value store. If you’re trying to use it as a file system, you’re doing it wrong :)
Unix directories map directory and file names(keys) to inode numbers(values). And a file's inode(key) is a map to the data blocks(values) on disk that make up a file's content.
S3 might not have Posix semantics but to say it's not a filesystem is incorrect.
Earlier quoted context omitted.
https://www.netapp.com/cloud-services/cloud-volumes-service-... You could try NetApp - it's significantly faster than EFS in my experience.
This solution is at least 40k per year for just the disk controllers, so be prepared for that.
https://aws.amazon.com/marketplace/pp/B07MF4GHYW
Depending on the use-case, Veritas InfoScale might be a better option. It can aggregate different tiers of storage, and present a multi-node shared-nothing file-system, with options for replication between AZs, regions, or different cloud providers.
Priced by core not capacity.
Earlier quoted context omitted.
> It usually starts with NFS, which is a big huge single point of failure. NFS is just the protocol. Whether it's a single point of failure depends on the server-side implementation. In Amazon EFS it is not. (disclaimer: I'm a PM-T on the EFS team)
Hi, my use case for EFS Lambda is burts of small writes. Could you add a perf mode that favors bursts of small writes? We did some perf tests and just couldn’t make it fast enough.
Earlier quoted context omitted.
If I understand the distinction you're making, then yes, you're wrong. The really beautiful thing about ObjectiveFS is that it's distributed, but the user doesn't really have to be concerned about that. A user can mount the same S3-backed ObjectiveFS filesystem on multiple machines, and they will somehow coordinate their reads and writes to that one S3 bucket, without having to communicate with any central component…
That's not what I meant by a distributed file system. Is the metadata layer of ObjectiveFS distributed - scale-out. Can you add nodes to increase the metadata layer's capacity and throughput, so it can scale to handle millions of ops/sec? That's what I meant by a distributed file system (not just a client server, with a single metadata server).
S3 isn’t a file system. It’s a key-value store. If you’re trying to use it as a file system, you’re doing it wrong :)
That's exactly what a filesystem is. Unix directories map directory and file names(keys) to inode numbers(values). And a file's inode(key) is a map to the data blocks(values) on disk that make up a file's content. S3 might not have Posix semantics but to say it's not a filesystem is incorrect.
Even if you drop the Unix (posix) part, most practically used file systems have features that are hard to guarantee in a distributed setting, and in either case simply don't exist in S3.
Earlier quoted context omitted.
> write your app against S3 and call it a day. ... and then later you notice that you need a special SLA and Amazon Redshift to guarantee that a read from S3 will return the same value as the last write. Even S3 is only eventually consistent and especially if a US user uploads images into your US bucket and then you serve the URLs to EU users, you might have loading problems. The correct solution, according to our su…
> S3 requests to US buckets might be served by delayed replicas in other countries if the request comes from outside the US What? That makes no sense. Do you have a source for that? I thought the explicit choice of region when creating a bucket limits where data is located. Why would the give you geo-replication for free? Also: "Amazon S3 creates buckets in a Region you specify. To optimize latency, minimize costs, o…
Mind the documented caveat for this case:
> The caveat is that if you make a HEAD or GET request to a key name before the object is created, then create the object shortly after that, a subsequent GET might not return the object due to eventual consistency.
Earlier quoted context omitted.
>There's EBS for that Don't you have to spin up an EC2 instance to use that?
Lambdas can use EBS these days, for persistent storage.
Earlier quoted context omitted.
That's exactly what a filesystem is. Unix directories map directory and file names(keys) to inode numbers(values). And a file's inode(key) is a map to the data blocks(values) on disk that make up a file's content. S3 might not have Posix semantics but to say it's not a filesystem is incorrect.
A Unix file system is not only a map from paths to inodes to bytes. Unix file systems support a host of additional apis, some of which are hard to implement on top of something like s3 (eg atomically renaming a parent path). Even if you drop the Unix (posix) part, most practically used file systems have features that are hard to guarantee in a distributed setting, and in either case simply don't exist in S3.
Also Nowhere did I say or even imply that a key/value store is all a Unix file system is. Different filesystems have different features. Object storage is a type of filesystem with a very specific feature set.
Earlier quoted context omitted.
"Amazon S3 automatically scales to high request rates. For example, your application can achieve at least 3,500 PUT/COPY/POST/DELETE or 5,500 GET/HEAD requests per second per prefix in a bucket. There are no limits to the number of prefixes in a bucket. You can increase your read or write performance by parallelizing reads. For example, if you create 10 prefixes in an Amazon S3 bucket to parallelize reads, you could…
S3 has quotas. Go try and get 1m ops/sec as a quota.
S3 isn't a file system? There's a reason it's called buckets. There's no 'renaming' of any file in S3. I don't think AWS had S3 as file system in mind. There's EBS for that.
> I don't think AWS had S3 as file system in mind. There's EBS for that. EBS isn't a file system, it's lower level than that (it's. block store, hence the name; you bring your own filesystem.) EFS and FSx are filesystems.