I feel like a lot of applications can S3 but due to latency needs typically build a layer that sits in front that basically writes logs out to SSDs and then tiers to S3. If S3 offered a fast, reasonably priced Append() API that probably go a long way in capturing those use-cases.
S3 is files, but not a filesystem
341–350 of 456 posts
Re: S3 is files, but not a filesystem
#342> Filesystem software, especially databases, can't be ported to Amazon S3
Except they can be. You don't need to overwrite the whole DB file on every INSERT/UPDATE/DELETE; those can be (and often are) stored in memory and periodically checkpointed. You might lose some writes if the process goes down between checkpoints, but for a lot of applications that's entirely acceptable.
Indeed, for SQLite in particular there are tools like Litestream that support replication to and restoration from S3.
Alternately, you could split the DB across multiple files, and then an INSERT/UPDATE/DELETE would only need to overwrite the files actually affected. This is already how server-based RDBMSs usually work.
Re: S3 is files, but not a filesystem
#343> And listing files is slow. While the joy of Amazon S3 is that you can read and write at extremely, extremely, high bandwidths, listing out what is there is much much slower. Slower than a slow local filesystem. I was taken aback by this recently. At my coworkers request, I was putting some work into a script we have to manage assets in S3. It has a cache for the file listing, and my coworker who wrote it sent me hi…
> Why is this something Amazon has not fixed?
It's common to store metadata on DynamoDB where it can be queried, and just have whatever arbitrary links to the values in the buckets.
Re: S3 is files, but not a filesystem
#344> And listing files is slow. While the joy of Amazon S3 is that you can read and write at extremely, extremely, high bandwidths, listing out what is there is much much slower. Slower than a slow local filesystem. I was taken aback by this recently. At my coworkers request, I was putting some work into a script we have to manage assets in S3. It has a cache for the file listing, and my coworker who wrote it sent me hi…
It's not "fixed" because it's not a problem. You're just using it wrong.
Re: S3 is files, but not a filesystem
#345Earlier quoted context omitted.
Fun experiment I made with my mum, building a storage independent dropbox like UI [1] for anything that implement this interface: type IBackend interface { Ls(path string) ([]os.FileInfo, error) Cat(path string) (io.ReadCloser, error) Mkdir(path string) error Rm(path string) error Mv(from string, to string) error Save(path string, file io.Reader) error Touch(path string) error } My mum really couldn't care less about…
I think this interface is less interesting than the semantics behind it, particularly when it comes to concurrency: what happens when you delete a folder, and then try and create a file in that folder at the same time? What happens when you move a folder to a new location, and during that move, delete the new or old folders? Like yes, for your mum's use case, with a single user, it's probably not all that important t…
All those considerations are important when implementing the interface but the interface itself isn't invalidated by those concerns and can cope with those constraints fine.
Re: S3 is files, but not a filesystem
#346> I haven't heard of people having problems [with S3's Durability] but equally: I've never seen these claims tested. I am at least a bit curious about these claims. Believe the hype. S3's durability is industry leading and traditional file systems don't compare. It's not just the software - it's the physical infrastructure and safety culture. AWS' availability zone isolation is better than the other cloud providers.…
What’s your experience like at other storage outfits? I only ask because your post is a bit like singing praises for Cinnabon that they make their own dough. The things that you mentioned are standard storage company activities. Checksum-all-the-things is a basic feature of a lot of file systems. If you can already set up your home computer to detect bitrot and alert you, you can bet big storage vendors do it. Keepin…
As a point of order: Not all Cinnabon locations make their own dough. The one I worked at the summer of 2001 made their own dough and frosting that summer, but switched to premade rolls partially that holiday season, and fully to premade rolls and frosting by the 2002 holiday season.
Also: you had to be eighteen or older to operate the mixer. It was something like a 60 quart machine, and all the recipes were pre-programmed for time and power with pauses to change from the paddle to the whip if needed.
Re: S3 is files, but not a filesystem
#347It can be misused as a filesystem. S3 wants objects? Here are some 512 or 4096 byte objects called clusters…
Re: S3 is files, but not a filesystem
#348Earlier quoted context omitted.
If I were to upload a 50kb object to S3 (standard tier), about how many unique physical copies would exist?
At least 3.
At least 3, in at least 3 seperate datacenters.
According to https://nuclearsecrecy.com/nukemap/ - it'd take at least a 1 megaton warhead to take out two of the ap-southeast-2 datacenters, and over 10MT to take out 3.
I suspect you'd need a lot less than that though, the 1MT warhead would probably take out enough outside-the-datacenter infrastructure to take the entire AZ offline. I don't care too much though, if someone's dropped a warhead that close to home I have other things to worry about than whether all the cat pictures and audit logs survive.
Re: S3 is files, but not a filesystem
#349Earlier quoted context omitted.
That was not how we treated the 9's at Google. Those had been tested through natural experiments (disasters). I was not at Google for the Clichy fire, but it wasn't the first datacenter fire Google experienced. I think your information about Google's data placement may be incorrect, or you may be mapping AWS concepts onto Google internal infrastructure in the wrong way.
I would not lose sleep over storing data on GCS, but have heard from several Google Cloud folks that their concept of zones is a mirage at best.
I think also Google as a whole has pretty good diversity. But Cloud customers demanded regions in big population centers and smaller countries where Google traditionally avoided due to cost reasons. This lead to less redundant sites that were often owned and/or operated by third parties. So in the US and Europe you can probably trust GCP zones quite literally. But other regions (I have heard lots of rumours in the APAC) they may not be quite as diverse as they appear.
Re: S3 is files, but not a filesystem
#350Earlier quoted context omitted.
How do read-only filesystems align with your definition?
You can't create new things on a read-only filesystem, you can in S3; not a good analogy.
I wasn’t making an analogy. I was asking how read-only filesystem works given the parent commenters description of what makes something a filesystem.