Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

S3 is files, but not a filesystem

calpaterson.com

Post reply on HN
Print view View original

391–400 of 456 posts

Re: S3 is files, but not a filesystem

#391

Post by orf »

jameshart wrote: post #313
electroly wrote: post #309

Earlier quoted context omitted.

No, don't do that. Set up a lifecycle rule that expires all of the objects and wait 24 hours. You won't pay for API calls and even the cost of storing the objects themselves is waived once they are marked for expiration. The article has a mistake about this too: expirations do NOT count as lifecycle transitions and you don't get charged as such. You will, of course, get charged if you prematurely delete objects that…

Still counts as nontrivial.

Literally 1 API call.

Re: S3 is files, but not a filesystem

#392

Post by rfoo »

alexchamberlain wrote: post #284
breckognize wrote: post #216

Earlier quoted context omitted.

Backing up across two different regions is possible for any provider with two "regions" but requires either doubling your storage footprint or accepting a latency hit because you have to make a roundtrip from Fremont to Denver. The neat thing about AWS' AZ architecture is that it's a sweet spot in the middle. They're far enough apart for good isolation, which provides durability and availability, but close enough tha…

> They're far enough apart for good isolation, which provides durability and availability It can't possibly be enough for critical data though, right? I'm guessing a fire in 1 is unlikely to spread to another, but could it affect the availability of another? What about a deliberate attack on the DCs or the utilities supplying the DCs?

> but could it affect the availability of another

Availability is a different beast than durability. I think people are paranoid here about durability instead of availability.

S3 advertises four nines availability and 12 nines durability.

Re: S3 is files, but not a filesystem

#393

Post by rfoo »

cuno wrote: post #330
crabbone wrote: post #269

Earlier quoted context omitted.

The tests are very weird... Normally, from someone working in the storage, you'd expect tests to be in IOPS, and the goto tool for reproducible tests is FIO. I mean, of course "reproducibility" is a very broad subject, but people are so used to this tool that they develop certain intuition and interpretation for it / its results. On the other hand, seeing throughput figures is kinda... it tells you very little about…

IOPS is a really lazy benchmark that we believe can greatly diverge from most real life workloads, except for truly random I/O in applications such as databases. For example, in Machine Learning, training usually consists of taking large datasets (sometimes many PBs in scale), randomly shuffling them each Epoch, and feeding them into the engine as fast as possible. Because of this, we see storage vendors for ML workl…

> And if its pseudo-random then it is predictable, and if its predictable then we can exploit that to great effect

This is an interesting hack. However, an IOP is an IOP, no matter how good you predicted it and prefetch it so that you hide the latency it's going to be translated to a GetObject.

I think what you really exploited here is that even though S3 is built on HDDs (and have very low IOPS per TiB) their scale is so large that even if you milk 1M+ IOPS out of it AWS still doesn't care and is happy to serve you. But if my back-of-envelope calculation is correct this isn't going to work well if everyone starts to do it.

How do you get around S3's 5.5k GET per second per prefix limit? If I only have ~200 20GiB files can you still get decent IOPS out of it?

and...

> IOPS is a really lazy benchmark that we believe can greatly diverge from most real life workloads

No, it's not. I have a workload training a DL model on time series data which demands 600k 8KiB IOPS per compute instance. None of the thing I tested work well. Had to build a custom one with bare metal NVMe-s.

Re: S3 is files, but not a filesystem

#394

Post by mattjaynes »

cynicalsecurity wrote: post #110

Backblaze B2 is worth mentioning while we are speaking of S3. I'm absolutely in love with their prices (3 times lower than of S3). (I'm not their representative).

Backblaze's B2 is cheap - but if your'e using them in production you must include these costs:

* their weekly 2 hour maintenance window 11:30-13:30 PST (which usually has no downtime, but sometimes is a full outage in the middle of the US day)

* having to file support tickets when your error rates increase above a usable threshold (for us about once a year for the last few years)

* support which does not look into the issue, just asks tons of questions as if they do not have error logs or any visibility on their end

* false success on uploads where B2 says it successfully saved your file but it is 0 bytes on their system (ALWAYS verify the upload despite B2's success code)

* extended outages if there's a high severity CVE (ex: they shut down for 10 hours for the Log4j2 CVE)

They have the best price - but when comparing options, it is simply not a directly comparable product to more mature cloud storage services.

(edit: formatting)

Re: S3 is files, but not a filesystem

#395

Post by mike_hearn »

jillesvangurp wrote: post #108
YouWhy wrote: post #23

The article is well written, but I am annoyed at the attempt to gatekeep the definition of a filesystem. Like literally any abstraction out there, filesystems are associated with a multitude of possible approaches with conceptually different semantics. It's a bit sophistic to say that Postgres cannot be run on S3 because S3 is not a filesystem; a better choice would have been to explore the underlying assumptions; (I…

The notion of postgres not being able to run on s3 has more to do with the characteristics of how it works than with it not being a filesystem. After all, people have developed fuse drivers for s3 so they can actually pretend it's a filesystem. But using that to store a database is going to end in tears for the same reasons that using e.g. NFS for this is also likely to end in tears. You might get it to work but it w…

Well, RocksDB never overwrites files except the manifest which is small. And you can write DB features on top of that. So that's an example of a database that can work with the S3 limitations.

Re: S3 is files, but not a filesystem

#396

Post by konfusinomicon »

tomcam wrote: post #371
dmvdoug wrote: post #362

Earlier quoted context omitted.

I’m picturing you having a slideshow of audit logs that you make guests to your home sit down and watch with you, like the vacation pictures slideshow of old.

Yes but my audit logs are special and everyone just loves them, although they playfully act bored

scrolling through them like it's the matrix can't be all that boring!

Re: S3 is files, but not a filesystem

#397

Post by [deleted] »

throwitaway222 wrote: post #274
dathery wrote: post #47

Earlier quoted context omitted.

Yes, which is why it's not ideal to reuse the folder metaphor here. Users have an idea how directories work on well-known filesystems and get confused when these fake folders don't behave the same way.

Are all your s3 keys opaque strings (like UUIDs)?, do you use / (slash) in your keys? If you truly believe S3 has absolutely no connection to folders, you would answer Yes and No.

[deleted]

Re: S3 is files, but not a filesystem

#398

Post by klodolph »

throwitaway222 wrote: post #274
dathery wrote: post #47

Earlier quoted context omitted.

Yes, which is why it's not ideal to reuse the folder metaphor here. Users have an idea how directories work on well-known filesystems and get confused when these fake folders don't behave the same way.

Are all your s3 keys opaque strings (like UUIDs)?, do you use / (slash) in your keys? If you truly believe S3 has absolutely no connection to folders, you would answer Yes and No.

It sounds to me like you’re arguing about what the definition of “folders” is.

“Any hierarchical path structure is a folder” is maybe your definition of “folder”, from what I can tell. I would say that S3 lets you treat paths as hierarchical, but that S3 does not have folders—obviously I have a different definition of “folder” than you do.

We’ve discovered that we have different definitions of “folder”, and therefore, we are not going to agree about whether it is true that “S3 does not have folders” unless we have an argument about what the correct definition of “folder” is. I’m not really interested in that discussion—it’s enough to understand what somebody means when they say “S3 does not have folders” even if you think their definitions are wrong.

Re: S3 is files, but not a filesystem

#400

Post by calpaterson »

readyman wrote: post #300
calpaterson wrote: post #139

Earlier quoted context omitted.

Author here, I can't reply to the GP because that comment is "dead" but yes, please, be specific! Then I can fix the sentences that are bad and perhaps also improve in the future.

"Even though the file API handles all those concerns, but it doesn't expose them to you. A narrow interface handling a large number of concerns - that makes the unix file API a "deep" module." Both sentences here are incomplete, incoherent. I did not read past this point.

Thanks. I take your point on the first one and have corrected it (maybe you need to shift+F5 to bust your cache in order to see it).

For the second one, what's your objection? That it's fragmentary?

Post reply on HN