Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

S3 is files, but not a filesystem

calpaterson.com

Post reply on HN
Print view View original

451–456 of 456 posts

Re: S3 is files, but not a filesystem

#451

Post by urda »

pclmulqdq wrote: post #422
urda wrote: post #421

Earlier quoted context omitted.

Sorry I am not buying the personal anecdote as the public numbers from both orgs tell a different story. When reliability and long term support come up in conversation, Google is not a name to reach for.

Note that I said "durability," not any other reliability metric. GCP is pretty well-known for its outages and abysmal support. It's a reputation they want to change, but they did earn it. However, Google is very good at not losing data.

Yup still not buying it. People who want their production stable, secure, and durable do not choose Google.

Re: S3 is files, but not a filesystem

#452

Post by bigiain »

flybarrel wrote: post #438
bigiain wrote: post #348

Earlier quoted context omitted.

At least 3, in at least 3 seperate datacenters. According to https://nuclearsecrecy.com/nukemap/ - it'd take at least a 1 megaton warhead to take out two of the ap-southeast-2 datacenters, and over 10MT to take out 3. I suspect you'd need a lot less than that though, the 1MT warhead would probably take out enough outside-the-datacenter infrastructure to take the entire AZ offline. I don't care too much though, if som…

1MT will take out the infrastructure to make the data not Available. However the data is still in the 3rd datacenter, making it still accessible therefore no compromising on Durable but yes we don't need the cat pictures when that is close to home :)

I'm guessing that even though the 3rd datacenter is about 35km away from the other 2, and so the building isn't in the expected destruction zone of a 1MT warhead, the damage to the city's electricity/water/network infrastructure would take the 3rd datacenter offline as well - so while your cat pictures are probably still in existence on the no-longer-spinning-rust there, they'd be unaccessible for quite some time.

Re: S3 is files, but not a filesystem

#453

Post by cuno »

rfoo wrote: post #393
cuno wrote: post #330

Earlier quoted context omitted.

IOPS is a really lazy benchmark that we believe can greatly diverge from most real life workloads, except for truly random I/O in applications such as databases. For example, in Machine Learning, training usually consists of taking large datasets (sometimes many PBs in scale), randomly shuffling them each Epoch, and feeding them into the engine as fast as possible. Because of this, we see storage vendors for ML workl…

> And if its pseudo-random then it is predictable, and if its predictable then we can exploit that to great effect This is an interesting hack. However, an IOP is an IOP, no matter how good you predicted it and prefetch it so that you hide the latency it's going to be translated to a GetObject. I think what you really exploited here is that even though S3 is built on HDDs (and have very low IOPS per TiB) their scale…

Sorry for the late response - I didn't see your comment until now.

Our aim is to unleash all the potential that S3/Object has to offer for file system workloads. Yes, the scale of AWS S3 helps, as does erasure coding (which enhances flexibility for better load balancing of reads).

Is it suitable for every possible workload? No, which is why we have a mode called cunoFS Fusion where we let people combine a regular high-performance filesystem for IOPS, and Object for throughput, with data automatically migrated between the two according to workload behaviour. What we find is that most data/workloads need high throughput rather than high IOPS, and this tends to be the bulk of data. So rather than paying for PBs of ultra-high IOPS storage, they only need to pay for TBs of it instead. Your particular workload might well need high IOPS, but a great many workloads do not. We do have organisations doing large scale workloads on time-series (market) data using cunoFS with S3 for performance reasons.

Re: S3 is files, but not a filesystem

#454

Post by fiddlerwoaroof »

afiori wrote: post #78
fiddlerwoaroof wrote: post #33

Earlier quoted context omitted.

> That’s something users do. The API doesn’t imply anything is related. Querying ids by prefix doesn’t make any sense for a normal ID type. Just making this operation available and part of your public API indicates that prefixes are semantically relevant to your API’s ID type.

by this logic the file "foo/bar/" correspond to the filename "f:o:o:/:b:a:r:/" (using a different caracter as separator)

Exactly

Re: S3 is files, but not a filesystem

#455

Post by calpaterson »

smcin wrote: post #444
smcin wrote: post #443

Earlier quoted context omitted.

A narrow interface handling a large number of concerns... makes the UNIX file API a "deep" module." Seems self-explanatory to me. "Deep module" [0][1] is a well-defined term. [0]: https://dev.to/gosukiwi/software-design-deep-modules-2on9 [1]: https://www.amazon.com/Philosophy-Software-Design-John-Ouste...

Honestly the only minor criticism I can see of the OP's writing is to remove things that make it seem more disjointed than it is - dashes, unqualified pronouns (what is "it"? AWS? UNIX file system API? a particular module? all modules?). That's all.

Thanks for your advice. I agree and will try to improve on this for the future

Re: S3 is files, but not a filesystem

#456

Post by benlivengood »

catlifeonmars wrote: post #450
benlivengood wrote: post #435

Earlier quoted context omitted.

It's not entirely orthogonal; RAID5 plus stripe-level CRC (or better) can reliably correct bitrot at any single position in a stripe whereas RAID5 alone can only report an error. My guess is that S3 and other large object stores have the equivalent of stripe-level checksums for this purpose.

I’m positive something like this is the case. Yet it’s entirely orthogonal to the object hash in the user facing feature, which would need to be computed separately.

For append-only or write-once objects or for BLAKE-3 and other fully parallelizable hashes it's possible to store the intermediate hash function state with each chunk or stripe so that the final bytes of the data, once the hash is finished, yield the user-facing checksum as well.
Post reply on HN