Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

S3 is files, but not a filesystem

calpaterson.com

Post reply on HN
Print view View original

441–450 of 456 posts

Re: S3 is files, but not a filesystem

#441

Post by rakoo »

donatj wrote: post #189

> And listing files is slow. While the joy of Amazon S3 is that you can read and write at extremely, extremely, high bandwidths, listing out what is there is much much slower. Slower than a slow local filesystem. I was taken aback by this recently. At my coworkers request, I was putting some work into a script we have to manage assets in S3. It has a cache for the file listing, and my coworker who wrote it sent me hi…

> Why is this something Amazon has not fixed? From the outside really seems like they could slap some B-trees on the individual buckets and call it a day.

They fixed it already, it's called DynamoDB. With some SQS and Lambda glue you can index your S3 content in any way you want for later retrieval.

Re: S3 is files, but not a filesystem

#442

Post by tomcam »

dmvdoug wrote: post #362
tomcam wrote: post #355

Earlier quoted context omitted.

> if someone's dropped a warhead that close to home I have other things to worry about than whether all the cat pictures and audit logs survive. Speak for yourself. Many of us love our audit logs and show them to strangers whatever we can.

I’m picturing you having a slideshow of audit logs that you make guests to your home sit down and watch with you, like the vacation pictures slideshow of old.

Oh, like you don’t?

We’re among friends here.

Re: S3 is files, but not a filesystem

#443

Post by smcin »

readyman wrote: post #407
calpaterson wrote: post #400

Earlier quoted context omitted.

Thanks. I take your point on the first one and have corrected it (maybe you need to shift+F5 to bust your cache in order to see it). For the second one, what's your objection? That it's fragmentary?

I don't know what it's trying to say. Making it a complete sentence would be a good first step. Don't try fancy stuff like this unless showing off style is more important to you than communicating coherently. In technical writing, little is more important than clarity.

A narrow interface handling a large number of concerns... makes the UNIX file API a "deep" module."

Seems self-explanatory to me. "Deep module" [0][1] is a well-defined term.

[0]: https://dev.to/gosukiwi/software-design-deep-modules-2on9

[1]: https://www.amazon.com/Philosophy-Software-Design-John-Ouste...

Re: S3 is files, but not a filesystem

#444

Post by smcin »

smcin wrote: post #443
readyman wrote: post #407

Earlier quoted context omitted.

I don't know what it's trying to say. Making it a complete sentence would be a good first step. Don't try fancy stuff like this unless showing off style is more important to you than communicating coherently. In technical writing, little is more important than clarity.

A narrow interface handling a large number of concerns... makes the UNIX file API a "deep" module." Seems self-explanatory to me. "Deep module" [0][1] is a well-defined term. [0]: https://dev.to/gosukiwi/software-design-deep-modules-2on9 [1]: https://www.amazon.com/Philosophy-Software-Design-John-Ouste...

Honestly the only minor criticism I can see of the OP's writing is to remove things that make it seem more disjointed than it is - dashes, unqualified pronouns (what is "it"? AWS? UNIX file system API? a particular module? all modules?). That's all.

Re: S3 is files, but not a filesystem

#445

Post by IanCal »

tjoff wrote: post #166
orf wrote: post #158

Earlier quoted context omitted.

My overall point can be summarised as this: - Listing things is a very common operation to do. - The POSIX api and the directory/file hierarchy it provides is a restrictive one. - S3 does not suffer from this, you can recursively list and group keys into directories at “list time”. - If you find yourself needing to list gigantic numbers of keys in one go, you can do better by only listing a subset. S3 isn’t a filesys…

Listing things is very common, so why would you outsource that to S3 when all your bookkeeping is elsewhere? It's not like you would ever rely on the POSIX API for that anyway, even for when your files actually are on a POSIX filesystem. For sure, for maintenance tasks etc. it sounds quite useful. And good hygiene with prefixes sounds like a sane idea. But listing being a critical part of what "makes S3 useful"? That…

You may not need other bookkeeping. The prefix listing properties can be enough, removing the need to have two distinct systems kept in sync.

Re: S3 is files, but not a filesystem

#446

Post by anonymous-panda »

jmull wrote: post #411
anonymous-panda wrote: post #312

Earlier quoted context omitted.

This is a technical website discussing the nuances of filesystems. Common vernacular is how you choose to define it but even the Wikipedia definition says that directories and hierarchy are just one property of some filesystems. That they became the dominant model on local machines doesn’t take away from the more general definition that can describe distributed filesystems.

I'm kind of chuckling at this thread because you're working so hard to not understand. I think the previous poster could/should have said, "It is not a hierarchical file system and has no concept of directories." where I added the word "hierarchical". But it's also pretty obvious that was the point.

I disagree with that characterization because the contrast by OP was that S3 is “just a KV store implying” it doesn’t meet the criteria for being considered a filesystem.

For example, you could implement POSIX directory semantics on top of S3. About the only POSIX filesystem API you couldn’t implement it append / overwrite (well you could but it might be prohibitively expensive).

Re: S3 is files, but not a filesystem

#448

Post by looknohandsma »

logifail wrote: post #255
emodendroket wrote: post #246

Earlier quoted context omitted.

Sending the data to /dev/null is the cheapest option if that’s all you care about.

Seems the snark detector just went off :) Back on topic, I'd hope all of us would expect value for money for any and all services we recommend or purchase. Search for "site:news.ycombinator.com Away From AWS" to find dozens of discussions on how to save money by leaving AWS. EDIT: just one article of the many I've read recently: "What I’ve always found surprising about egress is just how expensive it is. On AWS, down…

And that is egress which works as expected, unlike the AWS S3 denial of wallet attack...: https://news.ycombinator.com/item?id=39625029

Re: S3 is files, but not a filesystem

#449

Post by overstay8930 »

EraYaN wrote: post #406
Kwpolska wrote: post #225

Earlier quoted context omitted.

How could they launch a cloud service like this and not have IPv6 in 2015? What other basic things did they cheap out on?

Most mayor cloud vendors are still not fully dual stack capable so it's not that surprising. And plenty of ISPs have barely started rollout, or even said they just wont.

AWS enabled dualstack S3 almost 10 years ago because object storage is pretty much the use case for IPv6.

I’m pretty sure the only other large object storage provider that is v4 only is Azure, and even then they offer a compatibility layer. Backblaze just flat out won’t work unless you pay extra to connect to them.

Honestly the only cloud provider I think you’re talking about is Azure, I don’t know of any other that are IPv4 only because it’s just cost prohibitive.

Re: S3 is files, but not a filesystem

#450

Post by catlifeonmars »

benlivengood wrote: post #435
catlifeonmars wrote: post #233

Earlier quoted context omitted.

The S3 checksum feature applies to the objects, so that’s entirely orthogonal to erasure codes. Unless you know something I don’t and SHA256 has commutative properties. You’d still need to compute the object hash independent of any blocks. Source: https://docs.aws.amazon.com/AmazonS3/latest/userguide/checki...

It's not entirely orthogonal; RAID5 plus stripe-level CRC (or better) can reliably correct bitrot at any single position in a stripe whereas RAID5 alone can only report an error. My guess is that S3 and other large object stores have the equivalent of stripe-level checksums for this purpose.

I’m positive something like this is the case. Yet it’s entirely orthogonal to the object hash in the user facing feature, which would need to be computed separately.
Post reply on HN