Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

S3 is files, but not a filesystem

calpaterson.com

Post reply on HN
Print view View original

291–300 of 456 posts

Re: S3 is files, but not a filesystem

#291

Post by yencabulator »

afiori wrote: post #83
zX41ZdbW wrote: post #38

Earlier quoted context omitted.

ClickHouse can work with S3 as a main storage. This is possible because a table is a set of immutable data parts. Data parts can be written once and deleted, possibly as a result of a background merge operation. S3 API is almost enough, except for cases of concurrent database updates. In this case, it is not possible to rely on S3 only because it does not support an atomic "write if not exists" operation. That's why…

Is a "write if not exists" atomic operation enouhg as a concurrency primitive for database locks?

When talking about analytical databases for "big data", yeah. They generally just want a "atomically replace the list of Parquet files that make up this table", with one writer succeeding at a time.

That would not be a great base to build a transactional database on.

Re: S3 is files, but not a filesystem

#292

Post by fsociety »

pclmulqdq wrote: post #287
breckognize wrote: post #266

Earlier quoted context omitted.

9's are overblown. When cloud providers report that, they're really saying "Assuming random hard drive failure at the rates we've historically measured and how we quickly we detect and fix those failures, what's the mean time to data loss". But that's burying the lede. By far the greatest risks to a file's durability are: 1. Bugs (which aren't captured by a durability model). This is mitigated by deploying slowly and…

That was not how we treated the 9's at Google. Those had been tested through natural experiments (disasters). I was not at Google for the Clichy fire, but it wasn't the first datacenter fire Google experienced. I think your information about Google's data placement may be incorrect, or you may be mapping AWS concepts onto Google internal infrastructure in the wrong way.

I would not lose sleep over storing data on GCS, but have heard from several Google Cloud folks that their concept of zones is a mirage at best.

Re: S3 is files, but not a filesystem

#293

Post by yencabulator »

zX41ZdbW wrote: post #38
YouWhy wrote: post #23

The article is well written, but I am annoyed at the attempt to gatekeep the definition of a filesystem. Like literally any abstraction out there, filesystems are associated with a multitude of possible approaches with conceptually different semantics. It's a bit sophistic to say that Postgres cannot be run on S3 because S3 is not a filesystem; a better choice would have been to explore the underlying assumptions; (I…

ClickHouse can work with S3 as a main storage. This is possible because a table is a set of immutable data parts. Data parts can be written once and deleted, possibly as a result of a background merge operation. S3 API is almost enough, except for cases of concurrent database updates. In this case, it is not possible to rely on S3 only because it does not support an atomic "write if not exists" operation. That's why…

Google Cloud Storage supports create-if-not-exist and compare-and-swap on generation counter. S3 is much harder to use as a building block without tying your code into a second system like DynamoDB etc.

https://pkg.go.dev/cloud.google.com/go/storage#Conditions

Re: S3 is files, but not a filesystem

#294

Post by pclmulqdq »

fsociety wrote: post #292
pclmulqdq wrote: post #287

Earlier quoted context omitted.

That was not how we treated the 9's at Google. Those had been tested through natural experiments (disasters). I was not at Google for the Clichy fire, but it wasn't the first datacenter fire Google experienced. I think your information about Google's data placement may be incorrect, or you may be mapping AWS concepts onto Google internal infrastructure in the wrong way.

I would not lose sleep over storing data on GCS, but have heard from several Google Cloud folks that their concept of zones is a mirage at best.

Yeah, that's definitely true. Google sort of mapped an AWS concept onto its own cluster splits. However, there are enough regional-scale outages at all the major clouds that I don't personally place much stock in the idea of zones to begin with. The only way to get close to true 24/7 five-9's uptime with clouds is to be multi-region (and preferably multi-cloud).

Re: S3 is files, but not a filesystem

#295

Post by surajrmal »

spintin wrote: post #275
breckognize wrote: post #192

> I haven't heard of people having problems [with S3's Durability] but equally: I've never seen these claims tested. I am at least a bit curious about these claims. Believe the hype. S3's durability is industry leading and traditional file systems don't compare. It's not just the software - it's the physical infrastructure and safety culture. AWS' availability zone isolation is better than the other cloud providers.…

Correct me if I'm wrong but bitrot only affects spinning rust since NAND uses ECC? If you see this I wonder if S3 is planning on adding hardlinks?

Nand is constantly moving around your data to prevent it from bit rotting. If you leave data too long without moving it, you may not be able to read the data from the nand.

Re: S3 is files, but not a filesystem

#296

Post by buremba »

mlhpdx wrote: post #218
zX41ZdbW wrote: post #38

Earlier quoted context omitted.

ClickHouse can work with S3 as a main storage. This is possible because a table is a set of immutable data parts. Data parts can be written once and deleted, possibly as a result of a background merge operation. S3 API is almost enough, except for cases of concurrent database updates. In this case, it is not possible to rely on S3 only because it does not support an atomic "write if not exists" operation. That's why…

Conditional PUT would be a great addition to S3, indeed.

That would probably require them to rewrite a non-trivial part of S3 from scratch.

Re: S3 is files, but not a filesystem

#297

Post by yencabulator »

chubot wrote: post #239
est31 wrote: post #227

Earlier quoted context omitted.

My employer (Neon) offers Postgres databases that run on top of a couple of caching layers at the end of which there is S3: https://neon.tech/docs/introduction/architecture-overview Directly exposing every write to S3 gives you the partial overwrite issues as described. But one can collect a bunch of traffic and push state to S3 once it reaches a threshold. Instead, a few writes in the postgres WAL are held outside o…

Thanks for the link. But I searched the docs for "durability" and got zero results. Before I use anything like this, I'd like to see what durability settings are used: https://www.postgresql.org/docs/current/non-durability.html Litestream documents the their data loss window, it seems like Neon should too: https://litestream.io/tips/ By default, Litestream will replicate new changes to an S3 replica every second. Dur…

Neon handles that by staging the WAL segments on 3x replicated Safekeeper nodes. Durability relies on not having all of those blow up at the same time. I'd expect it to be much safer than traditional Postgres replication mechanisms (with the trade-off having a comparatively large minimum node count; Neon really is built for multitenancy where that cost can be amortized across lots of databases).

Re: S3 is files, but not a filesystem

#298

Post by buremba »

jillesvangurp wrote: post #108
YouWhy wrote: post #23

The article is well written, but I am annoyed at the attempt to gatekeep the definition of a filesystem. Like literally any abstraction out there, filesystems are associated with a multitude of possible approaches with conceptually different semantics. It's a bit sophistic to say that Postgres cannot be run on S3 because S3 is not a filesystem; a better choice would have been to explore the underlying assumptions; (I…

The notion of postgres not being able to run on s3 has more to do with the characteristics of how it works than with it not being a filesystem. After all, people have developed fuse drivers for s3 so they can actually pretend it's a filesystem. But using that to store a database is going to end in tears for the same reasons that using e.g. NFS for this is also likely to end in tears. You might get it to work but it w…

Neon.tech runs Postgresql runs on S3. They persist the WAL to S3 so that they can replicate the data and bring it to local ssds I assume.

Re: S3 is files, but not a filesystem

#299

Post by fijiaarone »

cuno wrote: post #162
calpaterson wrote: post #137

Earlier quoted context omitted.

I have to say that I'm not hugely convinced. I don't really think that being able to pull out the keys before or after a prefix is particularly impressive. That is the basis for database indices going back to the 1970s after all. Perhaps the use-cases you're talking about are very different from mine. That's possible of course. But for me, often the slow speed of listing the bucket gets in the way. Your bucket doesn'…

We and our customers use S3 as a POSIX filesystem, and we generally find it faster than a local filesystem for many benchmarks. For listing directories we find it faster than Lustre (a real high performance filesystem). Our approach is to first try listing directories with a single ListObjectV2 (which on AWS S3 is in lexicographic order) and if it hasn't made much progress, we start listing with parallel ListObjectV2…

If you think s3 is fast, you should try FTP. It’s at least a hundred times faster. And combined with rsync, dozens of times more reliable.

Re: S3 is files, but not a filesystem

#300

Post by readyman »

calpaterson wrote: post #139
smcin wrote: post #50

Earlier quoted context omitted.

Their writing seems okay to me; which specific parts do you find atrocious?

Author here, I can't reply to the GP because that comment is "dead" but yes, please, be specific! Then I can fix the sentences that are bad and perhaps also improve in the future.

"Even though the file API handles all those concerns, but it doesn't expose them to you. A narrow interface handling a large number of concerns - that makes the unix file API a "deep" module."

Both sentences here are incomplete, incoherent. I did not read past this point.

Post reply on HN