71–80 of 240 posts
Fantastic. Any write-ups or descriptions for how they made it happen?
[0] https://cloud.google.com/spanner/docs/true-time-external-con...
Earlier quoted context omitted.
Yes, we have. Specifically: 1. LIST 2. PUT 3. LIST would trigger situations where (3) wouldn't include the object inserted in (2). This is well-known however.
Also: 1. HEAD key -> 404 2. PUT key -> 200 3. GET key -> 404 (what? But I just put it!) This is commonly used for "upload file if it doesn't exist"
Could someone describe a few real-life scenarios where this is useful and noticeable?
We used to use S3 for Maven artifact storage. This is mostly an append-only workload, however Maven updates maven-metadata.xml files in place. These files contain info about what versions exist, when they were updated, what the latest snapshot version is, etc. We would see issues where a Maven build publishes to S3, and then a downstream build would read an out-of-date maven-metadata.xml and blow up. Or worse, it cou…
With this update, ordering the metadata update after the artifact upload means this failure is now impossible.
Earlier quoted context omitted.
Our image processing worker queues/servers write an S3 object and dispatch a follow up job. Currently we have to delay the next job (we use 5 seconds) otherwise then next job may start processing before the S3 object is available (it 404's if the next job is run straight away).
That sounds _very_ wrong, what type of throughput are we talking about here?
Disclosure: I work on Google Cloud. This is super awesome for customers. I am also beyond excited for all the open-source connectors to finally be simplified so that they don't have to deal with the "oh right, gotta be careful because of list consistency". It was super awesome when we were able to delete a huge chunk of the GCS Connector for Hadoop, and I hope to see the same across the S3-focused connectors. I'm now…
> It was super awesome when we were able to delete a huge chunk of the GCS Connector for Hadoop It's been a few years, but I lost customer data on gs:// due to listing consistency - create a bunch of files, list dir, rename them one by one to commit the dir into Hive - listing missed a file & the rename skipped the missing one. Put a breakpoint and the bug disappears, hit the rename and listing from the same JVM, the…
https://cloud.google.com/blog/products/gcp/how-google-cloud-...
We posted that blog in early 2018, but stated:
> Last year we migrated all of Cloud Storage metadata to Spanner, Google’s globally distributed and strongly consistent relational database.
So maybe pre-2017?
Earlier quoted context omitted.
Yeah, I never had that problem either. But I wonder how they are addressing the https://en.wikipedia.org/wiki/CAP_theorem
Sacrificing availability I reckon.
It's interesting to read all these comments here that talk about the eventual consistency like it was some kind of bug.
Are there any scenarios where you would want eventual consistency over strong consistency? (Assuming pricing, performance, replication etc are the same.)
It's interesting to read all these comments here that talk about the eventual consistency like it was some kind of bug.
Are there any scenarios where you would want eventual consistency over strong consistency? (Assuming pricing, performance, replication etc are the same.)