41–50 of 240 posts
* Jeff Barr's blog post on this topic, has an interesting case study: https://aws.amazon.com/blogs/aws/amazon-s3-update-strong-rea...
* Collaboration with Hadoop/S3A maintainers leading to this release: https://aws.amazon.com/blogs/opensource/community-collaborat...
Could someone describe a few real-life scenarios where this is useful and noticeable?
Could someone describe a few real-life scenarios where this is useful and noticeable?
- step 1 generates/processes data, stores it in S3, overwriting the previous copy. triggers step 2 to run
- step 2 runs, fetches the data from s3 for its own processing. However, because only a few seconds have elapsed, step 2 fetches the old version of data from the S3 bucket
You can work around this by, for example, always using unique S3 object keys, but then you have to coordinate across the data processing steps, and it becomes harder to manage things like storing only the 10 latest versions.
The Argo workflow tool (https://argoproj.github.io/) is one example of a tool that can suffer from this problem.
Could someone describe a few real-life scenarios where this is useful and noticeable?
Because s3 was not strongly consistent, you would have the downstream job see a arbitrary subset of the files for a short while after creation, and not just the oldest files. This could cause your job to skip processing input files unless you provided some sort of manifest of all the files it would expect in that batch. So then you'd have to load the manifest, then keep retrying until all the input files showed up in whatever s3 node you were hitting.
This is one reason I have been a big fan of Google Cloud Storage over AWS S3: at a past company AWS consistency was a huge pain, and GCS has had this for years. https://cloud.google.com/blog/products/gcp/how-google-cloud-...
Could someone describe a few real-life scenarios where this is useful and noticeable?
Our image processing worker queues/servers write an S3 object and dispatch a follow up job. Currently we have to delay the next job (we use 5 seconds) otherwise then next job may start processing before the S3 object is available (it 404's if the next job is run straight away).