What's up with all the new AWS features in the past few days? Coincidence or not?
No coincidence. It’s reinvent season. https://reinvent.awsevents.com/
S3 Strong Consistency
31–40 of 240 posts
Re: S3 Strong Consistency
#32Re: S3 Strong Consistency
#33That's pretty incredible. With no price increase either. Using s3 as a makeshift database is possible in a lot more scenarios now. I especially like using s3 as a db when dealing with ETLs, where the state of some data is stored in its key prefix. This means that the etl can stop / resume at any point with a single source of truth as the database. The potential drawback is cost of course; moving (renaming) is free bu…
Yup me too! Checkout this state_doc Interface I wrote in Python for building a state machine where the state is stored in S3. I never bothered to solve for the edge case of inconsistent reads because my little state machine only ever measured in hours when it slept waiting for work and then minutes in-between waking up and doing work to verify restores.
With the news of S3's strong consistency my program is immediately safer to use for some set of defects, since the underlying datastore properties, S3 has made my codes naive/invalid assumptions, true.
https://github.com/remind101/dbsnap/blob/master/dbsnap_verif...
I should likely pull out the StateDoc class into it's own module.
Re: S3 Strong Consistency
#34Re: S3 Strong Consistency
#35What's up with all the new AWS features in the past few days? Coincidence or not?
[deleted]
Re: S3 Strong Consistency
#36Has anyone ever seen S3 behave eventually consistent? I have not seen a lot of eventual consistency in the real world but I wonder if I'm just working on the wrong problems?
I've never seen the minutes long delays that you were theoretically supposed to accommodate, but second-scale delays were pretty common in my team's experience. (We're writing a lot of files though, probably hundreds or thousands per second aggregated across all our customers.)
Yeah, I got seconds but people were saying there were minutes of data... max I ever saw was 3 seconds on a 2gb file...
Re: S3 Strong Consistency
#37The more I learn about S3 the more I feel the same awe I feel when looking at the Pyramids or Ankor Wat. > You can send 3,500 PUT/COPY/POST/DELETE or 5,500 GET/HEAD requests per second per prefix in an S3 bucket.
Side question -- why is s3fs so slow? I get only about 1 put per 10 seconds with s3fs. One would hope that you could use something like s3fs as a makeshift S3 API, but its performance is really horrible.
I can't answer this directly, but I thought I'd chime in with a possible alternative, or at least something else to test.
I'm using rclone with S3[1] (and others). There are commands[2] where you can use to just sync/copy/ files but it also has a mount option[3]. It also has caching[4] and other things that might help.
[2] https://rclone.org/commands/
Re: S3 Strong Consistency
#38Could someone describe a few real-life scenarios where this is useful and noticeable?
Re: S3 Strong Consistency
#39The more I learn about S3 the more I feel the same awe I feel when looking at the Pyramids or Ankor Wat. > You can send 3,500 PUT/COPY/POST/DELETE or 5,500 GET/HEAD requests per second per prefix in an S3 bucket.
Side question -- why is s3fs so slow? I get only about 1 put per 10 seconds with s3fs. One would hope that you could use something like s3fs as a makeshift S3 API, but its performance is really horrible.
One thing I noticed when I went digging through the s3fs source trying to answer that myself is that it's uploader is single threaded. Part of what makes AWS's cli reasonably fast is that it uses several threads to make sure to use as much bandwidth as possible. If I limit AWS's tool to one thread on a box with plenty of bandwidth, it turns my upload down from around 135 MiB/s to around 20 MiB/s
Not sure if that's all of it, or even the majority of the slowness.
Re: S3 Strong Consistency
#40This was a problem with data lakes / analytics. Small inconsistencies would trash runs; that problem now goes away and removes a lot of janky half-fixes to work around the issue. For most common use cases it's not really an issue.
Of course the real issue there is those tool treating S3 as something it's not (a filesystem) and building things on it with the expectation of certain guarantees that it explicitly didn't provide before now.
I dig the feature (strong consistency) but in some ways it just enables tools that were abusing it to just do so more easily.