Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

S3 Strong Consistency

aws.amazon.com

Post reply on HN
Print view View original

171–180 of 240 posts

Re: S3 Strong Consistency

#171

Post by speleding »

js4ever wrote: post #168
sudeepj wrote: post #129

Earlier quoted context omitted.

I have started working on this actually (the vfs being developed in Rust). The design was complex but now this simplifies a lot. The vfs will also have snapshotting capabilities i.e. you can have multiple versions of sqlite db.

How are you handling latency (up to 800ms on S3) or the fact that you have to write the whole file each time?

You could shard your SQLlite files to only contain the data for a single user, if your application allows you to do that. Would make it a bit painful to run migrations, but other than that you've now got an infinitely scaling database with 9 nines uptime for no money at all.

Re: S3 Strong Consistency

#172

Post by swyx »

darkr wrote: post #160
swyx wrote: post #154

Earlier quoted context omitted.

11 nines of reliability. I don't know anything else promising that (unclear if they actually reach it)

11 nines of _durability_. Uptime SLA for s3 is 99.9% AFAICR

thank you for the correction, I'm still pretty new to the backend world so I play too fast n loose with this terminology.

Re: S3 Strong Consistency

#174

Post by awssucs »

swyx wrote: post #157
awssucs wrote: post #149

“After a successful write of a new object or an overwrite of an existing object, any subsequent read request immediately receives the latest version of the object.” - being able to issue a read after a write has been the case in any system ever. AWS solving a fabricated issue, and the crowd cheering it as progress. Have to love herd mentality.

if you've ever worked on distributed storage you'll know that this isn't fabricated. it just takes effort to fix it. and S3 is pretty much the oldest distributed store in existence, lots of tech debt to overcome.

mmyeah never had this issue on non aws distributed storage. luckily for aws there are plenty of inexperienced devs who think this is a real issue in a properly designed large storage system.

Re: S3 Strong Consistency

#175

Post by awssucs »

jamesblonde wrote: post #151
awssucs wrote: post #149

“After a successful write of a new object or an overwrite of an existing object, any subsequent read request immediately receives the latest version of the object.” - being able to issue a read after a write has been the case in any system ever. AWS solving a fabricated issue, and the crowd cheering it as progress. Have to love herd mentality.

Previously, update-then-read was not guaranteed to be consistent. Now it is consistent.

yes for aws it was an issue, what i meant is that aws solved an issue that it itself created.

Re: S3 Strong Consistency

#176

Post by daviesliu »

ComodoHacker wrote: post #170
js4ever wrote: post #168

Earlier quoted context omitted.

How are you handling latency (up to 800ms on S3) or the fact that you have to write the whole file each time?

Obviously, a sane implementation would divide db file into blocks and write them into separate objects. Not as small as traditional FS blocks, 1-4 Mb, perhaps.

Yes, JuiceFS splits files into 4MB blocks and put them into S3.

Re: S3 Strong Consistency

#177

Post by ignoramous »

swyx wrote: post #154
damon_c wrote: post #18

The more I learn about S3 the more I feel the same awe I feel when looking at the Pyramids or Ankor Wat. > You can send 3,500 PUT/COPY/POST/DELETE or 5,500 GET/HEAD requests per second per prefix in an S3 bucket.

11 nines of reliability. I don't know anything else promising that (unclear if they actually reach it)

Route53 promises a 100% availability.

https://www.slideshare.net/AmazonWebServices/under-the-hood-...

https://aws.amazon.com/route53/sla/

Re: S3 Strong Consistency

#178

Post by pantulis »

tutfbhuf wrote: post #140

If someone is wondering why this is huge. It's because of the problems you face, when you don't have strong consistency. See this picture to get an understanding of what kind of problems you might run into with eventual consistency: https://i.ibb.co/DtxrRH3/eventual-consistency.png

Unbelievably good!

Re: S3 Strong Consistency

#179

Post by qt_scientist »

ashish-gandhi wrote: post #167
Scaevolus wrote: post #101

Earlier quoted context omitted.

That's analytics data, not file storage. If they have 34PB of analytics, they certainly have far more than that of blob storage.

Definitely much more of user data storage. More about that https://dropbox.tech/infrastructure/inside-the-magic-pocket . Happy to take other questions.

Did you consider building a S3 API in front of Magic Pocket so it could also serve as your analytics storage?

Or is your compute so bursty that it's more beneficial to be in the cloud on EC2 than to host on prem compute in front of Magic Pocket?

Re: S3 Strong Consistency

#180

Post by freemint »

rafaelturk wrote: post #165
tutfbhuf wrote: post #140

If someone is wondering why this is huge. It's because of the problems you face, when you don't have strong consistency. See this picture to get an understanding of what kind of problems you might run into with eventual consistency: https://i.ibb.co/DtxrRH3/eventual-consistency.png

I shall print this..

Please support the original artist and make sure you have a license to print it before you do.I
Post reply on HN