Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

In S3 simplicity is table stakes

allthingsdistributed.com

Post reply on HN
Print view View original

131–140 of 142 posts

Re: In S3 simplicity is table stakes

#131

Post by thiht »

progbits wrote: post #106
CobrastanJorji wrote: post #96

> When we moved S3 to a strong consistency model, the customer reception was stronger than any of us expected. This feels like one of those Apple-like stories about inventing and discovering an amazing, brand new feature that delighted customers but not mentioning the motivating factor of the competing products that already had it. A more honest sentence might have been "After years of customers complaining that the…

I mainly use GCP but keep hearing how great AWS is in comparison. Imagine my surprise when porting some GCS code to S3 last year and realizing there is no way to get consistency guarantees without external lock service.

> I mainly use GCP but keep hearing how great AWS is in comparison.

Where do you keep hearing this?

Having used both, AWS is trash in comparison. It's way to complicated to do anything simple. At work I wish we could migrate to GCP (or just something that's not AWS, really).

Re: In S3 simplicity is table stakes

#132

Post by JackSlateur »

ipsento606 wrote: post #54

Lots of comments here talking about how great S3 is. Anyone willing to give a cliff notes about what's good about it? I've been running various sites and apps for a decade, but have never touched S3 because the bandwidth costs are 1, sometimes 2 orders of magnitude more expensive than other static hosting solutions.

The bandwidth is free within AWS, the ingress is also free

I've used it lately in the following setup

  - stuff on internet push their data into my bucket => this is mostly free (you only pay s3 operations)
  - on object creation, an event is fired and a lambda is spawned => this is free
  - the lambda code reads the object => this is mostly free (again, you only pay s3 operations)
  - the lambda process the data, trim it, repackage it, compress it => you pay for the compute
  - the lambda store the resulting data somewhere else and delete the s3 object => you pay the egress
=> massive cost reduction

Re: In S3 simplicity is table stakes

#133

Post by yencabulator »

great_wubwub wrote: post #77
imiric wrote: post #71

Earlier quoted context omitted.

I instinctively distrust any software or protocol that implies it is "simple" in its name: SNMP, SMTP, TFTP, SQS, etc. They're usually the cause of an equal or more amount of headaches than alternatives. Maybe such solutions are a reaction to previous more "complex" solutions, and they do indeed start simple, but inevitably get swallowed by the complexity monster with age.

TFTP is probably the exception to that rule. All the other protocols started out easily enough and added more and more cruft. TFTP stayed the way it's always been - minimalist, terrifyingly awful at most things, handy for a few corner cases. If you know when to use it and when to use something like SCP, you're golden. If TFTP had gone the way of SNMP, we'd have 'tftp --proto tcp --tls --retries 8 --log-type json' or…

TFTP's usefulness in the modern day is strictly for things that don't have a TCP stack. Anything with a TCP stack is better off with HTTP. That doesn't leave much on the table except legacy & inertia.

Re: In S3 simplicity is table stakes

#134

Post by dangoodmanUT »

StratusBen wrote: post #95

For those interested in S3 Tables which is referenced in this blog post, we literally just published this overview on what they are and cost considerations of them that people might find interesting: https://www.vantage.sh/blog/amazon-s3-tables

> There are 1 million PUT requests and 10 million GET requests that month

> + 1,000,000 GET requests x ($0.004 / 1,000 requests) = $9

Re: In S3 simplicity is table stakes

#135

Post by TheNewsIsHere »

laluser wrote: post #19
_fat_santa wrote: post #11

S3 is up there as one of my favorite tech products ever. Over the years I've used it for all sorts of things but most recently I've been using it to roll my own DB backup system. One of the things that shocks me about the system is the level of object durability. A few years ago I was taking an AWS certification course and learned that their durability number means that one can expect to loose data about once every 1…

It’s designed for that level of durability, but it’s only as good as a single change or correlated set of hardware failures that can quickly change the theoretical durability model. Or even corrupting data is possible too.

This is true. While I prefer non-SaaS solutions generally, S3 is something that’s hard to cost effectively replace. I can setup an AWS account, create an S3 bucket, and have a system that can then persist at least one copy of my data to at least two data centers each within a goal of 1 second. And then layer cross-region replication if I need.

It’s by no means impossible to do that yourself, but it costs a lot more in time and upfront expense.

Re: In S3 simplicity is table stakes

#136

Post by TheNewsIsHere »

Cthulhu_ wrote: post #47
_fat_santa wrote: post #11

S3 is up there as one of my favorite tech products ever. Over the years I've used it for all sorts of things but most recently I've been using it to roll my own DB backup system. One of the things that shocks me about the system is the level of object durability. A few years ago I was taking an AWS certification course and learned that their durability number means that one can expect to loose data about once every 1…

I've used it for server backups too, just a simple webserver. Built a script that takes the webserver files, config files and makes a database dump, packages it all into a .tar.gz file on monday mornings, and uploads it to S3 using a "write only into this bucket" access key. In S3 I had it set up so it sends me an email whenever a new file was added, and that anything older than 3 weeks is put into cold storage. Of c…

If you haven’t already, make sure that versioning is enabled on that bucket!

Re: In S3 simplicity is table stakes

#137

Post by TheNewsIsHere »

8organicbits wrote: post #104
Gys wrote: post #25

Earlier quoted context omitted.

> their durability number means that one can expect to loose data about once every 10,000 years What does that mean? If I have 1 million objects, I loose 100 per year?

Isn't it just a marketing number? I didn't think durability was part of the S3 SLA, for example.

Object integrity isn’t part of the S3 SLA. I assume that is mostly because object integrity is something AWS can’t know about per se.

You could unknowingly upload a corrupted file, for example. By the time you discover that, there may not be a clear record of operations on that object. (Yes, you can record S3 data plane events but that’s not the point.)

Only the customer would know if their data is intact, and only the customer can ensure that.

The best S3 (or any storage system) can do is say “this is exactly what was uploaded”.

And you can overwrite files in S3 with the appropriate privileges. S3 will do what you ask if you have the proper credentials.

Otherwise, S3 is designed to be self-healing with erasure encoding and storing copies in at least two data centers per region.

Re: In S3 simplicity is table stakes

#138

Post by catlifeonmars »

TheNewsIsHere wrote: post #137
8organicbits wrote: post #104

Earlier quoted context omitted.

Isn't it just a marketing number? I didn't think durability was part of the S3 SLA, for example.

Object integrity isn’t part of the S3 SLA. I assume that is mostly because object integrity is something AWS can’t know about per se. You could unknowingly upload a corrupted file, for example. By the time you discover that, there may not be a clear record of operations on that object. (Yes, you can record S3 data plane events but that’s not the point.) Only the customer would know if their data is intact, and only t…

S3 supports checksumming, you just need to provide a hash in a header when you upload an object.

Re: In S3 simplicity is table stakes

#139

Post by progbits »

thiht wrote: post #131
progbits wrote: post #106

Earlier quoted context omitted.

I mainly use GCP but keep hearing how great AWS is in comparison. Imagine my surprise when porting some GCS code to S3 last year and realizing there is no way to get consistency guarantees without external lock service.

> I mainly use GCP but keep hearing how great AWS is in comparison. Where do you keep hearing this? Having used both, AWS is trash in comparison. It's way to complicated to do anything simple. At work I wish we could migrate to GCP (or just something that's not AWS, really).

All my coworkers who mainly used AWS before say that. I agree with you, but my AWS experience is quite limited, so what do I know.

But I feel like being used to something + grass being greener on other side both play a big role in these opinions.

Post reply on HN