Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

In S3 simplicity is table stakes

allthingsdistributed.com

Post reply on HN
Print view View original

71–80 of 142 posts

Re: In S3 simplicity is table stakes

#71

Post by imiric »

bentobean wrote: post #70

It's funny—S3 started as a "simple" storage service, and now it's handling entire table abstractions. Reminds me how SQL was declared dead every few years, yet here we are, building ever more complex data solutions on top of supposedly simple foundations.

I instinctively distrust any software or protocol that implies it is "simple" in its name: SNMP, SMTP, TFTP, SQS, etc. They're usually the cause of an equal or more amount of headaches than alternatives.

Maybe such solutions are a reaction to previous more "complex" solutions, and they do indeed start simple, but inevitably get swallowed by the complexity monster with age.

Re: In S3 simplicity is table stakes

#72

Post by user9999999999 »

user9999999999 wrote: post #43

if only metadata could be queried without processing a csv output file first, imagine storing thumbnails in there even! copied objects had actual events, not something you have to dig cloudtrail for, you could get last update time from a bucket to make caching easier

whoops. I was wrong! you can store base64 encoded metadata in the object then run a HEAD request to get it, but its limited to 2kb. also, you ~can~ query the metadata, but its latency is more suited to batch processing and not lambdas

Re: In S3 simplicity is table stakes

#73

Post by wodenokoto »

_fat_santa wrote: post #11

S3 is up there as one of my favorite tech products ever. Over the years I've used it for all sorts of things but most recently I've been using it to roll my own DB backup system. One of the things that shocks me about the system is the level of object durability. A few years ago I was taking an AWS certification course and learned that their durability number means that one can expect to loose data about once every 1…

I’ve never worked with AWS, but have a certification from GCP and currently use Azure.

What do you see as special for S3? Isn’t it just another bucket?

Re: In S3 simplicity is table stakes

#74

Post by shepherdjerred »

alberth wrote: post #26

S3 is the simplest CRUD app you could create. It's essentially just the 4 functions of C.R.U.D done to a file. Most problems in tech are not that simple. Note: not knocking the service. just pointing out not all things are so inherently basic (and valuable at the same time).

Anyone can create a CRUD API. It takes a _lot_ of work to make a CRUD API that scales with high availability and a reasonable consistency model. The vast majority of engineers would take months or years to write demo.

If you don't believe me, you might want to reconsider how skilled the average developer _really_ is.

Re: In S3 simplicity is table stakes

#75

Post by pavlov »

victorp13 wrote: post #16
_fat_santa wrote: post #11

S3 is up there as one of my favorite tech products ever. Over the years I've used it for all sorts of things but most recently I've been using it to roll my own DB backup system. One of the things that shocks me about the system is the level of object durability. A few years ago I was taking an AWS certification course and learned that their durability number means that one can expect to loose data about once every 1…

> Last thing I'll say is, you know your API is good when "S3 compatable API" is a selling point of your competitors. Counter-point: You know that you're the dominant player. See: .psd, .pdf, .xslx. Not particularly good file types, yet widely supported by competitor products.

Photoshop, PDF and Excel are all products that were truly much better than their competitors at the time of their introduction.

Every file format accumulates cruft over thirty years, especially when you have hundreds of millions of users and you have to expand the product for use cases the original developers never imagined. But that doesn’t mean the success wasn’t justified.

Re: In S3 simplicity is table stakes

#76

Post by jalk »

pavlov wrote: post #75
victorp13 wrote: post #16

Earlier quoted context omitted.

> Last thing I'll say is, you know your API is good when "S3 compatable API" is a selling point of your competitors. Counter-point: You know that you're the dominant player. See: .psd, .pdf, .xslx. Not particularly good file types, yet widely supported by competitor products.

Photoshop, PDF and Excel are all products that were truly much better than their competitors at the time of their introduction. Every file format accumulates cruft over thirty years, especially when you have hundreds of millions of users and you have to expand the product for use cases the original developers never imagined. But that doesn’t mean the success wasn’t justified.

PDF is not a product. I get what you are day but I can’t say that I’ve ever liked Adobe Acrobat

Re: In S3 simplicity is table stakes

#77

Post by great_wubwub »

imiric wrote: post #71
bentobean wrote: post #70

It's funny—S3 started as a "simple" storage service, and now it's handling entire table abstractions. Reminds me how SQL was declared dead every few years, yet here we are, building ever more complex data solutions on top of supposedly simple foundations.

I instinctively distrust any software or protocol that implies it is "simple" in its name: SNMP, SMTP, TFTP, SQS, etc. They're usually the cause of an equal or more amount of headaches than alternatives. Maybe such solutions are a reaction to previous more "complex" solutions, and they do indeed start simple, but inevitably get swallowed by the complexity monster with age.

TFTP is probably the exception to that rule. All the other protocols started out easily enough and added more and more cruft. TFTP stayed the way it's always been - minimalist, terrifyingly awful at most things, handy for a few corner cases. If you know when to use it and when to use something like SCP, you're golden.

If TFTP had gone the way of SNMP, we'd have 'tftp --proto tcp --tls --retries 8 --log-type json' or some horrendous mess like that.

Re: In S3 simplicity is table stakes

#78

Post by dekhn »

golly_ned wrote: post #66
alberth wrote: post #26

S3 is the simplest CRUD app you could create. It's essentially just the 4 functions of C.R.U.D done to a file. Most problems in tech are not that simple. Note: not knocking the service. just pointing out not all things are so inherently basic (and valuable at the same time).

A file system is simple. Open, read, close. Most tech problems are not that simple. How hard could a filesystem be?

Locking, checks after unclean shutdown, sparse files, high performance, reliabilty.... are all things that make filesystems harder.

Re: In S3 simplicity is table stakes

#80

Post by pavlov »

jalk wrote: post #76
pavlov wrote: post #75

Earlier quoted context omitted.

Photoshop, PDF and Excel are all products that were truly much better than their competitors at the time of their introduction. Every file format accumulates cruft over thirty years, especially when you have hundreds of millions of users and you have to expand the product for use cases the original developers never imagined. But that doesn’t mean the success wasn’t justified.

PDF is not a product. I get what you are day but I can’t say that I’ve ever liked Adobe Acrobat

PDF is a product, just like PostScript was.
Post reply on HN