Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS S3: Sometimes you should press the $100k button

cyclic.sh

Post reply on HN
Print view View original

161–170 of 240 posts

Re: AWS S3: Sometimes you should press the $100k button

#161

Post by wackget »

abhishekjha wrote: post #151
wackget wrote: post #150

As a web developer who has never used anything except locally-hosted databases, can someone explain what kind of system actually produces billions or trillions of files which each need to be individually stored in a low-latency environment? And couldn't that data be stored in an actual database?

An image service.

Yeah that use-case I get. Binary files which would be difficult/impractical to index in a database.

However it feels like something at that scale will only ever realistically be dealt with by enterprise-level software, and I'd hazard a guess that most developers - even those reading HN - are not working on enterprise-level systems.

So I'm wondering what "regular devs" are using cloud buckets for at such a scale over regular DBs.

Re: AWS S3: Sometimes you should press the $100k button

#162

Post by wackget »

rgallagher27 wrote: post #152
wackget wrote: post #150

As a web developer who has never used anything except locally-hosted databases, can someone explain what kind of system actually produces billions or trillions of files which each need to be individually stored in a low-latency environment? And couldn't that data be stored in an actual database?

Things like mobile/webisite analytics events. User A clicked this menu item, User B viewed this images etc All streamed into S3 in chunks of smallish files. It's cheaper to store them in S3 over a DB and use tools like Athena or Redshift spectrum to query.

Wow. What makes it cheaper than using a DB? Is it just because the DB will create some additional metadata about each stored row or something?

Re: AWS S3: Sometimes you should press the $100k button

#163

Post by inopinatus »

throwhauser wrote: post #61
inopinatus wrote: post #38

Earlier quoted context omitted.

To help you fool yourself. It affects how object list results are presented in the api response.

"To help you fool yourself" seems like a euphemism for "to fool you". It's gotta be tough to go from "scant knowledge of S3" to genuine knowledge if the documentation is doing this to you. If the docs are misrepresenting the details, who can blame the author of the post?

The documentation is very clear on the purpose of the delimiter parameter.

The OP does not read the docs, makes bad assumptions repeatedly throughout, and then reaps the consequences.

Re: AWS S3: Sometimes you should press the $100k button

#164

Post by inopinatus »

asim wrote: post #60

The AWS horror stories never cease to amaze me. It's like we're banging our heads against the wall expecting a different outcome each time. What's more frustrating, the AWS zealots are quite happy to tell you how you're doing it wrong. It's the users fault for misusing the service. The reality is, AWS was built for a specific purpose and demographic of user. It's now complexity and scale makes it unusable for newer d…

That’s just it, though: it isn’t an AWS horror story. It’s the sorcerer’s apprentice.

Re: AWS S3: Sometimes you should press the $100k button

#165

Post by akdor1154 »

wodenokoto wrote: post #2

I've never been in this situation, but I do wish you could query files with more advanced filters on these blob storage services. - But why SageMaker? - Why do some orgs choose to put almost everything in 1 buckets?

> But why SageMaker?

You could ask the same thing of most times it gets used for ML stuff as well.

> Why do some orgs choose to put almost everything in 1 buckets?

Anecdote: ours does because we paid (Multinational Consulting Co)™ a couple of million to design our infra for us, and that's what the result was.

Re: AWS S3: Sometimes you should press the $100k button

#167

Post by rizkeyz »

I did the back-of-the-envelope math once. You get a Petabyte of storage today for $60K/year if you buy the hardware (retail disks, server, energy). It actually fits into the corner of a room. What do you get for $60K in AWS S3? Maybe a PB for 3 months (w/o egress).

If you replace all your hardware every year, the cloud is 4x more expensive. If you manage to use your getto-cloud for 5 year, you are 20x cheaper than Amazon.

To store one TB per person on this planet in 2022, it would take a mere $500M to do that. That's short change for a slightly bigger company these days.

I guess by 2030 we should be able to record everything a human says, sees, hears and speaks in an entire life for every human on this planet.

And by 2040 we should be able to have machines learning all about human life, expression and intelligence to slowly making sense of all of this.

Re: AWS S3: Sometimes you should press the $100k button

#168

Post by stingraycharles »

spelunker wrote: post #159
stingraycharles wrote: post #110

Earlier quoted context omitted.

How is encryption compliant? I’ve implemented GDPR data infrastructures twice now, and as far as I’m aware, the only way to be compliant with encryption is when you throw the decryption key away.

As mentioned, encrypt something and throw a way the key, often called "crypto shredding".

Ahh I see, and that way you can quickly “remove” a whole lot of data by just removing the key, which makes for cheap operations, and/or more flexible workflow (you can periodically compact the database and remove entries for which you have no key).

Is my understanding correct?

Re: AWS S3: Sometimes you should press the $100k button

#169

Post by dylan604 »

aeyes wrote: post #121
stingraycharles wrote: post #110

Earlier quoted context omitted.

How is encryption compliant? I’ve implemented GDPR data infrastructures twice now, and as far as I’m aware, the only way to be compliant with encryption is when you throw the decryption key away.

Sometimes it might be a single field in a 1MB nested structure that you have to remove. So it gets encrypted when the whole structure gets stored and when the field is to be deleted you just throw away the key instead of modifying the entire 1MB just to remove a few kB.

If you're comparing gov't regulations to delete data to saving a few KB, then I think you're looking at this wrong.

Re: AWS S3: Sometimes you should press the $100k button

#170

Post by dylan604 »

jabroni_salad wrote: post #92
whimsicalism wrote: post #68

Earlier quoted context omitted.

now this is a spin i havent heard before.

As a sysadmin I really wish you had. SO MANY problems have come to my desk because some dude 3 years ago did not consider retention or rotation and now I have to figure out what to do with a 4TB .txt that is apparently important.

Find out how important it is with a `mv 4TB.txt 4TB.old` type of things. See how many people come screaming
Post reply on HN