151–160 of 240 posts
As a web developer who has never used anything except locally-hosted databases, can someone explain what kind of system actually produces billions or trillions of files which each need to be individually stored in a low-latency environment? And couldn't that data be stored in an actual database?
As a web developer who has never used anything except locally-hosted databases, can someone explain what kind of system actually produces billions or trillions of files which each need to be individually stored in a low-latency environment? And couldn't that data be stored in an actual database?
It's cheaper to store them in S3 over a DB and use tools like Athena or Redshift spectrum to query.
I have caused billing spikes like this before those little warnings were invented and it was always a dark day. They are really a life saver. Lifecycle rules are also welcome. Writing them yourself was always a pain and tended to be expensive with list operations eating up that api calls bill. ---- Once I supported an app that dumped small objects into s3 and begged the dev team to store the small objects in oracle a…
Also S3 should no-op transitions for objects smaller than the break-even size for each storage class, even if you ask it to.
Earlier quoted context omitted.
Not really. Sometimes you actually have to understand things. If you're so concerned about your billing, someone on your team should probably invest a freaking hour to understand it. If that can't happen, you are just setting yourself up for failure.
I've been learning the ins and outs of the major 3 providers cloud billing setups for the last year, and I'm just getting started. This is not a 1 hour job, but you're right that someone in your team needs to understand it.
Earlier quoted context omitted.
Set a lifecycle rule to delete your objects. Come back a day later and AWS will have taken care of this for you.
The issue is this isn’t free. I played and emended up with a few hundred million object S3 bucket on a personal project and am trying to get rid of it without getting a bill. Seriously considering just getting suspended from aws if that’s a viable path lol.
From: https://docs.aws.amazon.com/AmazonS3/latest/userguide/lifecy...
Off topic: for people with a "million billion" objects, does the S3 console just completely freeze up for you? I have some large buckets that I'm unable to even interact with via the GUI. I've always wondered if my account is in some weird state or if performance is that bad for everyone. (This is a bucket with maybe 500 million objects, under a hundred terabytes)
Honestly this is when most folks move to using their own dashboards, metrics, and tooling. The AWS GUIs were designed for small to moderate use cases. You don't peer into a bucket with a billion objects and ask for a complete listing, or accounting of bytes. There are tools and APIs for that. That's what I do with my thousands of buckets and billions of files (dashboards).
Disclosure, co-founder here, we're building one of those CLIs. We started as an internal project at D2iQ (my co-founder Lukas commented further up), with tooling to collect an inventory of AWS resources and be able to search it easily.
Earlier quoted context omitted.
Their solution was to introduce entropy into the beginning of the object names, which used to be AWS's recommendation for how to ensure objects are placed in different partitions. AWS claims this is no longer necessary, although how their new design actually handles partitioning is opaque. "This S3 request rate performance increase removes any previous guidance to randomize object prefixes to achieve faster performan…
Seems like it's a much higher rate limit, but it exists none the less, and Shopify's scale has also grown significantly since 2018 (when that article was written) - so it was probably a valid way for them to go.
1) The performance per partition increased
2) The way AWS created partitions changed
When I was at Mozilla, one thing I worked on was Firefox's crash reporting system. It's S3 storage backend wrote raw crash data with the key in the format `{prefix}/v2/{name_of_thing}/{entropy}/{date}/{id}`. If I remember correctly, we considered this a limitation since the entropy was so far down in the key. However, when we talked to AWS Support they told us their was no longer a need to have the entropy early on; effectively S3 would "figure it out" and partition as needed.
EDIT: https://news.ycombinator.com/item?id=30373375 is a good related comment.
The AWS horror stories never cease to amaze me. It's like we're banging our heads against the wall expecting a different outcome each time. What's more frustrating, the AWS zealots are quite happy to tell you how you're doing it wrong. It's the users fault for misusing the service. The reality is, AWS was built for a specific purpose and demographic of user. It's now complexity and scale makes it unusable for newer d…
Actually, I disagree with one statement: "AWS was built for a specific purpose and demographic of user". AWS wasn't built for anyone. It was built for everyone, and is thus even reasonably productive for no one. AWS's entire product development methodology is "customer asks for this, build it"; there's no high level design, very few opinions, five different services can be deployed to do the same thing, it's absolute madness and getting worse every year. Azure's methodology is "copy whatever AWS is doing" (source: engineers inside Azure), so they inherit the same issues, which makes sense for Microsoft because they've always been an organization gone mad.
If there's one guiding light for Big Cloud, its: they're built to be sold to people who buy cloud resources. I don't even feel this is entirely accurate, given that this demographic of purchaser should at least, if nothing else, be considerate of the cost, and there's zero chance of Big Cloud winning that comparison without deceit, but if there was a demographic that's who it'd be.
> I'd argue, we need a completely new experience for the next generation.
Fortunately, the world is not all Big Cloud. The work Cloudflare is doing between Workers & Pages represents a really cool and productive application environment. Netlify is another. Products like Supabase do a cool job of vendoring open source tech with traditional SaaS ease-of-use, with fair billing. DigitalOcean is also becoming big in the "easy cloud" space, between Apps, their hosted databases, etc. Heroku still exists (though I feel they've done a very poor job of innovating recently, especially in the cost department).
The challenge really isn't in the lack of next-gen PaaS-like platforms; its in countering the hypnosis puked out by Big Cloud Sales in that they're the only "secure" "reliable" "whatever" option. This hypnosis has infected tons of otherwise very smart leaders. You ask these people "lets say we are at four nines now; how much are you willing to pay, per month, to reach five nines? and remember Jim, four-nines is one hour of downtime a year." No one can answer that. No one.
End point being: anyone who thinks Big Cloud will reign supreme forever hasn't studied history. Enterprise contracts make it impossible for them to clean the cobwebs from their closets. They will eventually become the next Oracle or IBM, and the cycle repeats. It's not an argument to always run your own infra or whatever; but it is an argument to lean on and support open source.
Earlier quoted context omitted.
In case of my previous employer it led to incredibly complicated encryption system. It took couple years to maybe implement in 10% of the system. Deleting any old data was rejected.
How is encryption compliant? I’ve implemented GDPR data infrastructures twice now, and as far as I’m aware, the only way to be compliant with encryption is when you throw the decryption key away.
I think it's really sad, because when I don't see docs clearly stating the limits, I assume the worst and avoid the service.