Does anyone have recommendations on how to compress the data (gzip or parquet).
AWS S3: Sometimes you should press the $100k button
91–100 of 240 posts
Re: AWS S3: Sometimes you should press the $100k button
#92Earlier quoted context omitted.
The data protection regulations really are so freeing, huh. It's amazing to be able to delete all this stuff without worrying about having to keep it forever.
now this is a spin i havent heard before.
As a sysadmin I really wish you had. SO MANY problems have come to my desk because some dude 3 years ago did not consider retention or rotation and now I have to figure out what to do with a 4TB .txt that is apparently important.
Re: AWS S3: Sometimes you should press the $100k button
#93DON'T PRESS THAT BUTTON.
The egress and early deletion fees on those "cheaper options" killed a company that I had to step in and save.
Re: AWS S3: Sometimes you should press the $100k button
#94Earlier quoted context omitted.
In case of my previous employer it led to incredibly complicated encryption system. It took couple years to maybe implement in 10% of the system. Deleting any old data was rejected.
I wonder sometimes if it would help if we collectively watched more anti-hoarding shows, in order to see how the consultants convince their customers they can get rid of stuff.
humans started their first 300k years as nomads – storing was just impossible and decrufing happened by itself when moving along.
So maybe that's why we're not good at it yet.
Re: AWS S3: Sometimes you should press the $100k button
#95Your website renders as a big empty blue page in Firefox unless I disable tracking protection (and in my case, since I have noscript, I have to enable javascript for "website-files.com", a domain that sounds totally legit).
Noscript breaking websites? Who woulda thunk.
How do you manage to navigate the web with that on by default? It breaks just about everything since nothing is a static site these days.
Re: AWS S3: Sometimes you should press the $100k button
#96The AWS horror stories never cease to amaze me. It's like we're banging our heads against the wall expecting a different outcome each time. What's more frustrating, the AWS zealots are quite happy to tell you how you're doing it wrong. It's the users fault for misusing the service. The reality is, AWS was built for a specific purpose and demographic of user. It's now complexity and scale makes it unusable for newer d…
I'm not sure any sizable group is banging their head against a wall. Yes, AWS is complex. Yes, AWS has cost foot guns. These are natural outcomes of removing friction from scaling. Sure we could start with something simpler, but as you may have noticed, even the more basic hosting providers like DigitalOcean and Linode have been adding S3-compatible object storage because of its proven utility. In terms of making som…
Heroku did so much right. I recently was toying with some bot frameworks (think Discord or IRC, nothing spammy or review-gaming) and getting everything set up on a free tier dyno with free managed sql backing it up, and a github test/build integration, all took an hour or so. Really exceeded my expectations.
Not sure how it scales for production loads but my experience was so positive I'll probably go back for future projects.
Re: AWS S3: Sometimes you should press the $100k button
#97TL-DR: Object stores are not databases. Don't treat them like one.
Try telling that to developers; they love using S3 as both a database and a filesystem. It's gotten to the point where we need a training for new devs to tell them what not to do in the cloud.
Communicating through the filesystem is one of the Classic Blunders.
It doesn't come up as often anymore since we generally have so many options at our fingertips, but when push comes to shove you will still discover this idea rattling around in people's skulls.
Re: AWS S3: Sometimes you should press the $100k button
#98Earlier quoted context omitted.
Set a lifecycle rule to delete your objects. Come back a day later and AWS will have taken care of this for you.
The issue is this isn’t free. I played and emended up with a few hundred million object S3 bucket on a personal project and am trying to get rid of it without getting a bill. Seriously considering just getting suspended from aws if that’s a viable path lol.
Lifecycle rules are free. Use them to empty the bucket.
Re: AWS S3: Sometimes you should press the $100k button
#99DON'T PRESS THAT BUTTON. The egress and early deletion fees on those "cheaper options" killed a company that I had to step in and save.
On a related note, suppose the Fed raises rates to mitigate inflation and indirectly kills thousands of zombie companies, including many SaaS renting the cloud. What happens to their data? Does the cloud unilaterally evict/delete it, or does it get handled like an asset -- auctioned off, etc?
Re: AWS S3: Sometimes you should press the $100k button
#100Earlier quoted context omitted.
>There’s no delimiter. What's the delimiter parameter for then? https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObje...
To help you fool yourself. It affects how object list results are presented in the api response.
They can’t present a directory abstraction for list operations without a delimiter. E.g. CommonPrefixes.