A terrible, horrible, no-good, very bad day at Slack

Post by foobiekr » Tue, Jul 07, 2020, 6:21 PM UTC

Jgrubb wrote: post #103

sudhirj wrote: post #31

This is one of the biggest arguments I see for serverless (AWS Lambda + DynamoDB) or at least managed PaaS systems (Google App Engine, Heroku with RDS or CloudSQL). These systems may seem to cost more for some workload curves (or might even be cheaper for your curve), but the difference is worth it because you're paying for specialized 24/7 dev-ops teams whose only job is to keep these systems running smoothly, and b…

I work for a PaaS. - The ideal fit for any hosting PaaS is a company who has a large hosting and infra footprint but for whom the technology is _not_ the core competency of the business. Slack is very much better off running their own systems with their own people. - As someone who deals with customers every day I can tell you that yes - we know our platform specifically and how the internet works generally better th…

Post by dang » Tue, Jul 07, 2020, 6:48 PM UTC

ulisesrmzroche wrote: post #164

Why was the “very” removed? It sounds so much better like that.

Post by cardine » Tue, Jul 07, 2020, 7:00 PM UTC

yjftsjthsd-h wrote: post #133

ryan_lane wrote: post #116

Earlier quoted context omitted.

Just because you're using managed services doesn't mean it's going to be considerably more expensive, when you consider labor costs, and general reliability. Your service will likely be more reliable if you use DynamoDB or AuroraDB. Your service will probably be more reliable if you build it in a way that assumes nodes will die at any point, will automatically come back in, and can scale up/down. It'll likely be more…

(Prices based on https://calculator.aws/#/createCalculator and https://www.hetzner.com/dedicated-rootserver?country=us ) An AuroraDB db.r5.xlarge with 10TB of storage, reserved instances 1Y term but no up-front, costs 1,301.40 USD per month. Take a Hetzner AX161 with 4x3.84 TB SATA SSD, using RAIDZ for 11.52TB usable storage (and 4 times the RAM), at €297.00 per month... so 335.88 USD per month. That's a difference o…

Post by cutemonster » Tue, Jul 07, 2020, 7:13 PM UTC

peterwwillis wrote: post #131

hyperman1 wrote: post #4

I've just been bitten by this too: The broken monitoring hadn’t been noticed partly because this system ‘just worked’ for a long time, and didn’t require any change. Any experience on how to deal with it? Who watches the watchers?

Google goes over this scenario in their SRE books. You literally plan to take down your service so you never have 100% uptime, and literally step through a real failure and recovery. In that process you would go, "Ok, the service is failed and I don't know why. How does the monitoring look?", and then you'd notice it was broken. This is best performed by the person on the team with the least experience, so they ask t…

Post by icedchai » Tue, Jul 07, 2020, 7:17 PM UTC

marcosdumay wrote: post #126

tinco wrote: post #40

Earlier quoted context omitted.

Would you consider it evil to use production backups for the QA step in CI/CD? It would catch two birds with one stone, continuously verifying the backups, and ensuring the new code works on real world data. We don't have any personal information in our production database, but even if we did, as long as the QA is thoroughly prevented from interacting with the outside world, it can't hurt to use production data right…

If you have a procedure for deriving QA data from production, change it to derive from the backup. Using it directly is full of dangers, not only of leaking information, but also of corrupting your backups. (Otherwise, why are you testing anything?) And deriving test data from production looks like a good thing to me, but make sure to restrict the access to the test environment and mask your data.