Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A terrible, horrible, no-good, very bad day at Slack

slack.engineering

Post reply on HN
Print view View original

201–210 of 282 posts

Re: A terrible, horrible, no-good, very bad day at Slack

#201

Post by yjftsjthsd-h »

bscanlan wrote: post #196
yjftsjthsd-h wrote: post #133

Earlier quoted context omitted.

(Prices based on https://calculator.aws/#/createCalculator and https://www.hetzner.com/dedicated-rootserver?country=us ) An AuroraDB db.r5.xlarge with 10TB of storage, reserved instances 1Y term but no up-front, costs 1,301.40 USD per month. Take a Hetzner AX161 with 4x3.84 TB SATA SSD, using RAIDZ for 11.52TB usable storage (and 4 times the RAM), at €297.00 per month... so 335.88 USD per month. That's a difference o…

You are forgetting to price in some minor features that Aurora provides: - Aurora's storage is spread across three availability zones. - Backups. - Automatic failover. - No need to configure anything, it just works. If your time is free, and you don't actually need anything resembling high availability for the data in the database, then that's a good price comparison. I'm not arguing that managed databases makes sens…

> You are forgetting to price in some minor features that Aurora provides

That's true and fair, although in both directions; skimming the docs it looks like aurora prices include 2 replicas? But backups aren't free (to store), bandwidth isn't free, and iops aren't free. Also, my difficulty in figuring out a fair pricing comparison highlights another point: a dedicated server has a fixed price. Other than more servers for more instances/replicas, you're never going to pay more, and even then it's a simple "adding another replica will increase our costs to X*(N+1) per month", not a "scaling out will add X to our costs, but if we use more I/O than expected we'll add Y to our costs, and exporting data will cost Z in bandwidth".

Re: A terrible, horrible, no-good, very bad day at Slack

#202

Post by Jgrubb »

awill wrote: post #137
Jgrubb wrote: post #103

Earlier quoted context omitted.

I work for a PaaS. - The ideal fit for any hosting PaaS is a company who has a large hosting and infra footprint but for whom the technology is _not_ the core competency of the business. Slack is very much better off running their own systems with their own people. - As someone who deals with customers every day I can tell you that yes - we know our platform specifically and how the internet works generally better th…

>>"but we do not know _your application_ at all." You're missing the point. A PaaS, or serverless service doesn't need to know your application. That's the whole point. They're just API calls, and they need to succeed with consistently low latency.

> That's the whole point.

Until something falls over and you haven't deployed any new code and all the vendor's systems are green. Part of the reason you went with a PaaS in the first place is that you didn't want to manage the infra yourself, you just want to ship application code.

This has worked great so far and so you've stopped thinking about the infrastructure at all. Disk and network I/O still exist though, you've just been incentivized to stop thinking about them and so all you know is that the vendor sucks, when in reality it's the application's fault.

Somebody's gotta roll up their sleeves and see what's going on in there though, so hopefully your PaaS vendor is as cool as we are :)

Re: A terrible, horrible, no-good, very bad day at Slack

#203

Post by staticassertion »

vondur wrote: post #194
m12k wrote: post #26

TL;DR First a performance bug was caught during rollout, and rolled back within a few minutes. However this triggered their auto-scaling of web apps to ramp up to more instances than a hard limit they had. This in turn triggered a bug in how they update the list of hosts in their load balancer, causing it to not get updated with new instances, and eventually go stale. After 8 hours the only real remaining instances i…

It almost seems like no one group understands the system as a whole, so when one part fails, no one has a clear idea of the domino effects that can happen. I'm guessing this is the result of really complex systems interactions.

Feels like you've just described every single bug ever.

Re: A terrible, horrible, no-good, very bad day at Slack

#204

Post by SilasX »

Stupid question, and admittedly off topic:

What's with the "terrible, horrible, no-good, very bad" expression I see a lot? It's a reference to something? From googling, it seems to be this [1], but ... why? Why do people reference it?

Usually you reference some work like this because a) the phrase is unusually creative, or b) the work is unusually memorable. Neither is true here.

[1] https://en.wikipedia.org/wiki/Alexander_and_the_Terrible,_Ho...

Re: A terrible, horrible, no-good, very bad day at Slack

#205

Post by nikanj »

> ” The reason that we haven’t been doing any significant work on this HAProxy stack is that we’re moving towards Envoy Proxy ”

I truly don’t understand the cycle of churn. Once most edge cases and bugs of Haproxy have been found, the right decision is not migrating to completely unknown territory again. No project is a silver bullet, and changing stacks after you find the bugs makes for a terrible return on your bug-hunting investments.

Re: A terrible, horrible, no-good, very bad day at Slack

#206

Post by nikanj »

SilasX wrote: post #204

Stupid question, and admittedly off topic: What's with the "terrible, horrible, no-good, very bad" expression I see a lot? It's a reference to something? From googling, it seems to be this [1], but ... why ? Why do people reference it? Usually you reference some work like this because a) the phrase is unusually creative, or b) the work is unusually memorable. Neither is true here. [1] https://en.wikipedia.org/wiki/Al…

This book sold well enough to score a tv series, a disney movie, a musical, and a theater play.

How much more successful do you need it to be? Odds seem very high that a kid growing up in the past 50 years was exposed to this story and phrase

Re: A terrible, horrible, no-good, very bad day at Slack

#207

Post by solumos »

SilasX wrote: post #204

Stupid question, and admittedly off topic: What's with the "terrible, horrible, no-good, very bad" expression I see a lot? It's a reference to something? From googling, it seems to be this [1], but ... why ? Why do people reference it? Usually you reference some work like this because a) the phrase is unusually creative, or b) the work is unusually memorable. Neither is true here. [1] https://en.wikipedia.org/wiki/Al…

It's memorable in that it's a lighthearted way of describing a pretty disastrous event without using expletives, with at least a dash of cultural significance.

I agree it's a lame title.

Re: A terrible, horrible, no-good, very bad day at Slack

#208

Post by bastawhiz »

nikanj wrote: post #205

> ” The reason that we haven’t been doing any significant work on this HAProxy stack is that we’re moving towards Envoy Proxy ” I truly don’t understand the cycle of churn. Once most edge cases and bugs of Haproxy have been found, the right decision is not migrating to completely unknown territory again. No project is a silver bullet, and changing stacks after you find the bugs makes for a terrible return on your bug…

And if there is no path forward to resolve the bugs? How many rough edges and bugs can you tolerate before a system is no longer worth investing in, especially when another newer system does not suffer from any of those bugs? To say there is no tipping point where the cost of continuing to use a system which no longer fits your needs exceeds the cost of migrating to a system which does (and will continue to) fit your needs ignores some amount of common sense.

Re: A terrible, horrible, no-good, very bad day at Slack

#209

Post by brown9-2 »

nikanj wrote: post #205

> ” The reason that we haven’t been doing any significant work on this HAProxy stack is that we’re moving towards Envoy Proxy ” I truly don’t understand the cycle of churn. Once most edge cases and bugs of Haproxy have been found, the right decision is not migrating to completely unknown territory again. No project is a silver bullet, and changing stacks after you find the bugs makes for a terrible return on your bug…

I think if you read this as they’re moving to Envoy because of this incident then you’ve misread.

But it also sounds like Envoy and HAProxy have fundamentally different approaches to service discovery:

https://twitter.com/mattklein123/status/1277729102271676416?...

https://twitter.com/mattklein123/status/1278114953497436161?...

Re: A terrible, horrible, no-good, very bad day at Slack

#210

Post by m463 »

staticassertion wrote: post #203
vondur wrote: post #194

Earlier quoted context omitted.

It almost seems like no one group understands the system as a whole, so when one part fails, no one has a clear idea of the domino effects that can happen. I'm guessing this is the result of really complex systems interactions.

Feels like you've just described every single bug ever.

Even at the start I'm sure Grace Hopper didn't anticipate a computer with a moth in it.

https://upload.wikimedia.org/wikipedia/commons/8/8a/H96566k....

Post reply on HN