Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A terrible, horrible, no-good, very bad day at Slack

slack.engineering

Post reply on HN
Print view View original

111–120 of 282 posts

Re: A terrible, horrible, no-good, very bad day at Slack

#111

Post by harpratap »

jrockway wrote: post #88

I had trouble getting through this article because my internal monologue was screaming "Envoy and xDS wouldn't have this problem". But that's exactly what they decided ;) HAProxy is a little behind the state of the art on "hey I could just ask some server where the backends are", and it shows in this case. (The "slots" are particularly alarming, as is having to restart when backends come and go.) xDS lets you give yo…

Isn't Ambassador doing the same thing?

Btw not sure if you read till the end, they are actually in the process of migrating to Envoy.

Re: A terrible, horrible, no-good, very bad day at Slack

#113

Post by miloignis »

k__ wrote: post #97
taormina wrote: post #95

Earlier quoted context omitted.

Well, Hipchat would be the literal Atlassian of chat tools.

Ah yes, they have their own thing now :D

They've actually been shutting it down for a while after Slack bought it from them to do just that.

Re: A terrible, horrible, no-good, very bad day at Slack

#114

Post by sudhirj »

Jgrubb wrote: post #103
sudhirj wrote: post #31

This is one of the biggest arguments I see for serverless (AWS Lambda + DynamoDB) or at least managed PaaS systems (Google App Engine, Heroku with RDS or CloudSQL). These systems may seem to cost more for some workload curves (or might even be cheaper for your curve), but the difference is worth it because you're paying for specialized 24/7 dev-ops teams whose only job is to keep these systems running smoothly, and b…

I work for a PaaS. - The ideal fit for any hosting PaaS is a company who has a large hosting and infra footprint but for whom the technology is _not_ the core competency of the business. Slack is very much better off running their own systems with their own people. - As someone who deals with customers every day I can tell you that yes - we know our platform specifically and how the internet works generally better th…

So this is where 12factor comes in - unless the application is operating at lower than layer 7 (and with some of the newer offerings even layer 3) there’s not much technology centric stuff going on. Everything Slack is doing is happening at the HTTP / gRPC / Websocket level, and it’s hard to make the case for self managed hosting.

The idea of requests going to an application server on a TCP or HTTP connection with the application server being able to access a database or datastore is now common enough for all the PaaS providers to have abstractions for it, with the load balancing and database/data stores being managed. If customers aren’t happy with the auto scaling logic that can be overridden, but it seems like everything else is pretty rock solid.

Slack’s USP isn’t to reinvent load balancing, distributed configuration or database management, so if I was running it and had a clean slate a PaaS seems like a better bet.

Re: A terrible, horrible, no-good, very bad day at Slack

#115

Post by aboringusername »

throwaway0a5e wrote: post #37
rkagerer wrote: post #25

Earlier quoted context omitted.

There's a reason the military does drills. Do them regularly, and keep your playbook of mock failure scenarios up to date with good coverage of all your systems. It's especially critical for disaster recovery (a DR plan that's never tested isn't worth the paper it's written on). Consider going one step further and randomly injecting artificial failures into production shards, so handling them becomes a regular affair…

>Do them regularly, and keep your playbook of mock failure scenarios up to date with good coverage of all your systems. All those man hours cost money. Is it worth it? Depends on how much downtime costs you.

Depends on how you value the potential outcomes. A while ago, I think it was Gitlab that didn't check their backups and they were a dud.

That caused quite a few problems...How much is it worth to them after the fact? I'd say a fair bit.

But hindsight...

Re: A terrible, horrible, no-good, very bad day at Slack

#116

Post by ryan_lane »

yjftsjthsd-h wrote: post #108
ryan_lane wrote: post #84

Earlier quoted context omitted.

Lock-in is one of the worst reasons to avoid serverless. If you're really putting in the effort to avoid lock-in, you're wasting engineering time that could be better spent on your product. AWS (or GCP, or whatever cloud provider you choose) has a number of really great products that ultimately save you time and help you get to market faster and better. It doesn't matter that you're locked-in, with slightly higher th…

I'd disagree pretty strongly. You may evaluate the tradeoffs and decide to go all-in on a single cloud (and you should do that analysis!), but it's far from a given. You're not wasting engineering time if the costs get out of hand and bankrupt you. Managed services may accelerate your time to market, but they do so at the expense of lock-in and your bill. Context: I work at a startup that benefits enormously by avoid…

Just because you're using managed services doesn't mean it's going to be considerably more expensive, when you consider labor costs, and general reliability.

Your service will likely be more reliable if you use DynamoDB or AuroraDB. Your service will probably be more reliable if you build it in a way that assumes nodes will die at any point, will automatically come back in, and can scale up/down. It'll likely be more reliable if you use SQS rather than your own message bus (and let's be honest, it'll probably be cheaper too).

Yes, you should always evaluate the costs, but reliability and the amount of time you're going to spend maintaining something is something that somehow always gets left out of these evaluations.

Re: A terrible, horrible, no-good, very bad day at Slack

#117

Post by whatsmyusername »

k__ wrote: post #85

After I used Discord in different contexts for months now (and Slack for years), I can't understand why someone willingly chooses Slack. It's the Atlassian of chat tools. Horrible performance and bad usabillity.

Because it's the Atlassian of chat tools. It's the lowest common denominator.

Discord also doesn't let you set message retention that I'm aware of which is an immediate nonstarter.

End up in one lawsuit where the other party demands a fishing expedition and you'll be real happy that you've got retention limited to 90 days by policy and in practice.

Re: A terrible, horrible, no-good, very bad day at Slack

#118

Post by FuckButtons »

ryan_lane wrote: post #84
gchamonlive wrote: post #33

Earlier quoted context omitted.

You can't assume that, these costs, specifically the server less stack at AWS, scale up quite opaquely. You can put billing alerts in place, but once you tie your infrastructure to one specific serverless vendor, even if you identify a harmful cost scale, you can't easily mitigate it. I am not saying serverless is expensive, all I am advocating is extensive planning and preparing before adopting any particular server…

Lock-in is one of the worst reasons to avoid serverless. If you're really putting in the effort to avoid lock-in, you're wasting engineering time that could be better spent on your product. AWS (or GCP, or whatever cloud provider you choose) has a number of really great products that ultimately save you time and help you get to market faster and better. It doesn't matter that you're locked-in, with slightly higher th…

Getting to market faster is fine for an early startup, less so for a public company which needs to justify costs to its shareholders.

Re: A terrible, horrible, no-good, very bad day at Slack

#119

Post by jeffbee »

jrockway wrote: post #82
vecio wrote: post #5

What's the differences of using HAProxy or Envoy between using the cloud load balancers of AWS or Google Cloud?

I've found that the cloud load balancers lag behind the state of the art in features and that their assumptions and configurations can be pretty brittle. I haven't used Amazon's ALB, but with the legacy ELB, they can't speak ALPN. So that means, if you use their load balancer to terminate TLS, you can't use HTTP/2. Their automatic certificate renewal silently broke for us as well; whereas using cert-manager to renew…

AWS weak point is over-reliance on DNS. Someone needs to evangelize the superiority of non-ttl-based updates, such as K8s endpoint watches.

Re: A terrible, horrible, no-good, very bad day at Slack

#120

Post by snazz »

k__ wrote: post #101
xtracto wrote: post #99

Earlier quoted context omitted.

Threaded comments. I tried using Discord with a group of 4 people for real job and we missed threaded comments. Also the ability to draw on screen while screen sharing. So simple yet so useful.

Interesting. I personally didn't like threads and this is the first time someone mentions them favorably.

Now that even iMessage supports threaded discussions (in iOS 14), I think it would be great if Discord added this too. They’re very nice when there are multiple overlapping conversations and you don’t want to break off into two separate channels.
Post reply on HN