121–130 of 282 posts
After I used Discord in different contexts for months now (and Slack for years), I can't understand why someone willingly chooses Slack. It's the Atlassian of chat tools. Horrible performance and bad usabillity.
Because Discord is very gaming oriented. But yes, Discord is great.
I was surprised how they dismissed HAProxy integration with Consul using SRV DNS records. Can anyone confirm the problems they highlight?
It seems like their service that broke would not be needed if they went the DNS route.
Super interesting post. Following blog links, the timeline in https://slack.engineering/all-hands-on-deck-91d6986c3ee also offers a look at the play by play. However, as far as I can read it, they have somewhat different views on the root cause? "Soon, it became clear we had stale HAProxy configuration files, as a result of linting errors preventing re-rendering of the configuration." vs. "The program which synced th…
Honestly it’s a bit tough for me to parse, but the way I’m reading it, 1. Stale configs led to an overabundance of web apps, and then 2. Old instances of the web app couldn’t be removed because of the consul-template bug. so, yes, a combination (in sequence) of the two. Hard for me to be sure because I’m by no means knowledgeable on this stuff.
- slots full
- to update slots with a new host you need an empty slot
- hosts went away but updating config was impossible -> errors because config referenced non-existing hosts
Earlier quoted context omitted.
You can't assume that, these costs, specifically the server less stack at AWS, scale up quite opaquely. You can put billing alerts in place, but once you tie your infrastructure to one specific serverless vendor, even if you identify a harmful cost scale, you can't easily mitigate it. I am not saying serverless is expensive, all I am advocating is extensive planning and preparing before adopting any particular server…
Lock-in is one of the worst reasons to avoid serverless. If you're really putting in the effort to avoid lock-in, you're wasting engineering time that could be better spent on your product. AWS (or GCP, or whatever cloud provider you choose) has a number of really great products that ultimately save you time and help you get to market faster and better. It doesn't matter that you're locked-in, with slightly higher th…
Earlier quoted context omitted.
I work for a PaaS. - The ideal fit for any hosting PaaS is a company who has a large hosting and infra footprint but for whom the technology is _not_ the core competency of the business. Slack is very much better off running their own systems with their own people. - As someone who deals with customers every day I can tell you that yes - we know our platform specifically and how the internet works generally better th…
So this is where 12factor comes in - unless the application is operating at lower than layer 7 (and with some of the newer offerings even layer 3) there’s not much technology centric stuff going on. Everything Slack is doing is happening at the HTTP / gRPC / Websocket level, and it’s hard to make the case for self managed hosting. The idea of requests going to an application server on a TCP or HTTP connection with th…
Earlier quoted context omitted.
This same sort of thing tends to creep into backup systems that work for a long time without if the backups are never restored. The more reliable the systems they backup the bigger the chance that a restore will fail when you need it most. So test your backups .
Would you consider it evil to use production backups for the QA step in CI/CD? It would catch two birds with one stone, continuously verifying the backups, and ensuring the new code works on real world data. We don't have any personal information in our production database, but even if we did, as long as the QA is thoroughly prevented from interacting with the outside world, it can't hurt to use production data right…
Using it directly is full of dangers, not only of leaking information, but also of corrupting your backups. (Otherwise, why are you testing anything?) And deriving test data from production looks like a good thing to me, but make sure to restrict the access to the test environment and mask your data.
Earlier quoted context omitted.
As someone who uses Discord, I can tell you it goes down too. My favorite part is looking at their status page, seeing that API response time is exceedingly high, and getting no updates from the team about whether or not they're fixing it.
I'm not saying Discord is perfect or "always up", it's just that their client UI is better structured and more responsive in most cases. I am on multiple OSS Discord servers with thousands of users, and it works just fine most of the time. I am on multiple Slack servers with just 10-20 users and it is unbearably slow.
Slack has a more mainstream look and feel which is probably one of the reasons it's preferred by companies.
> It’s worth noting that HAProxy can integrate with Consul’s DNS interface, but this adds lag due to the DNS TTL, it limits the ability to use Consul tags, and managing very large DNS responses often seems to lead to hitting painful edge-cases and bugs. I was surprised how they dismissed HAProxy integration with Consul using SRV DNS records. Can anyone confirm the problems they highlight? It seems like their service…
Pre k8s and before srv support we used consul template in prod as well but it always scared me, seemed like too many moving pieces for what should've been a simple system.
Great writeup. It's cool that they were able to figure it out as quickly as they did, all things considered. If I were brought in as a consultant on this, my first question would be: why are you using a fleet of HAProxies instead of the ALB? I'm not saying that's a bad choice, but I'd want to know why that choice was made. The second question I would ask is what kind of Chaos Engineering they are doing. Are they doin…
That said though, it does do it's job for the most part.
Earlier quoted context omitted.
But how, exactly? Sure, the chaos monkey could kill haproxy-server-state-management but that wouldn't uncover the bug in question — it'd just demonstrate that without it running HA proxy's view of the world goes stale, which anyone would expect. Triggering the bug would require reducing the number of HA proxy slots below the number of webapps running for many hours. This is clearly something chaos engineering could d…
> Sure, the chaos monkey could kill haproxy-server-state-management but that wouldn't uncover the bug in question No it won't. But it would uncover their missing alerts for a critical platform component. Their issue was exacerbated by the fact that state-management kept failing for nearly 12 hours and no one noticed.
I'm not totally against chaos testing. I just haven't seen it done well and think it's actually pretty hard to pull off (particularly the non-technical aspect of convincing people it's okay to let this thing go mad). I'd love to see how effective it was within Netflix.