Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Discord Postmortem from Friday

status.discordapp.com

Post reply on HN
Print view View original

1–10 of 45 posts

Re: Discord Postmortem from Friday

#3

Post by katastic »

atomical wrote: post #2

Does anyone use discord for work?

I have informally with co-workers. But not in any official capacity.

It's like 20x better than every other product out there though. And their new video chat + screen sharing is pretty great. The bandwidth is far higher than any other competitors I've used.

My brother and I were playing 1080p videos on each of our screens and watching the other's, just to test it out. Obviously it wasn't full quality, but it kept the frame rate up and looked presentable at least to 720p.

Re: Discord Postmortem from Friday

#4

Post by avree »

katastic wrote: post #3
atomical wrote: post #2

Does anyone use discord for work?

I have informally with co-workers. But not in any official capacity. It's like 20x better than every other product out there though. And their new video chat + screen sharing is pretty great. The bandwidth is far higher than any other competitors I've used. My brother and I were playing 1080p videos on each of our screens and watching the other's, just to test it out. Obviously it wasn't full quality, but it kept the…

Weird, I find Discord's audio quality especially to be terrible.

And their lack of scalable monetization leads me worried about its longterm success as a platform - they are adding more cost-intensive features and continuing to try to support it with what is essentially a $5 monthly donation model.

Re: Discord Postmortem from Friday

#7

Post by b1naryth1ef »

avree wrote: post #4
katastic wrote: post #3

Earlier quoted context omitted.

I have informally with co-workers. But not in any official capacity. It's like 20x better than every other product out there though. And their new video chat + screen sharing is pretty great. The bandwidth is far higher than any other competitors I've used. My brother and I were playing 1080p videos on each of our screens and watching the other's, just to test it out. Obviously it wasn't full quality, but it kept the…

Weird, I find Discord's audio quality especially to be terrible. And their lack of scalable monetization leads me worried about its longterm success as a platform - they are adding more cost-intensive features and continuing to try to support it with what is essentially a $5 monthly donation model.

Can you give more explicit examples of the bad audio quality you experience? I'd be happy to forward this onto our native team to look into if there are some concrete things they can look at. Generally 99% of the audio issues we see people experience are due to ISP/peering/DDoS/etc issues, most of which are handled automatically by our servers within a few minutes.

Re: Discord Postmortem from Friday

#8

Post by jhgg »

It's worth noting that the instance migration basically null-routed the redis VM for a good 30 minutes, until we manually intervened and restarted it. The instance was completely disconnected from the internal network immediately following the migration. From what we could gather from instance logs, the routing table on the VM was completely dropped and it could not even connect to the magic metadata service (metadata.internal - we saw "no route to host" errors for that). This is a pretty serious bug within GCP and we've already opened a case with them hoping they can get a fix. I think this is the 4th or 5th major bug we've encountered with their live migration system that could have, or has led to an outage or internal service degradation. GCP team has seriously investigated and fixed every bug we've reported to them so far, so props to them for that! Live migration is incredibly difficult to get right.

We believe this triggered a bug in the redis-py python driver we use (specifically this one: https://github.com/andymccurdy/redis-py/pull/886) that made us have to rolling restart our API cluster in the first place, to get the connection pools back into a working state. redis-sentinel had appropriately detected the instance going away, and initiated a fail-over almost immediately following the instance going offline, but due to the odd network situation that was caused by the migration (absolute packet loss instead of connections being reset) - the client driver was unable to properly fail-over to the new master. We already have work planned for our own connection pooling logic for redis-py - as right now the state of the drive in HA redis is actually pretty awful, and the maintainer doesn't appear to have the time to close or look at PRs that address these issues (we opened one that fixes a pretty serious bug during fail-over in march https://github.com/andymccurdy/redis-py/pull/847 that has yet to be addressed).

Re: Discord Postmortem from Friday

#9

Post by gizmo385 »

avree wrote: post #4
katastic wrote: post #3

Earlier quoted context omitted.

I have informally with co-workers. But not in any official capacity. It's like 20x better than every other product out there though. And their new video chat + screen sharing is pretty great. The bandwidth is far higher than any other competitors I've used. My brother and I were playing 1080p videos on each of our screens and watching the other's, just to test it out. Obviously it wasn't full quality, but it kept the…

Weird, I find Discord's audio quality especially to be terrible. And their lack of scalable monetization leads me worried about its longterm success as a platform - they are adding more cost-intensive features and continuing to try to support it with what is essentially a $5 monthly donation model.

One thing that I've noticed helps is swapping the region for the server. It doesn't result in any noticeable outage on your server and has resolved any voice quality issues that we've run into most of the time. Usually just swapping between US-Central and US-West in our case.

Re: Discord Postmortem from Friday

#10

Post by gizmo385 »

ZeroCool2u wrote: post #6

I'm really impressed. I was using Discord for most of this weekend, specifically Friday and Saturday. Never noticed any issues.

They don't have a posted outage for Saturday, but I noticed issues with it on Saturday evening/night I believe. I'm wondering if it was related to the issues that they included in the post-mortem.
Post reply on HN