Live data from Hacker News

Lichess: Post-Mortem of Our Longest Downtime

lichess.org

31–40 of 55 posts

Re: Lichess: Post-Mortem of Our Longest Downtime

#31

Earlier quoted context omitted.

The post-mortem is honest, but the infrastructure is well below what I'd expect from commercial services. If a commercial provider told me they're dependent on a single physical server, with no real path or plans to fail over to another server if they need to, I would consider it extremely negligent. It's fine to not use big cloud providers, but frankly it's pretty incompetent to not have the ability to quickly deplo…

And yet even Meta recently had a multiple hours downtime, despite a budget thousands if not million times higher. Would you call them negligent too? By increasing the complexity you multiply the failure points and increase ongoing maintenance, which is the bottleneck (even more than money) for volunteer-driven projects.

To be clear, you don't need to make it more complex / failure-prone. I didn't say failover needs to be automated.

Kubernetes or complex cloud services are not required to have some basic deployment automation.

You can do it with a simple bash script if you need to. It's just pretty surprising to see the reaction to a hardware failure being to wait around for it to be repaired instead of simply spinning up a new host.

Re: Lichess: Post-Mortem of Our Longest Downtime

#32

Earlier quoted context omitted.

I worked with him once on a job -- incredibly nice guy and obviously talented developer who used to work for the French agency responsible for the Scala Play Framework. https://github.com/lichess-org/lila and https://github.com/lichess-org/scalachess are great resources for anyone ever curious to see a production quality Scala3 web application using Cats and all the properly functional properties of the language.

Would you recommend it as a deep-dive to observe Scala in production?

I haven't looked at the code in ages, but it's probably the only scaled consumer web application written in Scala and moreover running on Scala 3 that you can see the end-to-end source for. You have all the Twitter open source Scala projects, of course, but that's just infrastructure for running a web application, rather than an actual production quality app -- and my sense is that in 2024 there aren't many product teams outside of Twitter using their application tooling (as opposed to some of their data infrastructure, certainly the area where Scala sees the most use today with Spark etc).

TLDR if you want to see production-quality Scala code that this very second is serving 40k chess games -- and mostly bullet/blitz where ms latency is of course crucial -- definitely take a look.

Not as much hype for the language at the moment over Rust or Kotlin, say, but it remains my language of choice for web backends by far.

Re: Lichess: Post-Mortem of Our Longest Downtime

#33
post #25

Earlier quoted context omitted.

its also crazy how much cheaper it is than AWS. the primary DB is around $500/month with 32 CPU and 256 GB of RAM and 7TB. AWS RDS db.m6gd.8xlarge which is 32 CPU and 128 GB of RAM costs $2150/month before paying for storage as well.

Yeah, but you get what you pay for. That m6gd.8xlarge would never be subject to such a long network outage as once the hardware fault was detected, it would be moved to another machine

Yup, and you also get to make AWS deal with OS upgrades, DB upgrades, backups, etc.

Re: Lichess: Post-Mortem of Our Longest Downtime

#34
post #28

Earlier quoted context omitted.

The post-mortem is honest, but the infrastructure is well below what I'd expect from commercial services. If a commercial provider told me they're dependent on a single physical server, with no real path or plans to fail over to another server if they need to, I would consider it extremely negligent. It's fine to not use big cloud providers, but frankly it's pretty incompetent to not have the ability to quickly deplo…

We're an understaffed charity.

As a general thought, any idea if people have looked at something like (for example) using Proxmox on the physical hardware so the services can be put on VMs which can be migrated between hosts if there are problems?

Re: Lichess: Post-Mortem of Our Longest Downtime

#35
post #25

Earlier quoted context omitted.

Yeah, but you get what you pay for. That m6gd.8xlarge would never be subject to such a long network outage as once the hardware fault was detected, it would be moved to another machine

Yup, and you also get to make AWS deal with OS upgrades, DB upgrades, backups, etc.

You have to pay 2x for multi-AZ or you get downtime for upgrades. And DB major version upgrades require manual effort unless you want to roll the dice on their new blue-green feature, which can take hours to fail or finish cutting over.

Re: Lichess: Post-Mortem of Our Longest Downtime

#36

I guess some of my questions are addressed in the latter half of the post, but I'm still puzzled why a prominent service didn't have a plan for what looked like a run of the mill hardware outage. It's hard to know exactly what happened as I'm having trouble parsing some of the post (what is a 'network connector'? is it a cable? nic?). What were some of the 'increasingly outlandish' workarounds? Are they actually stan…

You are not wrong that this is puzzling, especially when viewed through the perspective lens of a professional with background in these areas (10 years).

There are many red flags which beg questions.

That said, I stopped taking them at their word years ago, this isn't the first time they've had dubious announcements following entirely preventable failures. In my mind, they really don't have any professional credibility.

People in the business of System Administration would follow basic standard practices that eliminate most of these risks.

The linked post isn't a valid post-mortem, if it were it would contain unambiguous details of the timetables and specifics, both of the failure domains and resolutions.

As you say, a network connector could mean any number of things. Its ambiguous, and ambiguity in technical material is used to hide or mislead most times which is why professionals detailing a post mortem would remove any possible ambiguity they could.

It is common professional practice to have a recovery playbook, and a plan for disaster recovery for business continuity which is tested at least every 6 months, usually quarterly. This is true of both charities and business.

Based on their post, they don't have one and they don't follow this well known industry practice. You really cannot call yourself a System Administrator if you don't follow the basics of the profession.

TPOSNA covers these basics for those not in the profession, its roughly two decades old now, it is well established, and ignorance of the practices isn't a valid excuse.

Professional budgets also always have a fund for emergencies based on these BC/DR plans. Additionally, using resilient design is common practice; single points of failures are not excusable in production failure domains especially when zero-downtime must be achieved.

Automated Deployment is a standard practice as well factoring into RTO and capacity planning improvements. Cattle not Pets.

Also, you don't ever wait on a vendor to take action. You make changes, and revert when the issue gets resolved.

First thing I would have done is set the domain DNS TTL to 5 minutes upon alerted failures (as a precaution), and then if needed point the DNS to a viable alternative server (either deployed temporarily or running in parallel).

Failures inevitably happen which is why you risk manage this using a topology with load balancers/servers set up in HA groups, eliminating any single provider as a single point of failure.

This is so basic that any junior admin knows these things.

Outlandish workarounds only happen when you do not have a plan and you are dredging the bottom of the barrel.

Re: Lichess: Post-Mortem of Our Longest Downtime

#37
post #10

This response and post-mortem is superior to most commercial services I have seen in recent years.

The post-mortem is honest, but the infrastructure is well below what I'd expect from commercial services. If a commercial provider told me they're dependent on a single physical server, with no real path or plans to fail over to another server if they need to, I would consider it extremely negligent. It's fine to not use big cloud providers, but frankly it's pretty incompetent to not have the ability to quickly deplo…

Poe's law. Lichess is 14 years old and their longest outage is less than 12 hours. Google and AWS have both had ~6 hour outages and that's with billions od dollars depending on them and thousands of engineers. Simpler is working just fine.

Re: Lichess: Post-Mortem of Our Longest Downtime

#38
post #36

I guess some of my questions are addressed in the latter half of the post, but I'm still puzzled why a prominent service didn't have a plan for what looked like a run of the mill hardware outage. It's hard to know exactly what happened as I'm having trouble parsing some of the post (what is a 'network connector'? is it a cable? nic?). What were some of the 'increasingly outlandish' workarounds? Are they actually stan…

You are not wrong that this is puzzling, especially when viewed through the perspective lens of a professional with background in these areas (10 years). There are many red flags which beg questions. That said, I stopped taking them at their word years ago, this isn't the first time they've had dubious announcements following entirely preventable failures. In my mind, they really don't have any professional credibili…

Why put so much effort when at worst you have a few hours of downtime

Re: Lichess: Post-Mortem of Our Longest Downtime

#39
post #36

I guess some of my questions are addressed in the latter half of the post, but I'm still puzzled why a prominent service didn't have a plan for what looked like a run of the mill hardware outage. It's hard to know exactly what happened as I'm having trouble parsing some of the post (what is a 'network connector'? is it a cable? nic?). What were some of the 'increasingly outlandish' workarounds? Are they actually stan…

You are not wrong that this is puzzling, especially when viewed through the perspective lens of a professional with background in these areas (10 years). There are many red flags which beg questions. That said, I stopped taking them at their word years ago, this isn't the first time they've had dubious announcements following entirely preventable failures. In my mind, they really don't have any professional credibili…

I've worked with Thibault before he could self-sustain on lichess donations, he's a professional software developer and sysadmin and one of the best I've worked with.

The people behind lichess are very much professionals, have worked in companies before, and know about everything you're writing. However instead of building a business they decided to run a completely free and ad-free non profit living off donations.

You don't get the same budget doing that compared than a subscription base / ad supported service. That's true for the number of people maintaining it as well as the cloud cost you can afford.

If you look at their track record, uptime have been pretty good. Shit happens, but if you ask me it's worth it to have a service like Lichess that can exist completely on donations.

Re: Lichess: Post-Mortem of Our Longest Downtime

#40
post #36

I guess some of my questions are addressed in the latter half of the post, but I'm still puzzled why a prominent service didn't have a plan for what looked like a run of the mill hardware outage. It's hard to know exactly what happened as I'm having trouble parsing some of the post (what is a 'network connector'? is it a cable? nic?). What were some of the 'increasingly outlandish' workarounds? Are they actually stan…

You are not wrong that this is puzzling, especially when viewed through the perspective lens of a professional with background in these areas (10 years). There are many red flags which beg questions. That said, I stopped taking them at their word years ago, this isn't the first time they've had dubious announcements following entirely preventable failures. In my mind, they really don't have any professional credibili…

This isn't a billion dollar company trading on the NYSE. Its a free website to play chess.
Post reply on HN