Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lichess: Post-Mortem of Our Longest Downtime

lichess.org

Post reply on HN
Print view View original

21–30 of 55 posts

Re: Lichess: Post-Mortem of Our Longest Downtime

#21

Post by agentcoops »

hyperbovine wrote: post #13
holsta wrote: post #10

This response and post-mortem is superior to most commercial services I have seen in recent years.

That's basically every aspect of their service. The founder Thibault Duplessis is criminally undercompensated (his choice) for running a site that is better designed, faster, and more popular than 99% of commercial websites out there.

I worked with him once on a job -- incredibly nice guy and obviously talented developer who used to work for the French agency responsible for the Scala Play Framework. https://github.com/lichess-org/lila and https://github.com/lichess-org/scalachess are great resources for anyone ever curious to see a production quality Scala3 web application using Cats and all the properly functional properties of the language.

Re: Lichess: Post-Mortem of Our Longest Downtime

#22

Post by hilux »

carlsborg wrote: post #2

The main lichess engine (lila, open source) is a single monolith program that's deployed on a single server. It serves ~5 million games per day. But there are a several other pieces too. They discuss the architecture here https://www.youtube.com/watch?v=crKNBSpO2_I BTW consider donating if you use lichess.

I'm a patron!

I really appreciate the benefits package for patrons. Thibault is zee best.

Re: Lichess: Post-Mortem of Our Longest Downtime

#23

Post by theideaofcoffee »

I guess some of my questions are addressed in the latter half of the post, but I'm still puzzled why a prominent service didn't have a plan for what looked like a run of the mill hardware outage. It's hard to know exactly what happened as I'm having trouble parsing some of the post (what is a 'network connector'? is it a cable? nic?). What were some of the 'increasingly outlandish' workarounds? Are they actually standing up production hosts manually, and was that the cause of a delay or unwillingness to get new hardware goin? I think it would be important to have all of that set down either in documentation or code seeing as most of their technical staff are either volunteers, who may come and go, or part timers. Maybe they did, it's not clear.

It's also weird seeing that they are still waiting on their provider to tell them exactly what was done to the hardware to get it going again, that's usually one of the first things a tech mentions: "ok, we replaced the optics in port 1" or "I replaced that cable after seeing increased error rates", something like that.

Re: Lichess: Post-Mortem of Our Longest Downtime

#24

Post by morgante »

holsta wrote: post #10

This response and post-mortem is superior to most commercial services I have seen in recent years.

The post-mortem is honest, but the infrastructure is well below what I'd expect from commercial services.

If a commercial provider told me they're dependent on a single physical server, with no real path or plans to fail over to another server if they need to, I would consider it extremely negligent.

It's fine to not use big cloud providers, but frankly it's pretty incompetent to not have the ability to quickly deploy to a new server.

Re: Lichess: Post-Mortem of Our Longest Downtime

#25

Post by bryan_w »

benmmurphy wrote: post #19
justinclift wrote: post #3

Earlier quoted context omitted.

Wow. ~US$40k/mo running costs, with about US$5k/mo for server hosting: https://lichess.org/costs It looks like the servers are individually managed via OVH or similar, rather than running their own gear in co-location or similar. Wonder why?

its also crazy how much cheaper it is than AWS. the primary DB is around $500/month with 32 CPU and 256 GB of RAM and 7TB. AWS RDS db.m6gd.8xlarge which is 32 CPU and 128 GB of RAM costs $2150/month before paying for storage as well.

Yeah, but you get what you pay for. That m6gd.8xlarge would never be subject to such a long network outage as once the hardware fault was detected, it would be moved to another machine

Re: Lichess: Post-Mortem of Our Longest Downtime

#26

Post by notagoodidea »

agentcoops wrote: post #21
hyperbovine wrote: post #13

Earlier quoted context omitted.

That's basically every aspect of their service. The founder Thibault Duplessis is criminally undercompensated (his choice) for running a site that is better designed, faster, and more popular than 99% of commercial websites out there.

I worked with him once on a job -- incredibly nice guy and obviously talented developer who used to work for the French agency responsible for the Scala Play Framework. https://github.com/lichess-org/lila and https://github.com/lichess-org/scalachess are great resources for anyone ever curious to see a production quality Scala3 web application using Cats and all the properly functional properties of the language.

Would you recommend it as a deep-dive to observe Scala in production?

Re: Lichess: Post-Mortem of Our Longest Downtime

#28

Post by lukhas »

morgante wrote: post #24
holsta wrote: post #10

This response and post-mortem is superior to most commercial services I have seen in recent years.

The post-mortem is honest, but the infrastructure is well below what I'd expect from commercial services. If a commercial provider told me they're dependent on a single physical server, with no real path or plans to fail over to another server if they need to, I would consider it extremely negligent. It's fine to not use big cloud providers, but frankly it's pretty incompetent to not have the ability to quickly deplo…

We're an understaffed charity.

Re: Lichess: Post-Mortem of Our Longest Downtime

#29

Post by KolmogorovComp »

morgante wrote: post #24
holsta wrote: post #10

This response and post-mortem is superior to most commercial services I have seen in recent years.

The post-mortem is honest, but the infrastructure is well below what I'd expect from commercial services. If a commercial provider told me they're dependent on a single physical server, with no real path or plans to fail over to another server if they need to, I would consider it extremely negligent. It's fine to not use big cloud providers, but frankly it's pretty incompetent to not have the ability to quickly deplo…

And yet even Meta recently had a multiple hours downtime, despite a budget thousands if not million times higher. Would you call them negligent too?

By increasing the complexity you multiply the failure points and increase ongoing maintenance, which is the bottleneck (even more than money) for volunteer-driven projects.

Re: Lichess: Post-Mortem of Our Longest Downtime

#30

Post by morgante »

lukhas wrote: post #28
morgante wrote: post #24

Earlier quoted context omitted.

The post-mortem is honest, but the infrastructure is well below what I'd expect from commercial services. If a commercial provider told me they're dependent on a single physical server, with no real path or plans to fail over to another server if they need to, I would consider it extremely negligent. It's fine to not use big cloud providers, but frankly it's pretty incompetent to not have the ability to quickly deplo…

We're an understaffed charity.

Yeah I'm not criticizing it as a charity, just pointing out this definitely isn't "superior to most commercial services."

That being said, removing dependence on single hardware nodes isn't something you need a big team for. I've done failover at 1-person startups.

Post reply on HN