Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lichess: Post-Mortem of Our Longest Downtime

lichess.org

Post reply on HN
Print view View original

41–50 of 55 posts

Re: Lichess: Post-Mortem of Our Longest Downtime

#41

Post by 0cf8612b2e1e »

high_na_euv wrote: post #38
trod123 wrote: post #36

Earlier quoted context omitted.

You are not wrong that this is puzzling, especially when viewed through the perspective lens of a professional with background in these areas (10 years). There are many red flags which beg questions. That said, I stopped taking them at their word years ago, this isn't the first time they've had dubious announcements following entirely preventable failures. In my mind, they really don't have any professional credibili…

Why put so much effort when at worst you have a few hours of downtime

As they say, each 9 of uptime increases costs by an order of magnitude. For a non profit service, a few hours of downtime seems a fine trade off vs engineering all of the “right” redundancies. All of which have their own operational costs.

Re: Lichess: Post-Mortem of Our Longest Downtime

#42

Post by DiggyJohnson »

trod123 wrote: post #36
theideaofcoffee wrote: post #23

I guess some of my questions are addressed in the latter half of the post, but I'm still puzzled why a prominent service didn't have a plan for what looked like a run of the mill hardware outage. It's hard to know exactly what happened as I'm having trouble parsing some of the post (what is a 'network connector'? is it a cable? nic?). What were some of the 'increasingly outlandish' workarounds? Are they actually stan…

You are not wrong that this is puzzling, especially when viewed through the perspective lens of a professional with background in these areas (10 years). There are many red flags which beg questions. That said, I stopped taking them at their word years ago, this isn't the first time they've had dubious announcements following entirely preventable failures. In my mind, they really don't have any professional credibili…

This is so far out of line I wonder what the background is for this issue. Lichess is not emergency dispatch software running in a 911 Call Center, if they have an outage the cost is that users can't play online chess until it is fixed. Additionally, the founder of this open source project is objectively good at what he does. Exhibit the fact that he built and hosts a top 2 online chess platform that competes well against the biggest commercial sites. How does that not lend some professional credibility.

Re: Lichess: Post-Mortem of Our Longest Downtime

#43

Post by jefurii »

carlsborg wrote: post #2

The main lichess engine (lila, open source) is a single monolith program that's deployed on a single server. It serves ~5 million games per day. But there are a several other pieces too. They discuss the architecture here https://www.youtube.com/watch?v=crKNBSpO2_I BTW consider donating if you use lichess.

There's also a nice architectural diagram on their GitHub page: https://github.com/lichess-org/lila

Re: Lichess: Post-Mortem of Our Longest Downtime

#44

Post by trod123 »

eloisant wrote: post #39
trod123 wrote: post #36

Earlier quoted context omitted.

You are not wrong that this is puzzling, especially when viewed through the perspective lens of a professional with background in these areas (10 years). There are many red flags which beg questions. That said, I stopped taking them at their word years ago, this isn't the first time they've had dubious announcements following entirely preventable failures. In my mind, they really don't have any professional credibili…

I've worked with Thibault before he could self-sustain on lichess donations, he's a professional software developer and sysadmin and one of the best I've worked with. The people behind lichess are very much professionals, have worked in companies before, and know about everything you're writing. However instead of building a business they decided to run a completely free and ad-free non profit living off donations. Y…

There are many problems with what you've written here as well as bot-like behavior in the responses that have telltale signs of vote manipulation and propaganda similar to Chinese state-run campaigns.

We will have to disagree. You have clearly contradicted yourself in at least one way, and attempt to mislead readers in a number of other ways which I won't go into here.

From these, I have to come to the conclusion that you don't have credibility.

The downtime would not have happened if they had followed professional practices. Even a qualified Administrator coming into the outage fresh would have had a fix within 30 minutes if they were working at a professional level.

Yes shit happens, but professionals have processes in place so that common shit does not just happen. This was preventable.

Re: Lichess: Post-Mortem of Our Longest Downtime

#45

Post by OkayPhysicist »

trod123 wrote: post #44
eloisant wrote: post #39

Earlier quoted context omitted.

I've worked with Thibault before he could self-sustain on lichess donations, he's a professional software developer and sysadmin and one of the best I've worked with. The people behind lichess are very much professionals, have worked in companies before, and know about everything you're writing. However instead of building a business they decided to run a completely free and ad-free non profit living off donations. Y…

There are many problems with what you've written here as well as bot-like behavior in the responses that have telltale signs of vote manipulation and propaganda similar to Chinese state-run campaigns. We will have to disagree. You have clearly contradicted yourself in at least one way, and attempt to mislead readers in a number of other ways which I won't go into here. From these, I have to come to the conclusion tha…

What kind of Tom Clancy novel do you live in that intelligence agencies are astroturfing for free chess sites?

Re: Lichess: Post-Mortem of Our Longest Downtime

#46

Post by chmod775 »

beaviskhan wrote: post #33
bryan_w wrote: post #25

Earlier quoted context omitted.

Yeah, but you get what you pay for. That m6gd.8xlarge would never be subject to such a long network outage as once the hardware fault was detected, it would be moved to another machine

Yup, and you also get to make AWS deal with OS upgrades, DB upgrades, backups, etc.

Given that the full AWS setup that would replace that one server would cost closer to $6000-8000 / month, they could just use that money to buy a bunch of extra hard drives, a backup server, and hire a junior dev/sysadmin whose only job is to watch over it, still coming out ahead of AWS.

Re: Lichess: Post-Mortem of Our Longest Downtime

#47

Post by acdha »

chmod775 wrote: post #46
beaviskhan wrote: post #33

Earlier quoted context omitted.

Yup, and you also get to make AWS deal with OS upgrades, DB upgrades, backups, etc.

Given that the full AWS setup that would replace that one server would cost closer to $6000-8000 / month, they could just use that money to buy a bunch of extra hard drives, a backup server, and hire a junior dev/sysadmin whose only job is to watch over it , still coming out ahead of AWS.

I’m seeing more like $5k/mo with reservations but even at those figures … how many skilled DBAs are you getting for $72-96k? Don’t forget that rolling it yourself means you have to build and test all of the hardware, maintenance processes, backups, multi-data center HA, etc. yourself. That’s not junior trainee level work and some of it is ongoing (e.g. every OS and hardware change) or at intervals not of your choosing (say you discover a kernel or firmware issue when it’s crashing randomly - how many months of savings will be canceled out by pulling senior people off of whatever they’re working on?).

You can beat AWS on pricing but not like this. You need to be finding areas where you have a lot of baseline demand – enough to amortize the cost of all of the lower level work – and can cut some of the things they do which you don’t need. For example, if you can afford more downtime in a disaster scenario or can rely on an external rebuild process if the database backups turn out to be unusable.

Re: Lichess: Post-Mortem of Our Longest Downtime

#48

Post by trod123 »

OkayPhysicist wrote: post #45
trod123 wrote: post #44

Earlier quoted context omitted.

There are many problems with what you've written here as well as bot-like behavior in the responses that have telltale signs of vote manipulation and propaganda similar to Chinese state-run campaigns. We will have to disagree. You have clearly contradicted yourself in at least one way, and attempt to mislead readers in a number of other ways which I won't go into here. From these, I have to come to the conclusion tha…

What kind of Tom Clancy novel do you live in that intelligence agencies are astroturfing for free chess sites?

I'm going to assume that your question is genuine and sincere, and not meant sarcastically.

If you read the following books by established experts, you should be able to rationally answer the question for yourself as to the why and the how. The subject matter involves torture for thought reform, real not fantasy. This differs from SERE training which is geared towards resisting information extraction.

China by their own words (internal leaked documents), seeks the destruction of the national will, of their enemies. This involves an identity based approach to torture/thought reform, which falls under the military strategy, Divide and Conquer. Digital attacks are cost effective when weighed against other options.

Anything you believe, love, or common experiences that you share with other people is fair game for inducement and then destructive interference to promote nihilism, while segmenting individuals into two groups, disassociative responses (apathetic/non-response), and psychotic break responses.

The items targeted include chess, along with many other things. Inducement of struggle sessions to break people.

If you spend the time to review the material I mentioned, you'll likely find out that a core belief of yours is untrue, that belief being that something like this is fantasy and impossible. This has a way of breaking the weak-willed, often in a psychological reversal/delusion.

I hope you are a strong person, we need more rational people if we are to survive as a species.

Robert Lifton (Thought Reform and The Psychology of Totalism) Joost Meerloo (Rape of the Mind) Robert Cialdini (Influence) USMC Press (Political Warfare)(Free ebook at their website)

Re: Lichess: Post-Mortem of Our Longest Downtime

#49

Post by AlexeyBelov »

trod123 wrote: post #48
OkayPhysicist wrote: post #45

Earlier quoted context omitted.

What kind of Tom Clancy novel do you live in that intelligence agencies are astroturfing for free chess sites?

I'm going to assume that your question is genuine and sincere, and not meant sarcastically. If you read the following books by established experts, you should be able to rationally answer the question for yourself as to the why and the how. The subject matter involves torture for thought reform, real not fantasy. This differs from SERE training which is geared towards resisting information extraction. China by their…

You need to seek professional help.

Re: Lichess: Post-Mortem of Our Longest Downtime

#50

Post by mplewis »

trod123 wrote: post #48
OkayPhysicist wrote: post #45

Earlier quoted context omitted.

What kind of Tom Clancy novel do you live in that intelligence agencies are astroturfing for free chess sites?

I'm going to assume that your question is genuine and sincere, and not meant sarcastically. If you read the following books by established experts, you should be able to rationally answer the question for yourself as to the why and the how. The subject matter involves torture for thought reform, real not fantasy. This differs from SERE training which is geared towards resisting information extraction. China by their…

Log off.
Post reply on HN