Live data from Hacker News

Lucky Lotto, chaos engineering but for teams

danlebrero.com

1–10 of 38 posts

Re: Lucky Lotto, chaos engineering but for teams

#2
I have used similar discussion points as in this article to argue that parental leaves, long vacations and other similar employee benefits build resilient teams and products. If you have to plan for people going missing (without the possibility for immediate replacement) you are forced to spread knowledge in the organization and you can't be too dependent on individuals.

Re: Lucky Lotto, chaos engineering but for teams

#4
I heard a good presentation about this. They did it for a team at Google. But it was daily, and you weren't allowed to tell anyone else. You just didn't respond to any requests for that day.

But even worse, you could also be assigned the "liar" task. In which case you were supposed to reply to emails but give intentionally wrong information some of the time. But in that case you would tell people that you were the liar and that your answers aren't to be trusted.

It seemed like a good way to make sure at least two people could do everything and that documentation was solid enough that you could recognize when someone was wrong.

Re: Lucky Lotto, chaos engineering but for teams

#5

I like this - we’ve had “reduce our bus factor” on the to-do list for a while, but not found a way to do it yet.

I don't think this is a mechanism to reduce the bus factor, to me it seems like a mechanism to make people realize what the bus factor is for the team. The actual solutions were not described in the article but I guess that is very team specific.

Re: Lucky Lotto, chaos engineering but for teams

#6

I like this - we’ve had “reduce our bus factor” on the to-do list for a while, but not found a way to do it yet.

I don't think this is a mechanism to reduce the bus factor, to me it seems like a mechanism to make people realize what the bus factor is for the team. The actual solutions were not described in the article but I guess that is very team specific.

Aligning the pain with the people who can solve it is the first rule of getting things done. Make the developers who will have to take on the bus-factor workload realize what the bus factor is, and they'll figure out how to reduce their dependencies real quickly.

Another good take on this is the "Wheel of Misfortune"; Take a real incident (or synthesize one in testing) that paged someone inconveniently. They've already solved it - They're running the exercise. Now have everyone else on the team, individually or as a group, figure out how to solve it. Not from the debugging steps or postmortem of the incident, but before that's been shared - Have them all suggest what they'd look for, and how to fix it. Their most important resource is their instructor/adversary for this, so give them all the data, but no map to it. Further, have the instructor figure out how to respond to all the unknowns - What else was broken because of the incident? What would have happend if they tried different debugging steps? Builds team knowledge real fast.

Re: Lucky Lotto, chaos engineering but for teams

#7

I like this - we’ve had “reduce our bus factor” on the to-do list for a while, but not found a way to do it yet.

I don't think this is a mechanism to reduce the bus factor, to me it seems like a mechanism to make people realize what the bus factor is for the team. The actual solutions were not described in the article but I guess that is very team specific.

Yes, good point, I guess the first step is realising you have a problem and proving it to management!

Re: Lucky Lotto, chaos engineering but for teams

#8

I like this - we’ve had “reduce our bus factor” on the to-do list for a while, but not found a way to do it yet.

The challenge there is that you need to hire and train more people than you have right now, and hiring is difficult. I mean probably not so much if you have a lot of money.

I'm currently in a high bus factor job, I'm the only developer on the UI - our CTO can do some small jobs here and there, but nobody's touched or even looked at the new UI I'm building (Go + React).

I want to be able to leave, but the way things are going - and the way recruiters are spinning up again - I'm afraid I'll have to bring them the bad news that I got an offer I can't refuse (and that they can't match; we're talking up to 150% pay rise / benefits).

What my company needs is a big bag of money so we can hire contractors to fast forward this project. Which will be a short term solution, but still. But this company isn't eager, it's run by mid-late career veterans who are happy with things bumbling along and a decent 15%/year growth. I can kinda respect that, but for my project that's not good enough. And it's an unsexy company, so they struggle to hire anyone.

Re: Lucky Lotto, chaos engineering but for teams

#10

I like this - we’ve had “reduce our bus factor” on the to-do list for a while, but not found a way to do it yet.

The most practical solution: Hire new people. Let the low-bus factor people mentor the new people to become them 2.0 (long term).

Downside: Takes time (and removes mentor from tasks deemed more important short term), costs money, people might need to get used to becoming mentors (I found most people enjoy this even if they cannot imagine doing it the first time around)

Upsides: Long term thinking and investing in employees is rarely wrong (imo)

Post reply on HN