Live data from Hacker News

Lucky Lotto, chaos engineering but for teams

danlebrero.com

31–38 of 38 posts

Re: Lucky Lotto, chaos engineering but for teams

#31

Could you imagine being part of this. Told the company was running DiRT testing or whatever, and then getting fired or punished for listening and following instructions? Could you imagine afterwards being lectured to use common sense. Also, we have secret corporate policies you need to know about and follow. Resolve that catch-22. Repeat for a decade. You'd be justified in never trusting them with your life or anothe…

... did you intend to reply here, or on some other post?

Re: Lucky Lotto, chaos engineering but for teams

#32
post #4

I heard a good presentation about this. They did it for a team at Google. But it was daily, and you weren't allowed to tell anyone else. You just didn't respond to any requests for that day. But even worse, you could also be assigned the "liar" task. In which case you were supposed to reply to emails but give intentionally wrong information some of the time. But in that case you would tell people that you were the li…

I wonder if this encouraged people to be more open with their ideas also.

If I could say things knowing that if I made a mistake, someone might friendly correct me with "Ah, you're the liar today, S3 doesn't support that format", I'd be happier to make them.

Re: Lucky Lotto, chaos engineering but for teams

#33

Earlier quoted context omitted.

Maybe you didn't read the whole article then. It totally describes how to do it. The bottom has a nice summary but I'll quote the solution parts of it here: The winner will work on some side project. Still work. Not a solution to the bus-factor of people but a good solution to the "we never get to work on tech debt / platform work because features" problem. This alone is awesome about this. Everybody, including produ…

Keeping aside the insulting nature of suggesting someone did not read the article, here is my response. I like to differentiate between mechanism to make people realize a problem and actual solutions to the problem. Its very easy to say mentor someone to do your job but if that was doable and easy they would be already doing it. The problem is precisely that there is no real good way to do KTs. Forcing someone to sol…

FWIW I operate under the assumption that HN is just as bad as Slashdot with regards to commenting without reading the article :) and the solutions were pretty clear to me, even without the nice summary at the end. Sorry if it wasn't worded softly enough, no insult meant, more an observation.

The parts I mentioned are not the 'realization' part. They are the solution parts. They aren't the 'implementation details' of the solution, I would agree, but they are the solution.

If you ask me the problem is not usually that there is no good way to do KT or that it just isn't doable at all. The problem is that in most businesses due to their 'culture' (for lack of a better word - not to start "that" culture discussion) you do not actually get to do it. A good way to mentor and transfer knowledge for example is to do pair programming. Not many places allow for that and will look at you funny for even suggesting it (and I'm not even personally on the extreme end of that like some companies, where everyone literally pair programs 100% of the time - I like a sort of hybrid model, where people pair for as long as it makes sense to them, which could be sitting there "designing" for a couple of hours together, maybe dividing things up after a basic structure is in place and then working on their own for the rest of the day w/ some quick 5 minute sync ups and questions going back and forth from time to time). Another very doable way to do knowledge transfer is to specifically not give let the one guy that wrote that part of the code and knows it inside and out work on the next ticket that will need changes to that part. But many Product Managers/businesses/team leads will not allow that because it would mean that the task will be delivered slower.

The beauty of this approach is that you don't need to actively gather data, make a decision etc. Gathering this data is usually very error prone in that you can fill out forms and skills matrices and such all you want (been there done that), you always forget about something or it doesn't really tell you the whole story (skills matrices are particularly bad)

With this, it just happens! It's the self organizing way of dealing with the problem. I think you might be putting too much emphasis on the "completely unavailable" part, whereas I see the "it's a soft rule" part bigger. Instead of waiting for someone to go on vacation and then you find out that he was the only one that can do X (there's your data point that you missed during collection and analysis) and now you're in deep trouble, because he's touring the Amazon rain forest (i.e. definitely no cell reception there), you get to figure it out because he won the lottery and he's actually at work and can tutor someone through it all.

Re: Lucky Lotto, chaos engineering but for teams

#34
post #13
post #8

Earlier quoted context omitted.

The challenge there is that you need to hire and train more people than you have right now, and hiring is difficult. I mean probably not so much if you have a lot of money. I'm currently in a high bus factor job, I'm the only developer on the UI - our CTO can do some small jobs here and there, but nobody's touched or even looked at the new UI I'm building (Go + React). I want to be able to leave, but the way things a…

> I'm currently in a high bus factor job, [...] That's actually a low bus factor, isn't it? https://en.wikipedia.org/wiki/Bus_factor

From the article you linked: "There is a rare alternative definition for the bus factor, namely: the number of people who are indispensable for the project. In other words, it is the minimum number of people who are a single point of failure. If using this definition, then a high bus factor is considered a bad thing (since the loss of any person included destroys the project), and zero is considered the ideal bus factor."

Perhaps "bus risk" would be a better term for this usage?

Re: Lucky Lotto, chaos engineering but for teams

#35

Banks require a planned version of this: you have to take so many contiguous days off at least once a year. But it's an anti-fraud measure, as most of the frauds you can run internally require you to be there to juggle things, and you make the number of contiguous days long enough where such a scheme will come crashing down.

Some stopped in the last few years, or at least for certain job categories. It was a wonderful thing when it was around because you could take a vacation with absolutely no remote access.

Re: Lucky Lotto, chaos engineering but for teams

#36
post #4

I heard a good presentation about this. They did it for a team at Google. But it was daily, and you weren't allowed to tell anyone else. You just didn't respond to any requests for that day. But even worse, you could also be assigned the "liar" task. In which case you were supposed to reply to emails but give intentionally wrong information some of the time. But in that case you would tell people that you were the li…

How are people afterwards. I refuse to use email at work anymore because of the constant phishing tests. Sick of the gotcha mentality.

I agree. We have a security score at work and mine had been zero for a long time. I realized that you need to report the emails as spam or phishing and not just ignore/delete like I usually do.

Re: Lucky Lotto, chaos engineering but for teams

#37

Earlier quoted context omitted.

Keeping aside the insulting nature of suggesting someone did not read the article, here is my response. I like to differentiate between mechanism to make people realize a problem and actual solutions to the problem. Its very easy to say mentor someone to do your job but if that was doable and easy they would be already doing it. The problem is precisely that there is no real good way to do KTs. Forcing someone to sol…

FWIW I operate under the assumption that HN is just as bad as Slashdot with regards to commenting without reading the article :) and the solutions were pretty clear to me, even without the nice summary at the end. Sorry if it wasn't worded softly enough, no insult meant, more an observation. The parts I mentioned are not the 'realization' part. They are the solution parts. They aren't the 'implementation details' of…

Apology accepted. Thanks.

> But many Product Managers/businesses/team leads will not allow that because it would mean that the task will be delivered slower.

This is precisely the kind of communication breakdown I was talking about. Managers are thinking short term, forgetting all software needs to be maintained.

It may be a team personality thing, but I think there would be way more resistance to this kind of chaos engineering than a survey.

Re: Lucky Lotto, chaos engineering but for teams

#38

Earlier quoted context omitted.

FWIW I operate under the assumption that HN is just as bad as Slashdot with regards to commenting without reading the article :) and the solutions were pretty clear to me, even without the nice summary at the end. Sorry if it wasn't worded softly enough, no insult meant, more an observation. The parts I mentioned are not the 'realization' part. They are the solution parts. They aren't the 'implementation details' of…

Apology accepted. Thanks. > But many Product Managers/businesses/team leads will not allow that because it would mean that the task will be delivered slower. This is precisely the kind of communication breakdown I was talking about. Managers are thinking short term, forgetting all software needs to be maintained. It may be a team personality thing, but I think there would be way more resistance to this kind of chaos…

Depending on the company I would agree that there might be more initial resistance. The thing with a survey is that in way too many cases it goes nowhere after that. We did the survey. We filled out a skills matrix. Never to be heard of again (been there done that).

If you can get the chaos engineering "allowed", I think it's easier to actually follow through.

There are multiple scenarios I can think of here.

* This might be something that a team lead or a bunch of team leads want to initiate/try out. You ideally need buy in from at least your boss, potentially a bit higher up. * This could also be thought up/initiated by the development manager/VP of engineering/CTO level who now has to get his team leads to run this and might run into resistance from them actually or from Product. Sort of like with the technology based chaos engineering, where there might be a special SRE person/team responsible for running this, you could have a special team/person for this as well. If we were all in offices, it might be a bunch of tall dudes, physically escorting the chosen person from the team room :)

Post reply on HN