Be Kind

You are allowed to overpay. The drivers don't like it because they are Seattle nice and would prefer you just not pay at all.

In my experience this strategy has not worked out very well. When we tried to deploy risky changes during low traffic times, we would end up creating time bombs, scale issues being masked until a high traffic time rolled around.

This also creates like a kind "brain latency" for developers I think. I'm coming at this from the operations side of things, so maybe my observations are a little biased here. I have observed that if people deploy changes and it breaks something immediately, it's a very clear correlation and they can fix the bug generally pretty quick. If they deploy and then it breaks 72 hours later, any number of things could be the culprit, especially in a fast moving environment (times 1 billion percent if it's a microservices architecture without a strong devops culture, which most of 'em are). Debugging then takes much much longer. This is made worse if the person who deployed the change is not quickly available when their thing breaks, and it makes being on call for someone else's unproven feature very stressful.

So instead I think it's better to make sure deployment and build systems are rock solid, and deploys are as accessible and as idempotent as possible. Chatops type systems are good here. Then you can roll out big changes during peak traffic and be confident that you can quickly revert if it goes bad, and that the changes were reliable under load if it goes good. I also think it's critically important that big changes are behind rollout flags, such that you can dial up or dial down traffic at will. This is also useful when introducing new cache systems or something like CDN if you need to warm up slowly.

This is a better approach I think than trying to use the time of day to modulate user traffic. I would rather developers can control traffic to their feature themselves and have the person deploying the change with their hands on the wheel until they are confident they can take them off. That way people can do stuff independently, and everyone can trust everyone to deploy and yet still feel safe.

I think intuition and emotional impulses is immensely important for communicating with non-technical stakeholders. They haven't learned the art of emulating a computer so we've got to be a human being. IMHO.

When I was 19 I started working as a video conference support person for Europe in a multinational. This meant interfacing with the highest people in the company, where a failure could make an insecure CEO or similar complain to the IT manager screaming how this was shit etc. this was using multiple ISDN lines making international calls, so it was prone to failure once in a while. After the first instance, the manager told me to not worry, but to just let him know when something failed so he had answers when the angry user came complaining. This was my first learning experience in this.

A few years later, in the same company, I was managing the middleware servers that handled all communications for the sales people in Europe. Here, when there was a failure, the sales people would have to fax in their copies of the data if there was a failure, so even more pressure. This is where I really learned to both appreciate the criticality of my mistakes as it could make several hundred people redo their work. However, the most important thing I learned with this was to slow down, and pause for a few minutes so that I could explain what was going on to the non technical managers and team leads. This was more important sometimes then actually solve the problem.

Both of these instances and a lot of others that happened over 11 years in that company thought me ot really double, triple and quadrouple check any change. It created a healthy paranoia, but also taught me that mistakes are ok, as long as you learn from them, but also, lean to communicate with higher ups that could not follow the technical jargo. I think the communication part is something that a lot of very confident people have problems with. However, it's very hard to teach this to younger techs, without letting them fail, so as a senior tech, I have seen problems from new juniors, but I have let them actually make the mistake, and feel the stress and pressure that they have messed up. I have found this to be the quickest method to teach responsibility, while standing by them and their mistakes.

Came here for that :)