Sweating over here trying to make it through the week and praying that it slows at least for the first half of next week.
Tell HN: I salute everyone on call/working support through the holidays
31–40 of 207 posts
Re: Tell HN: I salute everyone on call/working support through the holidays
#32FYI Israelis are not on holiday - our holidays are on whole different dates. Hire Israelis and experience no down time while working with Silicon Valley level talent
Re: Tell HN: I salute everyone on call/working support through the holidays
#33Meanwhile a huge number of us (non-religious? introverted kernel compiling cave dwellers?) treat this period no differently than any other week in the year. I'll be here keepin the servers runnin :horns: It's actually my favorite time of the year. Everyone is gone, it is quiet, and I can get shit done.
Or a member of one of the religions that don't celebrate Christmas.
Re: Tell HN: I salute everyone on call/working support through the holidays
#34Always be kind, and say it’s your fault. If you don’t do it for the sake of the person you are asking for help, do it because it works better. That’s the most practical advice [0] ever given by Hans Rosling [1], the Fact master himself: > In fact, I have the secret to how to get the best help immediately from any customer service, like the phone company or the bank or anything. I have the best line, it always works.…
I do this with internal teams at work. I've found approaching other teams with issues with their library/framework in a "this could be our mistake" manner really helps in keeping them from getting defensive and stonewalling.
Re: Tell HN: I salute everyone on call/working support through the holidays
#35Yeah, we discourage production changes starting first or second december week, and start freezing changes third december week until it's frozen solid fourth december week until second week of january. December tends to be hell for our customers, so stability should be a priority there. And honestly, no one wants to work on holidays. So lets just wrap everything starting in december, maybe use the third week for some…
I think that’s a great policy as it’s clearly intended to help people when they need it, and get people to unplug when it’s valued by their loved ones. _However_ (that part is probably best bookmarked until Jan 2nd), it also betrays that your system is brittle and can be broken by a bad commit. Don’t do it because you want people to grind until Dec 24th at 6 pm. Do it because it’s great the rest of the year, too. I’d…
What is an error? Is a business logic bug going to be picked up by this process automatically, or is some manual steps involved?
Ie a point of sale app releases an update that automatically halves the amount to charge, but displays the full amount to the merchant in the UI. Unit tests pass (because an engineer made a human mistake). Backend calls are correctly used, no errors thrown, simply the wrong amount is used.
How would this be automatically detected and reverted?
Would anyone writing point of sale software want to risk this over one of the biggest trading periods of the year?
Re: Tell HN: I salute everyone on call/working support through the holidays
#36Yeah, we discourage production changes starting first or second december week, and start freezing changes third december week until it's frozen solid fourth december week until second week of january. December tends to be hell for our customers, so stability should be a priority there. And honestly, no one wants to work on holidays. So lets just wrap everything starting in december, maybe use the third week for some…
I think that’s a great policy as it’s clearly intended to help people when they need it, and get people to unplug when it’s valued by their loved ones. _However_ (that part is probably best bookmarked until Jan 2nd), it also betrays that your system is brittle and can be broken by a bad commit. Don’t do it because you want people to grind until Dec 24th at 6 pm. Do it because it’s great the rest of the year, too. I’d…
/However/, let me counter with the point: Just one of our customer has 8000 FTEs working with our system. During hell-time (aka, December and Christmas shopping and shipping), each of those dudes spends their shift taking customer calls lasting 2-4 minutes, which in turn require a few requests into our systems.
Due to the stress of their customers^2 (because it's Christmas and holidays and such), if an agent of a customer is unable to access our systems, they cannot handle the use case of the customer^2 and that will piss of the customer of the customer.
So if we push a bad change during this time, we're going to piss of hundreds of customers^2 per minute for that one customer alone. Even with a fast automatic rollback, that's a long time during hell-time. And they have people who know how to yell at vendors in nasty ways who don't like that.
I enjoy moving software fast and enabling moving software quickly, but customer focus and customer orientation means to understand when to move slow as well.
And hey, if that means more quiet holidays for the hard working operators on my team, who's gonna complain?
Re: Tell HN: I salute everyone on call/working support through the holidays
#37Re: Tell HN: I salute everyone on call/working support through the holidays
#38FYI Israelis are not on holiday - our holidays are on whole different dates. Hire Israelis and experience no down time while working with Silicon Valley level talent
Re: Tell HN: I salute everyone on call/working support through the holidays
#39Meanwhile a huge number of us (non-religious? introverted kernel compiling cave dwellers?) treat this period no differently than any other week in the year. I'll be here keepin the servers runnin :horns: It's actually my favorite time of the year. Everyone is gone, it is quiet, and I can get shit done.
Re: Tell HN: I salute everyone on call/working support through the holidays
#40Yes, absolutely thanks to all who keep our world running when no one is looking. To keep the yule log on Youtube, to keep our christmas tree lights on, to keep a fresh glass of water from the tap, warm natural gas to keep the freezing cold outside etc. Thank you for keeping society ticking away :)
Let's not confuse on-call firefighters or a water facility staff with the on-call admins that maintain money-making machines monetizing attention of billions. The latter is a net negative on society.