Live data from Hacker News

Tell HN: I salute everyone on call/working support through the holidays

news.ycombinator.com

21–30 of 207 posts

Re: Tell HN: I salute everyone on call/working support through the holidays

#21
Agreed, there are some gigs that just really require support to exist - I know this first-hand from working at a Zoo (very large exotic animal rescue basically). Animals do not take holidays. They need to eat and do animal things in spite of our costumes that day.

On the flip side, having worked Cinema on Christmas Day two years I think, there is no amount of Grace and Patience I can give that is enough to those earning their living. Still have a hat and polo. Why? I had to buy them!

Re: Tell HN: I salute everyone on call/working support through the holidays

#22

On call till 31st, so please don't hit refresh too much this days ;-)

Me too, but they pay a few bucks an hour to carry the phone so at least that adds up. Ultimate trick is to have a diverse team. Someone that doesn’t care about Christmas but absolutely needs some random day off in March (cool with us!). Someone that celebrates new years some other time.

Speaking of diversity, if you don’t do Christmas dinner I strongly recommend ordering takeout from a Chinese place on the 25th! There’s lots of happy photos of Chinese chefs and Jewish customers doing a Christmas fist-bump.

Re: Tell HN: I salute everyone on call/working support through the holidays

#24
post #12

Yeah, we discourage production changes starting first or second december week, and start freezing changes third december week until it's frozen solid fourth december week until second week of january. December tends to be hell for our customers, so stability should be a priority there. And honestly, no one wants to work on holidays. So lets just wrap everything starting in december, maybe use the third week for some…

I think that’s a great policy as it’s clearly intended to help people when they need it, and get people to unplug when it’s valued by their loved ones.

_However_ (that part is probably best bookmarked until Jan 2nd), it also betrays that your system is brittle and can be broken by a bad commit. Don’t do it because you want people to grind until Dec 24th at 6 pm. Do it because it’s great the rest of the year, too. I’d recommend you look into (or ask me about) feature flags, alerting, and automated roll-backs.

The short version is: there’s a meta-system on top of your release process that can tell (if you are using roll-back not features flags): - commits until xyzsdf are fine; - roll-outs starting from commit abcdef have a 2% error rate, 80% on Android; - revert to xyzsdf, send a message (low-priority, email) to the DevOps on call and the author of abcdef that it happened; - for all commits after abcdef: if there no conflicts with xyzsdf, re-try to roll them out; - if there is a conflict because they were on top or abcdef, send a message (low-priority email) to the authors that there is a conflict.

There are more sophisticated versions that can do things like, if you use feature flags, flagging Android users to use the previous version. Another way to do this is to scale who has access to abcdef gradually: say 1% every hour, and revert if you detect issues.

All those seem daunting to teams that haven’t worked like this before, but it my experience, they love it very fast.

Re: Tell HN: I salute everyone on call/working support through the holidays

#25
Always be kind, and say it’s your fault.

If you don’t do it for the sake of the person you are asking for help, do it because it works better. That’s the most practical advice [0] ever given by Hans Rosling [1], the Fact master himself:

> In fact, I have the secret to how to get the best help immediately from any customer service, like the phone company or the bank or anything. I have the best line, it always works. You want to know what it is? When I call, I say, “Hello. I am Hans Rosling and I have made a mistake.” People immediately want to help you when you put it this way. You get much more when you don’t offend people.

[0]: Unless you are in charge of a developing country’s budget and have to decide between education and healthcare.

[1]: https://blog.ted.com/qa_with_hans_ro_1/

Re: Tell HN: I salute everyone on call/working support through the holidays

#26
Meanwhile a huge number of us (non-religious? introverted kernel compiling cave dwellers?) treat this period no differently than any other week in the year. I'll be here keepin the servers runnin :horns:

It's actually my favorite time of the year. Everyone is gone, it is quiet, and I can get shit done.

Re: Tell HN: I salute everyone on call/working support through the holidays

#27
post #17
post #3

Yes, absolutely thanks to all who keep our world running when no one is looking. To keep the yule log on Youtube, to keep our christmas tree lights on, to keep a fresh glass of water from the tap, warm natural gas to keep the freezing cold outside etc. Thank you for keeping society ticking away :)

Let's not confuse on-call firefighters or a water facility staff with the on-call admins that maintain money-making machines monetizing attention of billions. The latter is a net negative on society.

Yeah, how dare Netflix provide entertainment on-demand and for cheaper than the other entertainment companies?

Re: Tell HN: I salute everyone on call/working support through the holidays

#29
post #17
post #3

Yes, absolutely thanks to all who keep our world running when no one is looking. To keep the yule log on Youtube, to keep our christmas tree lights on, to keep a fresh glass of water from the tap, warm natural gas to keep the freezing cold outside etc. Thank you for keeping society ticking away :)

Let's not confuse on-call firefighters or a water facility staff with the on-call admins that maintain money-making machines monetizing attention of billions. The latter is a net negative on society.

[deleted]

Re: Tell HN: I salute everyone on call/working support through the holidays

#30
post #3

Yes, absolutely thanks to all who keep our world running when no one is looking. To keep the yule log on Youtube, to keep our christmas tree lights on, to keep a fresh glass of water from the tap, warm natural gas to keep the freezing cold outside etc. Thank you for keeping society ticking away :)

[deleted]
Post reply on HN