We've been on freeze for weeks now in preparation for the holiday season.
Tell HN: I salute everyone on call/working support through the holidays
51–60 of 207 posts
Re: Tell HN: I salute everyone on call/working support through the holidays
#52Meanwhile a huge number of us (non-religious? introverted kernel compiling cave dwellers?) treat this period no differently than any other week in the year. I'll be here keepin the servers runnin :horns: It's actually my favorite time of the year. Everyone is gone, it is quiet, and I can get shit done.
Holidays are special because they’re special, both the winter solstice festival (rebranded for christianity) and the spring equinox one (same deal) can be treated differently for cultural variety by the non-observant.
I’m a militant proselytizing atheist raised by a jew and I still have a tree with pretty lights, give presents, and drink and eat some things I only drink/eat once per year (never make homemade eggnog if you ever want to enjoy it guilt free again, you’re basically drinking a megacalorie of heavy cream, yum). It’s fun to celebrate the generic concept of “holiday” - a time that is different from other times.
You’re allowed to feel nice about peppermint candy (and/or chocolate gelt, I go for both) at the end of December without bringing the supernatural into the equation. :)
\m/
Re: Tell HN: I salute everyone on call/working support through the holidays
#53Always be kind, and say it’s your fault. If you don’t do it for the sake of the person you are asking for help, do it because it works better. That’s the most practical advice [0] ever given by Hans Rosling [1], the Fact master himself: > In fact, I have the secret to how to get the best help immediately from any customer service, like the phone company or the bank or anything. I have the best line, it always works.…
[flagged]
Re: Tell HN: I salute everyone on call/working support through the holidays
#54FYI Israelis are not on holiday - our holidays are on whole different dates. Hire Israelis and experience no down time while working with Silicon Valley level talent
[deleted]
Re: Tell HN: I salute everyone on call/working support through the holidays
#55Thanks for the salute, but we also accept cash :)
Re: Tell HN: I salute everyone on call/working support through the holidays
#56> Thank you for keeping systems available and safe. theres that word "safe" again. What systems are dangerous otherwise? Do you mean like traffic lights or something? The API serving ads to your mobile game isn't dangerous.
'Safe' in the context of systems can mean hacking attempts, safe from data leaks and other emergencies relative to the system that may arise. It can refer to things that are dangerous for the system itself.
Re: Tell HN: I salute everyone on call/working support through the holidays
#57Meanwhile a huge number of us (non-religious? introverted kernel compiling cave dwellers?) treat this period no differently than any other week in the year. I'll be here keepin the servers runnin :horns: It's actually my favorite time of the year. Everyone is gone, it is quiet, and I can get shit done.
Holidays are special because they’re special, both the winter solstice festival (rebranded for christianity) and the spring equinox one (same deal) can be treated differently for cultural variety by the non-observant. I’m a militant proselytizing atheist raised by a jew and I still have a tree with pretty lights, give presents, and drink and eat some things I only drink/eat once per year (never make homemade eggnog i…
Oh ya same I love the smell of evergreen wreaths and trees and enjoy partaking in festive activities. A 4K cracklin’ Yule log goes a long way too.
Re: Tell HN: I salute everyone on call/working support through the holidays
#58Earlier quoted context omitted.
I think that’s a great policy as it’s clearly intended to help people when they need it, and get people to unplug when it’s valued by their loved ones. _However_ (that part is probably best bookmarked until Jan 2nd), it also betrays that your system is brittle and can be broken by a bad commit. Don’t do it because you want people to grind until Dec 24th at 6 pm. Do it because it’s great the rest of the year, too. I’d…
We use these systems liberally on other times of the year and no one notices, usually. If they do, downtime and interruption budgets handle this. /However/, let me counter with the point: Just one of our customer has 8000 FTEs working with our system. During hell-time (aka, December and Christmas shopping and shipping), each of those dudes spends their shift taking customer calls lasting 2-4 minutes, which in turn re…
You are a lot more ahead than most companies.
I’ve worked for too many places where the Christmas break was because of a lack of tooling. I’m glad you are two steps ahead.
Re: Tell HN: I salute everyone on call/working support through the holidays
#59Yeah, we discourage production changes starting first or second december week, and start freezing changes third december week until it's frozen solid fourth december week until second week of january. December tends to be hell for our customers, so stability should be a priority there. And honestly, no one wants to work on holidays. So lets just wrap everything starting in december, maybe use the third week for some…
The place I work for pushed v2 of their software, a full rewrite (nothing from the old system, not even databases) by a new team, into production this week for several customers. Mostly they did it so they could say they met their made up 2023 KPIs for the v2 rewrite. There was no good reason to push it out now other than that, and there were several reasons not to, such as it wasn’t well tested and it’s fucking Dece…
Ugh. Several years ago I spent an entire Christmas vacation, including all day Christmas Day, putting out fires because a team couldn't be bothered to do five minutes of cursory load testing. As a consequence, multiple production systems went down under load.
Later, after we regrouped after a month of this brutality, they wandered around the office bragging like they'd hung the fucking moon after they fixed the crippling, obvious design issue they'd released. I confronted the dev lead with the fact that they would have seen this after 30s of load testing and he just laughed, I think he literally said "LOL". A giant middle finger, that's what Ops got from Dev for Christmas that year.
Here's to the people who KTLO. My people.
Re: Tell HN: I salute everyone on call/working support through the holidays
#60Earlier quoted context omitted.
I think that’s a great policy as it’s clearly intended to help people when they need it, and get people to unplug when it’s valued by their loved ones. _However_ (that part is probably best bookmarked until Jan 2nd), it also betrays that your system is brittle and can be broken by a bad commit. Don’t do it because you want people to grind until Dec 24th at 6 pm. Do it because it’s great the rest of the year, too. I’d…
How do you detect errors like this? What is an error? Is a business logic bug going to be picked up by this process automatically, or is some manual steps involved? Ie a point of sale app releases an update that automatically halves the amount to charge, but displays the full amount to the merchant in the UI. Unit tests pass (because an engineer made a human mistake). Backend calls are correctly used, no errors throw…
As you point out, it really depends on what is an error. Most of the companies I know of have a Holiday freeze are video games, casual ones, even. Changes are minor fixes and optimization—glitches that a player likely won’t notice, but you want to detect them early to avoid losing your ability to detect more.
Back-end tools are different, and I definitely see reasons other than bugs to not change business logic this month.