Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Developer on Call

henrikwarne.com

Post reply on HN
Print view View original

201–210 of 246 posts

Re: Developer on Call

#201

Post by [deleted] »

wikibob wrote: post #16
jedberg wrote: post #3

> Pay. People on call should get paid extra for it I've been on call for 20+ years. I've never gotten paid extra for it. I just figure it's baked into the normal paycheck. As long as everyone on the team is doing on call about the same amount, it doesn't really matter. At the end of the year, it usually works out pretty evenly. > Scheduling. When I have been on call, it has always been one week at a time, I agree wit…

Google pays people, both in cash, and/or in compensatory time off. This is specifically called out in the SRE book [0]. They've noted that it's important to pay compensation, both to be fair to the employees, and as a closed-loop feedback mechanism to ensure the business prioritizes fixing pages. This concept of business feedback is also discussed in a chapter of the terrific Seeking SRE book, chapter "Against On-Cal…

[deleted]

Re: Developer on Call

#202

Post by gnulinux »

js8 wrote: post #11

In some companies, there is a difference between developers (people who create new features) and L2 or L3 support (people who fix bugs and resolve problems for the customer). The trend is not to have this division, and I disagree with that. I agree that it is good to try both things, and developers should try to be support once in a while, and vice versa. However, I think they are very different mindsets and doing bo…

If I introduce a bug, I fix that bug. I don't understand your comment, sorry.

Re: Developer on Call

#203

Post by watwut »

sdegutis wrote: post #200
watwut wrote: post #194

Earlier quoted context omitted.

That strategy would leave you with dozens and later hundreds of forks and variants. You need to make decision here and appoint product owner to make those decisions consistently.

Extrapolating from Alan Perlis's quote, I think it's better to have 3 products that each do 1 thing well rather than to have 1 product that does 3 things very poorly.

Practically speaking, you will have multiple products that are crappy, because you will be unable to maintain them all and keep track of what should work how in each version. Moreover, fixes that should go to each will take more time to develop, testing will take more time and there still will be additional bugs due to complexity of it all.

Also, organization and management of it all will take more time and effort. You need to reconcile them or make decision.

Re: Developer on Call

#204

Post by gaius »

jedberg wrote: post #171
gaius wrote: post #12

Earlier quoted context omitted.

always the on-call engineer, me (the team lead), and then my manager and then their manager. The best scheme in my experience is to have a 24/7 fully staffed L1 working shifts for initial response triage then waking the appropriate L2 to deal with the actual problem if it isn’t covered in the runbook. No good waking the DBA if you can’t login to the database because a router has crashed and failed to failover. The ne…

If you have 24/7 coverage using run books, perhaps a better system would be to take the money you're spending on the 5ish people you need to do 24/7 coverage, and instead pay two people twice as much to codify everything in the run book so that it happens automatically. Then have the alerts go straight to the L2.

The infra for that job had far too many moving parts to make that feasible unfortunately. Multiple fibre providers, satellites, microwave relays, etc etc. The L1 guys would be on the phone alot in the event of a serious incident. Even today there still needs to be humans in the loop, I don’t see that changing.

Re: Developer on Call

#205

Post by sheepmullet »

hardwaresofton wrote: post #82
sheepmullet wrote: post #66

Earlier quoted context omitted.

> If on call is optional what's with the social penalty for people not wanting to do it. Because many optional activities have an impact on your peers and they are unlikely to judge you strictly based upon your job duties?

I think we don't have the same definition of optional -- like someone else has noted, maybe the better word was "flexible". The way you're using it is the super manipulative "yeah it's optional, but why would you want the rest of your team to suffer?". Does that not sound manipulative to you? As far as your impact-on-peers argument -- you could "optionally" also stay 5 hours after when you normally go home to help re…

> Does that not sound manipulative to you?

You seem to be assuming there is a lot of peer pressure placed on you if you don’t want to do it.

Why?

I’m simply saying there are always social costs. For example you probably won’t be listened to as much when there are conversations around improving system stability.

It’s like our after work Friday drinks are entirely optional - but lots of people build friendships and trust there and this can often lead to higher productivity.

If you can build these friendships another way or have a different path to an equivalently high productivity then not going doesn’t have an impact on you.

Re: Developer on Call

#206

Post by watwut »

sheepmullet wrote: post #205
hardwaresofton wrote: post #82

Earlier quoted context omitted.

I think we don't have the same definition of optional -- like someone else has noted, maybe the better word was "flexible". The way you're using it is the super manipulative "yeah it's optional, but why would you want the rest of your team to suffer?". Does that not sound manipulative to you? As far as your impact-on-peers argument -- you could "optionally" also stay 5 hours after when you normally go home to help re…

> Does that not sound manipulative to you? You seem to be assuming there is a lot of peer pressure placed on you if you don’t want to do it. Why? I’m simply saying there are always social costs. For example you probably won’t be listened to as much when there are conversations around improving system stability. It’s like our after work Friday drinks are entirely optional - but lots of people build friendships and tru…

> For example you probably won’t be listened to as much when there are conversations around improving system stability.

That sounds like not listening to people about things they might be good at and know something about, because you want to punish them for something completely unrelated. Namely, punish them for not participating in "optional" activities. All the while you don't want to openly and transparently say what you expect from people.

Yes, it is manipulative and it is bad workplace.

> It’s like our after work Friday drinks are entirely optional - but lots of people build friendships and trust there and this can often lead to higher productivity.

It sounds sounds like nepotism where your ability to function and be promoted rests on your ability to make friends and be charming around beer.

No a meritocracy, but rather badly managed workplace.

-----------------

Seriously, you openly say that you would listen and judge system stability suggestions based on participation in supposedly optional activity unrelated to system stability. You also openly say that you trust people work based on Friday beer instead on how they act when working.

That sounds like horrible workplace for anyone who care about work and great workplace for charming bullshitters.

Re: Developer on Call

#207

Post by Verdex_3 »

sevensor wrote: post #172
Verdex_3 wrote: post #160

Earlier quoted context omitted.

Bad things can work out for your own personal good. Or even the good of the whole of society. However, that doesn't make them not bad. I'm glad that your situation worked out for your own personal good. Nice things happening to people make me happy. Things working out for people make me happy. However, the situation you describe is the latter not the former. That your bad situation, which ultimately worked out for yo…

So if your measure of "objective" goodness is utilitarian calculus, as it seems to be, you're leaving out the fact that employees getting the shaft tends to correlate with cheaper goods and services. I disagree that this is any more objective than my initial assessment that "this is OK for now" or my later assessment that "this sucks, I'm going back to school." But this all comes back to what I said about "objective"…

Not utilitarian calculus. It's closer to spider-man's Great power comes great responsibility.

For example, if I figured out the secret to creating strong AI with respect to writing software such that I could replace the entire software engineering industry with one large computer (note: this isn't something I believe will be possible for centuries if it is ever possible), then I would feel compelled to use the billions of dollars this would undoubtedly get me to help retrain all of the software engineers I just permanently put out of a job.

It's also stated as 'if it is within your power to do good, then you should do it'. My contention is that a powerful company should hire more people to cover additional work instead of finding creative ways to get additional work out of currently employed people for the same amount of money because hiring more people is a good that they are able to do and getting more work for less money isn't.

I'm fine with us not agreeing. But I'm also fine with me being right, which is why I'm still typing.

Re: Developer on Call

#208

Post by u801e »

ownagefool wrote: post #178
Verdex_3 wrote: post #104

Earlier quoted context omitted.

Division of labor is a thing. I'm good at transforming customer interactions into requirements and then transforming requirements into working code. I'm not good with dealing with issues with a deployed product while I should be busy living my life.

So then you should probably only work on products that don't require out of hours support, or where the team is willing and able to hire people in another time zone. Instead, you think some ops guy should do it?

Perhaps the company should invest the resources into hiring people to provide out of hours support rather than expecting their employees to be on call 24/7 for certain weeks of the month. Otherwise, why shouldthe company bother offering out of hours support if it's not willing to pay for it.

Re: Developer on Call

#209

Post by ascar »

HeyLaughingBoy wrote: post #119
ascar wrote: post #68

Earlier quoted context omitted.

My contract with a company that had on call read somewhere along the lines of 9 to 5 are the core working times and all work necessary outside these times, including Saturday and Sunday, is already remunerated in the base salary. In this case I was being paid for it and I only heard about it in the first week after starting. I asked "how often do I have to work on weekends", but not "will i be on call" in the contrac…

Why do you think it's a good thing?

Because it increases responsibility, when you are the who has to clean up the mess. It makes you care more to keep your system reliable and available, because your personal stakes are higher. There are some good comments here along those same lines.

Of course getting a call middle of the night, when your system goes down because of an AWS outage you can nothing do about just sucks ;)

Re: Developer on Call

#210

Post by ticmasta »

Novashi wrote: post #168
stevoski wrote: post #139

Earlier quoted context omitted.

That's a great description of the subtleties of product development, especially within a small team. Many books and articles make it seem like a simple procedure, but actually you often face decisions which will leave some people unhappy, no matter what you decide.

You basically have to get into supporting extensions or custom code and supporting an API if you run into enough of these customers and that’s when you declare that you only have so much responsibility for the customer’s code (and it quickly turns into another support revenue stream).

I worked for a company that had product managers that were very talented at turning requests for custom features into solutions that we enabled (and our core was required to implement) but using a third party to implement. Consulting was used to find new core features and keep a finger on the pulse but not as a significant revenue generator. It enabled us to grow a product company to big revenue per headcount metrics, which is hard to do as a service company.

The situation described by GP is primarily a expectation and people management problem, not a technical issue.

Post reply on HN