Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Do you find working on large distributed systems exhausting?

news.ycombinator.com

Post reply on HN
Print view View original

71–80 of 261 posts

Re: Ask HN: Do you find working on large distributed systems exhausting?

#71

Post by ClumsyPilot »

aeyes wrote: post #65
fxtentacle wrote: post #18

Yes, I used to, but No, I fixed it :) Among other things, I am team lead for a private search engine whose partner-accessible API handles roughly 500 mio requests per month. I used to feel powerless and stressed out by the complexity and the scale, because whenever stuff broke (and it always does at this scale), I had to start playing politics, asking for favors, or threatening people on the phone to get it fixed. Hi…

I feel that your problems aren't even remotely related to my problems with large distributed systems. My problems are all about convincing the company that I need 200 engineers to work on extremely large software projects before we hit a scalability wall. That wall might be 2 years in the future so usually it is next to impossible to convince anyone to take engineers out of product development. Even more so because w…

"That wall might be 2 years in the future so usually it is next to impossible to convince anyone to take engineers out of product development. Even more so because working on this changes absolutely nothing for the end user"

It seems to be the same story in fiels of Infrastructure maintenance, Aircraft design (boeing Max), and mortgage CDOs (2008). Was it always like this or the new management doesn not care untill something explodes?

Re: Ask HN: Do you find working on large distributed systems exhausting?

#72

Post by grogers »

solatic wrote: post #9

What do you find exhausting? One anti-pattern I've found is that most orgs ask a single team to handle on-call around the clock for their service. This rarely scales well, from a human standpoint. If you're getting paged at 2:00 in the morning on a regular basis you will start to resent it. There's not much you can do about that so long as only one team is responsible for uptime 24/7. The solution is to hire operatio…

The solution to get paged at off hours a lot is rarely to hire additional teams to cover those times for you, at least not long term. For things you can control, you should fix the root causes of those issues. For things you can't control you should spend effort on making them within your control (eg architecture improvement). This takes time, so follow-the-sun rotation might be a stop gap solution, but you need to make sure it doesn't cover over the real problems without them getting any better.

Re: Ask HN: Do you find working on large distributed systems exhausting?

#73

Post by fxtentacle »

nostrebored wrote: post #38
fxtentacle wrote: post #18

Yes, I used to, but No, I fixed it :) Among other things, I am team lead for a private search engine whose partner-accessible API handles roughly 500 mio requests per month. I used to feel powerless and stressed out by the complexity and the scale, because whenever stuff broke (and it always does at this scale), I had to start playing politics, asking for favors, or threatening people on the phone to get it fixed. Hi…

You had problems with management of a cloud based api and executive visibility… so you bought a set of data centers to handle 500mio req per month? The visibility you will get after the capex when there’s a truly disastrous outage will be interesting.

You might be surprised. The performance equivalent of $100k monthly in EC2 spend fits into a 16m2 cage with 52HU racks.

Re: Ask HN: Do you find working on large distributed systems exhausting?

#74

Post by notacoward »

solatic wrote: post #9

What do you find exhausting? One anti-pattern I've found is that most orgs ask a single team to handle on-call around the clock for their service. This rarely scales well, from a human standpoint. If you're getting paged at 2:00 in the morning on a regular basis you will start to resent it. There's not much you can do about that so long as only one team is responsible for uptime 24/7. The solution is to hire operatio…

This. Absolutely this. Working on large distributed system can be both exhilarating and exhausting. The two often go hand in hand. However, working on such systems without diligence tips the scales toward exhausting. If your testing and your documentation and your communication (both internal and with consumers) suck, you're in for a world of pain.

"But writing documentation is a waste of time because the code evolves so fast."

Yeah, I hear that, but there's also a lot of time lost to people harried during their on-call and still exhausted for a week afterward, to training new people because the old ones burned out or just left for greener pastures, to maintaining old failed experiments because customers (perhaps at your insistence) still rely on them and backing them out would be almost as much work than adding them was, and so on.

That's not really moving fast. That's just flailing. You can actually go further faster if you maintain a bit of discipline. Yes, there will still be some "wasted" time, but it'll be a bounded, controlled waste like the ablative tiles on a re-entry vehicle - not the uncontrolled explosion of complexity and effort that seems common in many of the younger orgs building/maintaining such systems nowadays.

Re: Ask HN: Do you find working on large distributed systems exhausting?

#75

Post by pojzon »

aeyes wrote: post #65
fxtentacle wrote: post #18

Yes, I used to, but No, I fixed it :) Among other things, I am team lead for a private search engine whose partner-accessible API handles roughly 500 mio requests per month. I used to feel powerless and stressed out by the complexity and the scale, because whenever stuff broke (and it always does at this scale), I had to start playing politics, asking for favors, or threatening people on the phone to get it fixed. Hi…

I feel that your problems aren't even remotely related to my problems with large distributed systems. My problems are all about convincing the company that I need 200 engineers to work on extremely large software projects before we hit a scalability wall. That wall might be 2 years in the future so usually it is next to impossible to convince anyone to take engineers out of product development. Even more so because w…

Had that issue in my previous job.

Higher management decided to migrate our properitary vendor locked platform from one cloud provider to the other one. Majority of migration fell on a single platform team that was constantly struggling with attricion.

Unfortunately I was not able (neither our architects) to explain the higherups that we need bigger team and overall way more resources to pull that off.

Hope that someone that comes after me will be able to make the miracle happen.

Re: Ask HN: Do you find working on large distributed systems exhausting?

#76

Post by 0x445442 »

jerjerjer wrote: post #56
ckdarby wrote: post #53

Earlier quoted context omitted.

>I used to feel powerless and stressed out by the complexity and the scale, because whenever stuff broke (and it always does at this scale), I had to start playing politics, asking for favors, or threatening people on the phone to get it fixed. Higher management would hold me accountable for the downtime even when the whole S3 AZ was offline and there was clearly nothing I could do except for hoping that we'll someho…

You never hosted on AWS, did you?

In the two years I worked on serverless AWS I filed four support tickets. Three out of those four I came up with the solution or fix on my own before support could find a solution. The other ticket was still open when I left the company. But the best part is when support wanted to know how I resolved the issues. I always asked how much they were going to pay me for that information.

Re: Ask HN: Do you find working on large distributed systems exhausting?

#77

Post by ahelwer »

grogers wrote: post #72
solatic wrote: post #9

What do you find exhausting? One anti-pattern I've found is that most orgs ask a single team to handle on-call around the clock for their service. This rarely scales well, from a human standpoint. If you're getting paged at 2:00 in the morning on a regular basis you will start to resent it. There's not much you can do about that so long as only one team is responsible for uptime 24/7. The solution is to hire operatio…

The solution to get paged at off hours a lot is rarely to hire additional teams to cover those times for you, at least not long term. For things you can control, you should fix the root causes of those issues. For things you can't control you should spend effort on making them within your control (eg architecture improvement). This takes time, so follow-the-sun rotation might be a stop gap solution, but you need to m…

From experience, it's really hard to fix the root causes of issues when you were woken up three times the night before and had two more of the same incident occur during the workday. In my case I struggled along for a couple years but the best thing to do was just leave and let it be someone else's problem.

Re: Ask HN: Do you find working on large distributed systems exhausting?

#78

Post by benjamir »

fxtentacle wrote: post #73
nostrebored wrote: post #38

Earlier quoted context omitted.

You had problems with management of a cloud based api and executive visibility… so you bought a set of data centers to handle 500mio req per month? The visibility you will get after the capex when there’s a truly disastrous outage will be interesting.

You might be surprised. The performance equivalent of $100k monthly in EC2 spend fits into a 16m2 cage with 52HU racks.

Which costs you more than $100k monthly to operate with the same level of manageability and reliability.

We don't use AWS, because our use cases don't require that level of reliability and we simply cannot afford it, but if I needed a company to depend on IT that generates enough revenue... I probably wouldn't argue about the AWS bill. So long, prepaid at hetzner + in-house works good enough, but I know what I cannot offer with the click of a button to my user!

Re: Ask HN: Do you find working on large distributed systems exhausting?

#79

Post by smoyer »

Using micro-services instead of monoliths is a great way for software engineers to reduce the complexities of their code. Unfortunately, it moves the complexity to operations. In an organization with a DevOps culture, the software engineers still share responsibility for resolving issues that occur between their micro-service and others.

In other organizations, individual teams have ICDs and SLAs for one or more micro-services and can therefore state they're meeting their interface requirements as well as capacity/uptime requirements. In these organizations, when a system problem occurs, someone who's less familiar with the internals of these services will have to debug complex interactions. In my experience, once the root-cause is identified, there will be one or more teams who get updated requirements - why not make them stakeholders at the system-level and expedite the process?

Re: Ask HN: Do you find working on large distributed systems exhausting?

#80

Post by Ozzie_osman »

solatic wrote: post #9

What do you find exhausting? One anti-pattern I've found is that most orgs ask a single team to handle on-call around the clock for their service. This rarely scales well, from a human standpoint. If you're getting paged at 2:00 in the morning on a regular basis you will start to resent it. There's not much you can do about that so long as only one team is responsible for uptime 24/7. The solution is to hire operatio…

I'd argue that timezone is just part of the problem. If you're responsible for a high oncall load, you are subjected to a steady, unpredictable stream of interrupts requiring you to act to minimize downtime or degradation. Obviously it's worse if you get these at night, but it's still bad during the day.

I think the anti-pattern is having one team responsible for another's burden. You want teams to both be responsible for fixing their own systems when they break, AND be empowered to build/fix their broken systems to minimize oncall incidents.

Post reply on HN