Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Do you find working on large distributed systems exhausting?

news.ycombinator.com

Post reply on HN
Print view View original

1–10 of 261 posts

Ask HN: Do you find working on large distributed systems exhausting?

#1

Post by wreath »

Ive been working on large distributed system for the last 4-5 years with teams owning few services or have different responsibilities to keep the system up and running. We run into very interesting problems due to scale (billions of requests per month for our main public apis) and the large amount of data we deal with.

I think it has progressed my career and expanded my skills but I feel it's pretty damn exhausting to manage all this even when following a lot of the best-practices and working with other highly skilled engineers.

I've been wondering recently if others feel this kind of burnout (for lack of better word). Is the expectation is that your average engineer should now be able to handle all this?

Re: Ask HN: Do you find working on large distributed systems exhausting?

#3

Post by heisenbit »

In these large scale systems the boundaries are usually not well defined (there are APIs but data flowing through the APIs is another matter as are operational and non functional requirements).

Stress is often caused by a mismatch of what you feel responsible and accountable for and what you really control. The more you know the more you feel responsible for but you are rarely able to expand control as much or as fast as your knowledge. It helps to be very clear about where you have ultimate say (accountability) or control within some framework (responsibility) or simply know and contribute. Clear in your mind, others and your boss. Look at areas outside your responsibility with curiosity and willingness to offer support but know that you are not responsible and others need to worry.

Re: Ask HN: Do you find working on large distributed systems exhausting?

#4

Post by z3t4 »

Often when I hear stories of billions of requests per second it's self inflicted because of over complicated architecture where all those requests are generated only by a few thousand customers... So it's usually a question of how the company operate, do you constantly fight fires ? or do you spend your time implementing stuff that have high value for the company and it's customers ? Fighting fires can get your burned out (no pun intended) while feeling that you deliver a lot of value will make you feel great.

Re: Ask HN: Do you find working on large distributed systems exhausting?

#5

Post by tylerrobinson »

heisenbit wrote: post #3

In these large scale systems the boundaries are usually not well defined (there are APIs but data flowing through the APIs is another matter as are operational and non functional requirements). Stress is often caused by a mismatch of what you feel responsible and accountable for and what you really control. The more you know the more you feel responsible for but you are rarely able to expand control as much or as fas…

This is spot on. Feeling frustrated working on large distributed systems could be generalized as “feeling frustrated working in a large organization” because the same limitations apply. You learn about things you cannot control, and it is important to see the difference between what you can control and contribute and what you can’t.

Re: Ask HN: Do you find working on large distributed systems exhausting?

#6

Post by dudul »

Let's set aside the "distributed" aspect. To effectively scale a team and a code base you need some concept of "modularization" and "ownership". It is unrealistic to expect engineers to know everything about the entire system.

The problem is that this division of the code base is really hard. It is really hard to find the time and the energy to properly section your code base in proper domains and APIs. Especially with the constantly moving target of what needs to be delivered next. Even in a monorepo it is exhausting.

Now, put on top of that the added burden brought by a distributed system (deployment, protocol, network issues, etc) and you have something that becomes even more taxing on your energy.

Re: Ask HN: Do you find working on large distributed systems exhausting?

#7

Post by readingnews »

alecbz wrote: post #2

Can you say more? What specifically is exhausting? Exhaustion/burnout isn't uncommon but without more context it's hard to say if it's a product of the type of work or your specific work environment.

This is on point... You also give no actual numerical context. Are you saying you are working 40 hours a week and leave work exhausted? Are you saying you work 40 at work, and are on call/email/remote terminals for 40 more hours coordinating teams, putting out fires, designing architecture?

Even then, I would ask you to be more specific. I have a normal 40 hour a week uni job as a sysadmin, but it typically takes somewhat more or less (hey, sometimes I can get it done in 35, sometimes its 50 hours). However, for the last several years we have been so shorthanded, faculty wise, that I teach (at a minimum) two senior level computer science classes every semester (I was a professor at another uni). About mid semester, things will break, professors will make unreasonable demands of building out new systems/software/architecture, and I find myself doing (again at a minimum) 80 hours a week. On the other hand, I am not exhausted, as I enjoy teaching quite a bit, and I have been a sysadmin for many years and also enjoy that work.

Re: Ask HN: Do you find working on large distributed systems exhausting?

#8

Post by ebbp »

It’d be interesting to know - what are the expectations made of you? In this environment, I’d expect there to be dedicated support for teams operating their services - i.e. SRE/DevOps/Platform teams who should be looking to abstract away some of the raw edges of operating at scale.

That said, I do think there’s a psychological overhead when working on something that serves high levels of production traffic. The stakes are higher (or at least, they feel that way), which can affect different people in different ways. I definitely recognise your feeling of exhaustion, but I wonder if it maybe comes from a lack of feeling “safe” when you deploy - either from insufficient automated testing or something else.

(For context - I’m an SRE who has worked in quite a few places exactly like this)

Re: Ask HN: Do you find working on large distributed systems exhausting?

#9

Post by solatic »

What do you find exhausting?

One anti-pattern I've found is that most orgs ask a single team to handle on-call around the clock for their service. This rarely scales well, from a human standpoint. If you're getting paged at 2:00 in the morning on a regular basis you will start to resent it. There's not much you can do about that so long as only one team is responsible for uptime 24/7.

The solution is to hire operations teams globally, and then setup follow-the-sun operations whereby the people being paged are always naturally awake at that hour, and allows them to work normal eight hour shifts. But this requires companies to, gasp, have specialized developers and specialized operators collaborate before allowing new feature work into production, to ensure that the operations teams understand what the services are supposed to do and keep it all online. It requires (oh, the horror!) actually maintaining production standards, runbooks, and other documentation.

So naturally, many orgs would prefer to burn out their engineers instead.

Re: Ask HN: Do you find working on large distributed systems exhausting?

#10

Post by seanwilson »

It's okay to prefer working on small single server systems with small teams for example. I do this while contracting quite often and enjoy how much control you get to make big changes with minimal bureaucracy.

Sometimes it feels like everyone is focused on eventually working with Google scale systems and following best practices that are more relevant towards that scale but you can pick your own path.

Post reply on HN