Ask HN: Do you find working on large distributed systems exhausting? • Hacker News Board

Re: Ask HN: Do you find working on large distributed systems exhausting?

#21

Post by avensec » Sat, Feb 19, 2022, 1:10 PM UTC

Yes, but in a different way. I work in Quality Engineering, and the scope of maturity in testing distributed systems has been exhausting.

Reading other comments from the thread, I see similar frustrations from teams I partner with. How to employ patterns like contact, hypothesis, doubles, or shape/data systems (etc.) typically gets conflated with System testing. Teams often disagree on the boundaries of the system start leaning towards System Testing, and end up adding additional complexity in tests that could be avoided.

My thought is that I see the desire to control more scope presenting itself in test. I typically find myself doing some bounded context exercises to try to hone in on scope early.

Re: Ask HN: Do you find working on large distributed systems exhausting?

#22

Post by eez0 » Sat, Feb 19, 2022, 1:16 PM UTC

I find it actually the other way around.

As you said, a benefit of large distributed systems is that usually its a shared responsibility, with different teams owning different services.

The exhaustion comes into place when those services are not really independent, or when the responsibility is not really shared, which in turn is just a worse version of a typical system maintained by sysadmins.

One thing that helps is bring the DevOps culture into the company, but the right way. It's not just about "oh cool we are now agile and deploy a few times a day", it's all down to shared responsibility.

Re: Ask HN: Do you find working on large distributed systems exhausting?

#23

Post by wilde » Sat, Feb 19, 2022, 1:16 PM UTC

Without more info it’s hard to say. When I felt like this, a manager recommended I start journaling my energy. I kept a Google doc with sections for each week. In each section, there’s a bulleted list of things I did that gave me energy and a list of things I did that took energy.

Once you have a few lists some trends become clear and you can work with your manager to shift where you spend time.

Re: Ask HN: Do you find working on large distributed systems exhausting?

#24

Post by lr4444lr » Sat, Feb 19, 2022, 1:19 PM UTC

Yes. But remember, with tools and automation getting better, this is a major source of value add that you bring as a software engineer which is likely to have long term career viability.

Re: Ask HN: Do you find working on large distributed systems exhausting?

#25

Post by ChrisMarshallNY » Sat, Feb 19, 2022, 1:21 PM UTC

zaphirplane wrote: post #20

ChrisMarshallNY wrote: post #16

TLDR; Yes, it is exhausting, but I have found ways to mitigate it. I don't develop stuff that runs billions of queries. More like thousands. It is, however, important infrastructure, on which thousands of people around the world, rely, and, in some cases, it's not hyperbole to say that lives depend on its integrity and uptime. One fairly unique feature of my work, is that it's almost all "hand-crafted." I generally a…

Front end and no dependencies, tell us more

Feel free to see for yourself. I have quite a few OS projects out there. My GH ID is the same as my HN one.

My frontend work is native Swift work, using the built-in Apple frameworks (I ship classic AppKit/UIKit/WatchKit, using storyboards and MVC, but I will be moving onto the newer stuff, as it matures).

My backend work has chiefly been PHP. It works quite well, but is not where I like to spend most of my time.

Re: Ask HN: Do you find working on large distributed systems exhausting?

#26

Post by jmyeet » Sat, Feb 19, 2022, 1:26 PM UTC

It's hard to answer this because you don't specify what exactly you find exhausting. Is it oncall? Deployment? Performance issues? Dealing with different teams? Failures and recovery? The right hand not knowing what the left hand is doing? Too many services? Something else?

It's not even clear how big your service is. You mention billions of requests per month. Every 1B requests/month translates to ~400 QPS, which isn't even that large. Like, that's single server territory. Obviously spikiness matters. I'd also be curious what you mean by "large amount of data".

Re: Ask HN: Do you find working on large distributed systems exhausting?

#27

Post by Arrezz » Sat, Feb 19, 2022, 1:29 PM UTC

I think our field is so broad that it is somewhat nebolous to talk about the average engineer. But from my experience taking car of such a large system with a large amount of requests and complexity is outside of what is expected of an average engineer. I think that there is an eventual limit for how much complexity a single engineer can handle for several years.

Re: Ask HN: Do you find working on large distributed systems exhausting?

#28

Post by daneel_w » Sat, Feb 19, 2022, 1:30 PM UTC

I find it very draining and vexing to work on systems that have all of its components distributed left and right without clear boundaries, instead of being more coalesced. Distribution in the typical sense - identical spares working in parallel for the sake of redundancy - doesn't faze me very much.

Re: Ask HN: Do you find working on large distributed systems exhausting?

#29

Post by gorgoiler » Sat, Feb 19, 2022, 1:31 PM UTC

My number one requirement for a distributed system is that the code all be one place.

There are good reasons for wanting multiple services talking through APIs. Perhaps you have a Linux scheduler that is marshalling test suites running on Android, Windows, macOS and iOS?

If all these systems originate from a single repository, preferably with the top level written in a dynamic language that runs from its own source code, then life can be much easier. Being able to change multiple parts of the infrastructure in a single commit is a powerful proposition.

You also stand a chance of being able to model your distributed system locally, maybe even in a single Python process, which can help when you want to test new infrastructure ideas without needing the whole distributed environment.

Your development velocity will be faster and less painless. Changes being slow and painful are what burn people out and grind progress to a halt.

Re: Ask HN: Do you find working on large distributed systems exhausting?

#30

Post by thebackup » Sat, Feb 19, 2022, 1:34 PM UTC

My experience is that the expectations on what your average engineer should be able to handle has grown enormously during the last 10 years or so. Working both with large distributed systems and medium size monolithic systems I have seen the expectations become a lot higher in both.

When I started my career the engineers at our company were assigned a very specific part of the product that they were experts on. Usually there were 1 or 2 engineers assigned to a specific area and they knew it really well. Then we went Agile(tm) and the engineers were grouped into 6 to 9 person teams that were assigned features that spanned several areas of the product. The teams also got involved in customer interaction, planning, testing and documentation. The days when you could focus on a single part of the system and become really good at it were gone.

Next big change came when the teams moved from being feature teams to devops teams. None of the previous responsibilities were removed but we now became responsible also for setting up and running the (cloud) infrastructure and deploying our own software.

In some ways I agree that these changes have empowered us. But it is also, as you say, exhausting. Once I was simply a programmer; now I'm a domain expert, project manager, programmer, tester, technical writer, database admin, operations engineer, and so on.