Yes. That's why you avoid building them unless you absolutely need to, and instead build libraries instead.
Ask HN: Do you find working on large distributed systems exhausting?
121–130 of 261 posts
Re: Ask HN: Do you find working on large distributed systems exhausting?
#122Earlier quoted context omitted.
The solution to get paged at off hours a lot is rarely to hire additional teams to cover those times for you, at least not long term. For things you can control, you should fix the root causes of those issues. For things you can't control you should spend effort on making them within your control (eg architecture improvement). This takes time, so follow-the-sun rotation might be a stop gap solution, but you need to m…
From experience, it's really hard to fix the root causes of issues when you were woken up three times the night before and had two more of the same incident occur during the workday. In my case I struggled along for a couple years but the best thing to do was just leave and let it be someone else's problem.
Best thing for what? Surely not software quality and customer satisfaction.
Re: Ask HN: Do you find working on large distributed systems exhausting?
#123Yes, I used to, but No, I fixed it :) Among other things, I am team lead for a private search engine whose partner-accessible API handles roughly 500 mio requests per month. I used to feel powerless and stressed out by the complexity and the scale, because whenever stuff broke (and it always does at this scale), I had to start playing politics, asking for favors, or threatening people on the phone to get it fixed. Hi…
What i read here was "Cloud is hard, so I took on even more responsibility"
What you should read is: At the monthly spend of a mid-sized company, it is impossible to get phone support from any public cloud provider.
Re: Ask HN: Do you find working on large distributed systems exhausting?
#124[deleted]
Re: Ask HN: Do you find working on large distributed systems exhausting?
#125Earlier quoted context omitted.
From experience, it's really hard to fix the root causes of issues when you were woken up three times the night before and had two more of the same incident occur during the workday. In my case I struggled along for a couple years but the best thing to do was just leave and let it be someone else's problem.
Best thing for what? Surely not software quality and customer satisfaction.
If they cared about that they would either pay me so much money I'd be insane to walk away or they would hire people in other time zones to cover the load. Instead they chose to pay for their customer satisfaction with my burnout. The thing about that strategy is... eventually the thing holding their customer satisfaction together gets burnt out. So I leave. And even then they're still getting the better half of the bargain.
Re: Ask HN: Do you find working on large distributed systems exhausting?
#126Using micro-services instead of monoliths is a great way for software engineers to reduce the complexities of their code. Unfortunately, it moves the complexity to operations. In an organization with a DevOps culture, the software engineers still share responsibility for resolving issues that occur between their micro-service and others. In other organizations, individual teams have ICDs and SLAs for one or more micr…
> Using micro-services instead of monoliths is a great way for software engineers to reduce the complexities of their code
Could you share why you think that's true?
IMO that it's exactly the opposite - microservices have potential to simplify operations and processes (smaller artifacts, independent development/deployments, isolation, architectural boundaries easier to enforce) but when it comes to code and their internal architecture - they are always more complex.
If you take microservices and merge them into a monolith - it will still work, you don't need to add code or increase complexity. You actually can remove code - anything related to network calls, data replication between components if they share a DB, etc.
Re: Ask HN: Do you find working on large distributed systems exhausting?
#127Earlier quoted context omitted.
>"they should be paying for enterprise support" This sounds a bit arrogant. I think they found better and overall cheaper solution.
>This sounds a bit arrogant. The parent thread talks about how the business could not go down even with a triple AZ outage for S3, and I don't think it is arrogant to state they should be paying for enterprise support if that level of expectation is set. >I think they found better and overall cheaper solution. Cheaper solution does not just include the cost but also the time. For the time we need to look at the time…
Back then, it was enough to saturate the S3 metadata node for your bucket and then all AZs would be unable to service GET requests.
And yes, this won't be financially useful in every situation. But if the goal is to gain operational control, it's worthwhile nonetheless. That said, for a high-traffic API, you're paying through the nose for AWS egress bandwidth, so it is one of those cases where it also very much makes financial sense.
Re: Ask HN: Do you find working on large distributed systems exhausting?
#128Handling scale is a technically challenging problem, if you enjoy it - then take advantage! however sometimes taking a break to work on something else can be more satisfying.
Typically on a "High scale" service spanning hundreds or thousands of servers you'll have to deal with problems like. "How much memory does this object consume?", "how many ms will adding this regex/class into the critical path use?", "We need to add new integ/load/unit tests for X to prevent outage Y from recurring", and "I wish I could try new technique Y, but I have 90% of my time occupied on upkeep".
It can be immensely satisfying to flip to a low/scale, low/ops problem space and find that you can actually bang out 10x the features/impact when you're not held back by scale.
Source: Worked on stateful services handling 10 Million TPS, took a break to work on internal analytics tools and production ML modeling, transitioning back to high scale services shortly.
Re: Ask HN: Do you find working on large distributed systems exhausting?
#129Relevant comedy video:
https://www.youtube.com/watch?v=y8OnoxKotPQ
This recent video they put out is pretty good, too:
Re: Ask HN: Do you find working on large distributed systems exhausting?
#130I don't find it exhausting, I find it *exhilarating*. After years of proving myself, earning trust and strategical positioning I am finally leading a system that will support millions of requests per second. I love my job and this is the most intellectually stimulating activity I have done in a long while. I think this is far from the expectation of the average engineer. You can find many random companies with very m…
> I am finally leading a system that will support millions of requests per second.
This is the difference. Millions of things per second is a super hard problem to get right in any reality. Pulling this off with any technology at all is rewarding.
Most distributed systems are not facing this degree of realistic challenge. In most shops, the challenge is synthetic and self-inflicted. For whatever reason, people seem to think saying things like "we do billions of x per month" somehow justifies their perverse architectures.