I find working on single services / components more exhausting.
Ask HN: Do you find working on large distributed systems exhausting?
141–150 of 261 posts
Re: Ask HN: Do you find working on large distributed systems exhausting?
#142Yes, I used to, but No, I fixed it :) Among other things, I am team lead for a private search engine whose partner-accessible API handles roughly 500 mio requests per month. I used to feel powerless and stressed out by the complexity and the scale, because whenever stuff broke (and it always does at this scale), I had to start playing politics, asking for favors, or threatening people on the phone to get it fixed. Hi…
If you rely on public cloud infrastructure, you should understand both the advantages and disadvantages. Seems like your company forgot about the disadvantages.
Re: Ask HN: Do you find working on large distributed systems exhausting?
#143Earlier quoted context omitted.
that cage is a liability, not an asset. How is the networking in that rack? What's its connection to large-scale storage (IE, petabytes, since that's what I work with). What happens if a meteor hits the cage? Etc.
That depends on what contracts you have. You could have multiple of these cages in different locations. Also, 1 PB is only 56 large enterprise HDDs. So you just put storage into the cage, too. But my point wasn't about how precisely the hardware is managed. My point was that with a large cloud, a mid-sized company has effectively NO SUPPORT. So anything that gives you more control is an improvement.
"1 pb is only 56 large enterprise hdds".
umm, what happens when one fails?
With large cloud my startup had excellent support. We negotiated a contract. That's how it works.
Re: Ask HN: Do you find working on large distributed systems exhausting?
#144Earlier quoted context omitted.
As a security guy I HATE the loss of visibility in going to the cloud. Can you duplicate it? Sure. Still not as easily as spanning a trunk and you still have to trust what you’re seeing to an extent.
The visibility I was mentioning in the parent comment was visibility from executives in your business, but I can see how it would be confusing. There are tradeoffs — cloud removes much of the physical security risks and gives you tools to help automated incident detection. Things like serverless functions let you build out security scaffolding pretty easily. But in exchange you do have to give some trust. And I total…
> cloud removes much of the physical security risks
Doesn't cloud increase the physical security risks, rather than decrease/remove?
Re: Ask HN: Do you find working on large distributed systems exhausting?
#145Earlier quoted context omitted.
I feel that your problems aren't even remotely related to my problems with large distributed systems. My problems are all about convincing the company that I need 200 engineers to work on extremely large software projects before we hit a scalability wall. That wall might be 2 years in the future so usually it is next to impossible to convince anyone to take engineers out of product development. Even more so because w…
> Why do large sites like Facebook, Amazon, Twitter and Instagram all essentially look the same after 10 years but some of them now have 10x the amount of engineers? I think they have so much data and so many dependencies between parts of the system that any fundamental change is extremely hard to pull off. They even cut back on features like API access. But I am pretty sure that most of them have rewritten the whole…
> Yeah, it was a dysfunctional environment and I obviously quit
What do you think could management have done better to make it not dysfunctional and have people quitting?
Re: Ask HN: Do you find working on large distributed systems exhausting?
#146What do you find exhausting? One anti-pattern I've found is that most orgs ask a single team to handle on-call around the clock for their service. This rarely scales well, from a human standpoint. If you're getting paged at 2:00 in the morning on a regular basis you will start to resent it. There's not much you can do about that so long as only one team is responsible for uptime 24/7. The solution is to hire operatio…
At the end of the day, there's a human cost to responding to pages, and there's a human cost to collaboration.
Both of those can drive burn out. Personally, I find all that collaboration work very hard and stressful, so I work better in a situation where I get pages for the services I control; but that would change if pages were frequent and mostly related to dependencies outside of my control. It also helps to have been working in organizations that prioritize a working service over features. Getting frequent overnight issues that can't be resolved without third party effort that's not going to happen anytime soon is a major problem that I see reports of in threads like this.
I can also get behind a team that can manage the base operations issues like ram/storage/cpu faults on nodes and networking. The runbooks for handling those issues are usually pretty short and don't need much collaboration.
Re: Ask HN: Do you find working on large distributed systems exhausting?
#147lets say for the argument sake it's 50 billion thats 20k/sec there is zero need to for a fancy setup at this scale
I am not sure you are aware that server load is never linearly distributed. And that's the exact problem OP is talking about. If everybody would get a ticket number and do requests when they're supposed to do them, we wouldn't need load balancers.
This is orthogonal to what causes the pain point. All the pain comes from distributed state and these load levels even if you peak at 800K requests per second you don't need distributed state. So most of this pain is self inflicted.
Re: Ask HN: Do you find working on large distributed systems exhausting?
#148What do you find exhausting? One anti-pattern I've found is that most orgs ask a single team to handle on-call around the clock for their service. This rarely scales well, from a human standpoint. If you're getting paged at 2:00 in the morning on a regular basis you will start to resent it. There's not much you can do about that so long as only one team is responsible for uptime 24/7. The solution is to hire operatio…
This. Absolutely this. Working on large distributed system can be both exhilarating and exhausting. The two often go hand in hand. However, working on such systems without diligence tips the scales toward exhausting. If your testing and your documentation and your communication (both internal and with consumers) suck, you're in for a world of pain. "But writing documentation is a waste of time because the code evolve…
> That's not really moving fast. That's just flailing.
Yes, a million times yes. This is moving me. Where do I find a team that understands this wisdom?
Re: Ask HN: Do you find working on large distributed systems exhausting?
#149Earlier quoted context omitted.
...how is the JIRA server mission critical? is it tied to CI/CD somehow?
Pre covid I would have laughed at this. But now, no one knows what a user story should be unless you can reas it off jira and there are no backups of course.
Gives me a fun idea: a program that randomly deletes items out of your backlog.
Re: Ask HN: Do you find working on large distributed systems exhausting?
#150My experience is that the expectations on what your average engineer should be able to handle has grown enormously during the last 10 years or so. Working both with large distributed systems and medium size monolithic systems I have seen the expectations become a lot higher in both. When I started my career the engineers at our company were assigned a very specific part of the product that they were experts on. Usual…
> None of the previous responsibilities were removed but we now became responsible also for setting up and running the (cloud) infrastructure and deploying our own software On the flipside, in the olden days when one set of people were churning features and another set of people were given a black box to run and be responsible for keep it running, it was very hard to get the damn thing to work reliably and the only r…
> only half of the picture
Can you please share some details about what you think is missing from most "agile"/devops teams?