Live data from Hacker News

Ask HN: Do you find working on large distributed systems exhausting?

news.ycombinator.com

51–60 of 261 posts

Re: Ask HN: Do you find working on large distributed systems exhausting?

#51
post #37

I've found that external tech requirements are horrible to work with, especially when the underlying stack simply doesn't support it. Normally these are pushed by certified cloud consultants or by an intrepid architect who found another "best practice blog." It's begins with small requirements such as coming up with a disaster recovery plan only for it to be rejected because your stack must "automatically heal" and d…

...how is the JIRA server mission critical? is it tied to CI/CD somehow?

In the enterprise you'll find that Jira is used for general workflow management not just CICD. I've encountered teams of analysts spend their working day moving and editing work items. It's the Quicken of workflow management solutions.

Jira Server is deliberately cobbled by the sequence table + no Aurora support and now EOL (no security updates 1 year after purchase!). DC edition scales horizontally if you have 100k.

Jira in general is a poorly thought out product (looking at you customfield_3726!) but it's held in such a high regard by users it's impossible to avoid.

Re: Ask HN: Do you find working on large distributed systems exhausting?

#52

Yes, I used to, but No, I fixed it :) Among other things, I am team lead for a private search engine whose partner-accessible API handles roughly 500 mio requests per month. I used to feel powerless and stressed out by the complexity and the scale, because whenever stuff broke (and it always does at this scale), I had to start playing politics, asking for favors, or threatening people on the phone to get it fixed. Hi…

You had problems with management of a cloud based api and executive visibility… so you bought a set of data centers to handle 500mio req per month? The visibility you will get after the capex when there’s a truly disastrous outage will be interesting.

Hmm that’s only 190Hz on average, but we don’t know what kind of search engine it is. For example if he’s doing ML inference for every query, it would make perfect sense to get a few cabinets at a data center. I’ve done so for a much smaller project that only needs 4 GPUs and saved a ton of money.

Re: Ask HN: Do you find working on large distributed systems exhausting?

#53

Yes, I used to, but No, I fixed it :) Among other things, I am team lead for a private search engine whose partner-accessible API handles roughly 500 mio requests per month. I used to feel powerless and stressed out by the complexity and the scale, because whenever stuff broke (and it always does at this scale), I had to start playing politics, asking for favors, or threatening people on the phone to get it fixed. Hi…

>I used to feel powerless and stressed out by the complexity and the scale, because whenever stuff broke (and it always does at this scale), I had to start playing politics, asking for favors, or threatening people on the phone to get it fixed. Higher management would hold me accountable for the downtime even when the whole S3 AZ was offline and there was clearly nothing I could do except for hoping that we'll somehow reach one of their support engineers.

If the business can't afford to have downtime then they should be paying for enterprise support. You'll be able to connect to someone in < 10 mins and have dedicated individuals you can reach out to.

Re: Ask HN: Do you find working on large distributed systems exhausting?

#54
post #37

I've found that external tech requirements are horrible to work with, especially when the underlying stack simply doesn't support it. Normally these are pushed by certified cloud consultants or by an intrepid architect who found another "best practice blog." It's begins with small requirements such as coming up with a disaster recovery plan only for it to be rejected because your stack must "automatically heal" and d…

...how is the JIRA server mission critical? is it tied to CI/CD somehow?

Pre covid I would have laughed at this. But now, no one knows what a user story should be unless you can reas it off jira and there are no backups of course.

Re: Ask HN: Do you find working on large distributed systems exhausting?

#55
I have 15 years xp in dev but all of that was in smaller projects and a small team. I recently took a gig in bigger org with a distributed system and on call etc. It's exhausting and information overload. I'll give myself more time to acclimate but if I feel like this still after a year I'm out.

Re: Ask HN: Do you find working on large distributed systems exhausting?

#56
post #53

Yes, I used to, but No, I fixed it :) Among other things, I am team lead for a private search engine whose partner-accessible API handles roughly 500 mio requests per month. I used to feel powerless and stressed out by the complexity and the scale, because whenever stuff broke (and it always does at this scale), I had to start playing politics, asking for favors, or threatening people on the phone to get it fixed. Hi…

>I used to feel powerless and stressed out by the complexity and the scale, because whenever stuff broke (and it always does at this scale), I had to start playing politics, asking for favors, or threatening people on the phone to get it fixed. Higher management would hold me accountable for the downtime even when the whole S3 AZ was offline and there was clearly nothing I could do except for hoping that we'll someho…

You never hosted on AWS, did you?

Re: Ask HN: Do you find working on large distributed systems exhausting?

#57
post #26

It's hard to answer this because you don't specify what exactly you find exhausting. Is it oncall? Deployment? Performance issues? Dealing with different teams? Failures and recovery? The right hand not knowing what the left hand is doing? Too many services? Something else? It's not even clear how big your service is. You mention billions of requests per month. Every 1B requests/month translates to ~400 QPS, which is…

[deleted]

Re: Ask HN: Do you find working on large distributed systems exhausting?

#58
post #53

Yes, I used to, but No, I fixed it :) Among other things, I am team lead for a private search engine whose partner-accessible API handles roughly 500 mio requests per month. I used to feel powerless and stressed out by the complexity and the scale, because whenever stuff broke (and it always does at this scale), I had to start playing politics, asking for favors, or threatening people on the phone to get it fixed. Hi…

>I used to feel powerless and stressed out by the complexity and the scale, because whenever stuff broke (and it always does at this scale), I had to start playing politics, asking for favors, or threatening people on the phone to get it fixed. Higher management would hold me accountable for the downtime even when the whole S3 AZ was offline and there was clearly nothing I could do except for hoping that we'll someho…

>"they should be paying for enterprise support"

This sounds a bit arrogant. I think they found better and overall cheaper solution.

Re: Ask HN: Do you find working on large distributed systems exhausting?

#59
I'm trying to relate this to my experiences. The best I can make of it is that burnout comes from dealing with either the same types of problems, or new problems at a rate that's higher than old problems get resolved.

I've been in those situations. My solution was to ensure that there was enough effort into systematically resolving long-known issues in a way that not only solves them but also reduces the number of new similar issues. If the strategy is instead to perform predominantly firefighting with 'no capacity' available for working on longer term solutions there is no end in sight unless/until you lose users or requests.

I am curious what the split is of problems being related to:

1. error rates, how many 9s per end-user-action, and per service endpoint

2. performance, request (and per-user-action) latency

3. incorrect responses, bugs/bad-data

4. incorrect responses, stale-data

5. any other categories

Another strategy that worked well was not to fix the problems reported but instead fix the problems known. This is like the physicist looking for keys under the streetlamp instead of where they were dropped. Tracing a bug report to a root cause and then fixing it is very time consuming. This of course needs to continue, but if sufficient effort it put to resolving known issues, such as latency or error rates of key endpoints, it can have an overall lifting effect reducing problems in general.

A specific example was how effort into performance was toward average latency for the most frequently used endpoints. I changed the effort instead to reduce the p99 latency of the worst offenders. This made the system more reliable in general and paid off in a trend to fewer problem reports, though it's not easy/possible to directly relate one to the other.

Re: Ask HN: Do you find working on large distributed systems exhausting?

#60
post #46

> We run into very interesting problems due to scale (billions of requests per month for our main public apis) and the large amount of data we deal with. So, if you are handling 10 billion requests per month, that would average out to about 4k per second. Are these API calls data/compute intensive, or is this more pedestrian data like logging or telemetry? Any time I see someone having a rough time with a distributed…

Yes! A single machine can handle tons of traffic in many cases.
Post reply on HN