Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Do you find working on large distributed systems exhausting?

news.ycombinator.com

Post reply on HN
Print view View original

161–170 of 261 posts

Re: Ask HN: Do you find working on large distributed systems exhausting?

#161

Post by dasil003 »

aeyes wrote: post #65
fxtentacle wrote: post #18

Yes, I used to, but No, I fixed it :) Among other things, I am team lead for a private search engine whose partner-accessible API handles roughly 500 mio requests per month. I used to feel powerless and stressed out by the complexity and the scale, because whenever stuff broke (and it always does at this scale), I had to start playing politics, asking for favors, or threatening people on the phone to get it fixed. Hi…

I feel that your problems aren't even remotely related to my problems with large distributed systems. My problems are all about convincing the company that I need 200 engineers to work on extremely large software projects before we hit a scalability wall. That wall might be 2 years in the future so usually it is next to impossible to convince anyone to take engineers out of product development. Even more so because w…

I don't know your specifics, but I have worked on some large scale architecture changes, and 200 engineers + 2 year feature freeze is generally not a reasonable ask. In practice you need to find an incremental path with validation and course correction along the way to limit the amount of concurrent change in flight at any moment. If you don't do this run a very high risk of the entire initiative collapsing under its own weight.

Assuming your estimation is more or less correct and it really is a 400 eng-year project, then you also need political capital as well as technical leadership to make it happen. There are lots of companies where a smart engineer can see a potential path out of a local maximum, but the org structure and lack of technical leadership in the highest ranks means that the problem is effectively intractable.

Re: Ask HN: Do you find working on large distributed systems exhausting?

#162

Post by kqr »

NationalPark wrote: post #120
solatic wrote: post #9

What do you find exhausting? One anti-pattern I've found is that most orgs ask a single team to handle on-call around the clock for their service. This rarely scales well, from a human standpoint. If you're getting paged at 2:00 in the morning on a regular basis you will start to resent it. There's not much you can do about that so long as only one team is responsible for uptime 24/7. The solution is to hire operatio…

I don’t think this is a stable long term solution. The “on call” teams end up frustrated with the engineers who ship bugs and this results in added process that delays deploys, arbitrary demands for test coverage, capricious error budgets, etc. It’s much better to have the engineers who wrote the code be responsible for running it, and if their operational burden becomes too high, to staff up the dev team to empower…

Correct. Throwing software over the wall to "other people" and letting them deal with the problems of running the software is guaranteed to lead to low quality, inefficient processes, or usually both.

Re: Ask HN: Do you find working on large distributed systems exhausting?

#163

Post by FpUser »

ckdarby wrote: post #82
FpUser wrote: post #58

Earlier quoted context omitted.

>"they should be paying for enterprise support" This sounds a bit arrogant. I think they found better and overall cheaper solution.

>This sounds a bit arrogant. The parent thread talks about how the business could not go down even with a triple AZ outage for S3, and I don't think it is arrogant to state they should be paying for enterprise support if that level of expectation is set. >I think they found better and overall cheaper solution. Cheaper solution does not just include the cost but also the time. For the time we need to look at the time…

So you basically saying that no matter what one should always stick to Amazon. I have my own experience that tells exactly the opposite. To each their own. We do not have to agree.

Re: Ask HN: Do you find working on large distributed systems exhausting?

#164

Post by [deleted] »

fxtentacle wrote: post #111
aeyes wrote: post #65

Earlier quoted context omitted.

I feel that your problems aren't even remotely related to my problems with large distributed systems. My problems are all about convincing the company that I need 200 engineers to work on extremely large software projects before we hit a scalability wall. That wall might be 2 years in the future so usually it is next to impossible to convince anyone to take engineers out of product development. Even more so because w…

My impression has always been that FAANG need lots of engineers because the 10xers refuse to work there. I've seen plenty of really scalable systems being built by a small core team of people who know what they are doing. FAANG instead seem to be more into chasing trends, inventing new frameworks, rewriting to another more hip language, etc. I would have no idea how to coordinate 200 engineers. But then again, I have…

[deleted]

Re: Ask HN: Do you find working on large distributed systems exhausting?

#165

Post by kqr »

ahelwer wrote: post #125
kqr wrote: post #122

Earlier quoted context omitted.

Best thing for what? Surely not software quality and customer satisfaction.

If they cared about that they would either pay me so much money I'd be insane to walk away or they would hire people in other time zones to cover the load. Instead they chose to pay for their customer satisfaction with my burnout. The thing about that strategy is... eventually the thing holding their customer satisfaction together gets burnt out. So I leave. And even then they're still getting the better half of the…

Sorry, I accidentally said you did the wrong thing for leaving. That wasn't my intention. Of course, leaving was the right choice for you.

What I meant was the company you were working for does not get the best quality or customer satisfaction by overworking you to the point where you have to leave. It would have been better for their software quality to handle things differently.

Re: Ask HN: Do you find working on large distributed systems exhausting?

#167

Post by fxtentacle »

winrid wrote: post #158
fxtentacle wrote: post #133

Earlier quoted context omitted.

Nah, it's text-only requests returning JSON arrays of which newspaper article URLs mention which influencer or brand name keyword. The biggest hardware price point is that you need insane amounts of RAM so that you can mmap the bloom hash for the mapping from word_id to document_ids.

You could have used a sharded database like Mongo. Just throw up 10 shards, use "source" (influencer or brand name) as shard key?

Yes, I could have used Mongo, but it would have been 100x to 1000x slower than an mmap-ed look up table.

Re: Ask HN: Do you find working on large distributed systems exhausting?

#168

Post by asymmetric »

softwarebeware wrote: post #116
hliyan wrote: post #35

The first ten years of my career, I worked with distributed systems built on this stack: C++, Oracle, Unix (and to some extent, MFC and Qt). There were hundreds of instances of dozens of different type of processes (we would now call these microservices) connected via TCP links, running on hundreds of servers. I seldom found this exhausting. The second ten years of my career, I worked with (and continue to work on) m…

You know what? This is a really great point. When I reflect back on my career experience (at companies like Expedia, eBay, Zillow, etc.) the best distributed systems experience I had was at companies that standardized on languages and frameworks and drew a pretty strong boundary around those choices. It wasn't that you technically couldn't choose another stack for a project, but to do so you had to justify the cost/b…

Reminds me of http://boringtechnology.club/

Re: Ask HN: Do you find working on large distributed systems exhausting?

#169

Post by fxtentacle »

dekhn wrote: post #143
fxtentacle wrote: post #106

Earlier quoted context omitted.

That depends on what contracts you have. You could have multiple of these cages in different locations. Also, 1 PB is only 56 large enterprise HDDs. So you just put storage into the cage, too. But my point wasn't about how precisely the hardware is managed. My point was that with a large cloud, a mid-sized company has effectively NO SUPPORT. So anything that gives you more control is an improvement.

"1 pb is only 56 large enterprise hdds". umm, what happens when one fails? With large cloud my startup had excellent support. We negotiated a contract. That's how it works.

Typically people use RAID or ZFS to prevent data loss when a few hdds fail.

Re: Ask HN: Do you find working on large distributed systems exhausting?

#170

Post by dekhn »

fxtentacle wrote: post #169
dekhn wrote: post #143

Earlier quoted context omitted.

"1 pb is only 56 large enterprise hdds". umm, what happens when one fails? With large cloud my startup had excellent support. We negotiated a contract. That's how it works.

Typically people use RAID or ZFS to prevent data loss when a few hdds fail.

OK, so basically you're in a completely different class of expectations about how systems perform under disk loss and heavy load then me. A drive array is very different from large-scale cloud storage.
Post reply on HN