Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Do you find working on large distributed systems exhausting?

news.ycombinator.com

Post reply on HN
Print view View original

61–70 of 261 posts

Re: Ask HN: Do you find working on large distributed systems exhausting?

#61

Post by phillu »

jerjerjer wrote: post #56
ckdarby wrote: post #53

Earlier quoted context omitted.

>I used to feel powerless and stressed out by the complexity and the scale, because whenever stuff broke (and it always does at this scale), I had to start playing politics, asking for favors, or threatening people on the phone to get it fixed. Higher management would hold me accountable for the downtime even when the whole S3 AZ was offline and there was clearly nothing I could do except for hoping that we'll someho…

You never hosted on AWS, did you?

Enterprise Support never disappointed me so far. Maybe not <10 minute response time, but we never felt left alone during an outage. But I guess this is also highly region/geo dependent.

Re: Ask HN: Do you find working on large distributed systems exhausting?

#62

Post by mistaPockets »

jmyeet wrote: post #26

It's hard to answer this because you don't specify what exactly you find exhausting. Is it oncall? Deployment? Performance issues? Dealing with different teams? Failures and recovery? The right hand not knowing what the left hand is doing? Too many services? Something else? It's not even clear how big your service is. You mention billions of requests per month. Every 1B requests/month translates to ~400 QPS, which is…

No post body was provided.

Re: Ask HN: Do you find working on large distributed systems exhausting?

#63

Post by harshaw »

ublaze wrote: post #15

Yeah, large-scale systems are often boring in my experience, because the scale limits what features you can add to make things better. Each and every decision has to take scale into account, and it's tricky to try experimenting. I think it has to do with the kind of engineer you are. Some engineers love iterating and improving such systems to be more efficient, more scalable, etc. But it can be limiting due to the sl…

I don't think they are boring, but very important on the kind of engineer you are. At AWS I try to encourage people who like the problem space and at the very least appreciate it, but can totally understand that you don't want to do your entire career on it. Many of our younger folks have never felt the speed and joy you can get with hammering out a simple app (web, python, ML) that doesn't have to work at scale.

Re: Ask HN: Do you find working on large distributed systems exhausting?

#64

Post by cookiengineer »

qaq wrote: post #17

lets say for the argument sake it's 50 billion thats 20k/sec there is zero need to for a fancy setup at this scale

I am not sure you are aware that server load is never linearly distributed. And that's the exact problem OP is talking about.

If everybody would get a ticket number and do requests when they're supposed to do them, we wouldn't need load balancers.

Re: Ask HN: Do you find working on large distributed systems exhausting?

#65

Post by aeyes »

fxtentacle wrote: post #18

Yes, I used to, but No, I fixed it :) Among other things, I am team lead for a private search engine whose partner-accessible API handles roughly 500 mio requests per month. I used to feel powerless and stressed out by the complexity and the scale, because whenever stuff broke (and it always does at this scale), I had to start playing politics, asking for favors, or threatening people on the phone to get it fixed. Hi…

I feel that your problems aren't even remotely related to my problems with large distributed systems.

My problems are all about convincing the company that I need 200 engineers to work on extremely large software projects before we hit a scalability wall. That wall might be 2 years in the future so usually it is next to impossible to convince anyone to take engineers out of product development. Even more so because working on this changes absolutely nothing for the end user, it is usually some internal system related to data storage or processing which can't cope anymore.

Imagine that you are Amazon and for some scalability reason you have to rewrite the storage layer of your product catalog. Immediately you have a million problems like data migration, reporting, data ingestion, making it work with all the related systems like search, recommendations, reviews and so on.

And even if you get the ball rolling you have to work across dozens of different teams which can be hard because naturally people resist change.

Why do large sites like Facebook, Amazon, Twitter and Instagram all essentially look the same after 10 years but some of them now have 10x the amount of engineers? I think they have so much data and so many dependencies between parts of the system that any fundamental change is extremely hard to pull off. They even cut back on features like API access. But I am pretty sure that most of them have rewritten the whole thing at least 3 times.

Re: Ask HN: Do you find working on large distributed systems exhausting?

#66

Post by ckdarby »

jerjerjer wrote: post #56
ckdarby wrote: post #53

Earlier quoted context omitted.

>I used to feel powerless and stressed out by the complexity and the scale, because whenever stuff broke (and it always does at this scale), I had to start playing politics, asking for favors, or threatening people on the phone to get it fixed. Higher management would hold me accountable for the downtime even when the whole S3 AZ was offline and there was clearly nothing I could do except for hoping that we'll someho…

You never hosted on AWS, did you?

>You never hosted on AWS, did you?

Previously 2k employee company, with the entire advertising back office on AWS.

Currently >$1M YR at AWS, you can get the idea of scale & what is running, here: https://www.youtube.com/playlist?list=PLf-67McbxkT6iduMWoUsh...

Re: Ask HN: Do you find working on large distributed systems exhausting?

#67

Post by ckdarby »

ddorian43 wrote: post #36
fxtentacle wrote: post #18

Yes, I used to, but No, I fixed it :) Among other things, I am team lead for a private search engine whose partner-accessible API handles roughly 500 mio requests per month. I used to feel powerless and stressed out by the complexity and the scale, because whenever stuff broke (and it always does at this scale), I had to start playing politics, asking for favors, or threatening people on the phone to get it fixed. Hi…

What are you using for aws alternatives? Example for S3?

>What are you using for aws alternatives? Example for S3?

Not OP but they're probably using Rook/Minio

Re: Ask HN: Do you find working on large distributed systems exhausting?

#68

Post by sillysaurusx »

Jobs aren’t exhausting. Teams are. If you find yourself feeling this way, consider that the higher ups may be mismanaging.

There’s often not a lot of organizational pressure to change anything. So the status quo stays static. But the services change over time, so the status quo needs to change with them.

Re: Ask HN: Do you find working on large distributed systems exhausting?

#69

Post by andai »

aeyes wrote: post #65
fxtentacle wrote: post #18

Yes, I used to, but No, I fixed it :) Among other things, I am team lead for a private search engine whose partner-accessible API handles roughly 500 mio requests per month. I used to feel powerless and stressed out by the complexity and the scale, because whenever stuff broke (and it always does at this scale), I had to start playing politics, asking for favors, or threatening people on the phone to get it fixed. Hi…

I feel that your problems aren't even remotely related to my problems with large distributed systems. My problems are all about convincing the company that I need 200 engineers to work on extremely large software projects before we hit a scalability wall. That wall might be 2 years in the future so usually it is next to impossible to convince anyone to take engineers out of product development. Even more so because w…

Heh, I wish they still looked the same. They added an order of magnitude of HTML and JS bloat while removing functionality.

Re: Ask HN: Do you find working on large distributed systems exhausting?

#70

Post by throwhauser »

m_herrlich wrote: post #48

I love building and developing software, and despite the fun and interesting challenges presented at my last job I quit because of the operations component. We adopted DevOps and it felt like "building" got replaced with "configuring" and managing complex configurations does not tickle my brain at all. Week-long on-call shifts are like being under house arrest 24/7. I understand the value that developers bring to ope…

After-hours on-call is a thing that needs to be destroyed. A company that is sufficiently large that the CEO doesn't get woken up for emergencies needs to have shifts in other timezones to handle them. I don't know why people put up with it.
Post reply on HN