Yes the complexity and scale of these systems is far beyond what companies understand. The salaries of engineers on these systems need to double asap or they risk collapse.
Ask HN: Do you find working on large distributed systems exhausting?
151–160 of 261 posts
Re: Ask HN: Do you find working on large distributed systems exhausting?
#152I love building and developing software, and despite the fun and interesting challenges presented at my last job I quit because of the operations component. We adopted DevOps and it felt like "building" got replaced with "configuring" and managing complex configurations does not tickle my brain at all. Week-long on-call shifts are like being under house arrest 24/7. I understand the value that developers bring to ope…
After-hours on-call is a thing that needs to be destroyed. A company that is sufficiently large that the CEO doesn't get woken up for emergencies needs to have shifts in other timezones to handle them. I don't know why people put up with it.
Part of it is a culture that discourages complaining about after hours work.
There's an expectation that everyone is a night owl and that night time emergency work is fun, and that these fires are to be expected.
Finally, engineers seem to get this feeling of being important because they wake up and work at night. It's really a form of insanity.
Re: Ask HN: Do you find working on large distributed systems exhausting?
#153Earlier quoted context omitted.
> None of the previous responsibilities were removed but we now became responsible also for setting up and running the (cloud) infrastructure and deploying our own software On the flipside, in the olden days when one set of people were churning features and another set of people were given a black box to run and be responsible for keep it running, it was very hard to get the damn thing to work reliably and the only r…
> only half of the picture Can you please share some details about what you think is missing from most "agile"/devops teams?
Proper staffing
Re: Ask HN: Do you find working on large distributed systems exhausting?
#154Earlier quoted context omitted.
> Why do large sites like Facebook, Amazon, Twitter and Instagram all essentially look the same after 10 years but some of them now have 10x the amount of engineers? I think they have so much data and so many dependencies between parts of the system that any fundamental change is extremely hard to pull off. They even cut back on features like API access. But I am pretty sure that most of them have rewritten the whole…
> Yeah, it was a dysfunctional environment and I obviously quit What do you think could management have done better to make it not dysfunctional and have people quitting?
[deleted]
Re: Ask HN: Do you find working on large distributed systems exhausting?
#155Worked on a team at BofA, our application would handle 800 million events per day. The logic we had for retry and failure was solid. We also had redundancy across multiple DCs. I think we processed like 99.9999999% of all events successfully. (Basically all of them, last year we lost about 2,000 events total) I didn’t find it very stressful at all. We build in JMX Utica for our production support teams be able to han…
[deleted]
Re: Ask HN: Do you find working on large distributed systems exhausting?
#156Worked on a team at BofA, our application would handle 800 million events per day. The logic we had for retry and failure was solid. We also had redundancy across multiple DCs. I think we processed like 99.9999999% of all events successfully. (Basically all of them, last year we lost about 2,000 events total) I didn’t find it very stressful at all. We build in JMX Utica for our production support teams be able to han…
Utils*
Re: Ask HN: Do you find working on large distributed systems exhausting?
#157Earlier quoted context omitted.
> only half of the picture Can you please share some details about what you think is missing from most "agile"/devops teams?
Proper staffing
Ah excellent. Yes. In my experience there's this idea of "scale at all costs"--a better way would probably be to limit scaling until the headcount is scaled. Although then you probably need more VC money.
Re: Ask HN: Do you find working on large distributed systems exhausting?
#158Earlier quoted context omitted.
Hmm that’s only 190Hz on average, but we don’t know what kind of search engine it is. For example if he’s doing ML inference for every query, it would make perfect sense to get a few cabinets at a data center. I’ve done so for a much smaller project that only needs 4 GPUs and saved a ton of money.
Nah, it's text-only requests returning JSON arrays of which newspaper article URLs mention which influencer or brand name keyword. The biggest hardware price point is that you need insane amounts of RAM so that you can mmap the bloom hash for the mapping from word_id to document_ids.
You could have used a sharded database like Mongo. Just throw up 10 shards, use "source" (influencer or brand name) as shard key?
Re: Ask HN: Do you find working on large distributed systems exhausting?
#159Earlier quoted context omitted.
"That wall might be 2 years in the future so usually it is next to impossible to convince anyone to take engineers out of product development. Even more so because working on this changes absolutely nothing for the end user" It seems to be the same story in fiels of Infrastructure maintenance, Aircraft design (boeing Max), and mortgage CDOs (2008). Was it always like this or the new management doesn not care untill s…
a manufacturing company is designed the ground up to works whit machine but isn't the same whit software, is hard to understand that triple data isn't only triple server but a totaly different software stack, and exponentially more complex is not only put more factories like textile.
There's still order of magnitude change analogies to real world processes, if people are willing to listen (which is the hard part). Use something that everybody can understand, like making pancakes or waffles or an omelet. Going from making 1 by hand, every 4 minutes at home for your family, to 1,000 pancakes per minute at a factory is obviously going to take a better system. You can scale horizontally, and do the equivalent of putting more VMs behind the load balancer, and hire 4,000+ people to cook, but you still need to have/make that load balancer in the first place for even that to work.
That's the tip of iceberg when going from 1 per 4 minutes to 1,000 per minute though. How do you make and distribute enough batter for that system, and plating and serving that is going to take a pub/sub bus, err, conveyor belt to support the cooks' output. Again though, you still gotta make that kafka queue, err, conveyor belt, plus the maintenance for that is going to a team of people if you need the conveyor belt to operate 24/7/52. If your standards are so high that the system can never go down for more than 52.6 minutes per year or 13.15 minutes per quarter, then that team needs to consist of highly-trained and smart (read: expensive) people to call when the system breaks in the middle of the night.
Re: Ask HN: Do you find working on large distributed systems exhausting?
#160Google's SRE books cover a lot of the things that large teams managing large distributed systems encounter and how to tackle it in a way that doesn't burn out engineers. Depending on organization size/spread, follow-the-sun oncall schedules drastically reduce burnout and apprehension about outages. Incident management procedures give confidence when outages do happen. Blameless postmortems provide a pathway to understanding and fixing the root causes of troublesome outages. Automation reduces manual toil. Google SRE has been keeping a lot of things running for a decade or more and has learned a lot of lessons. I did that from 2014 to 2018 and it seemed like a pretty mature organizational approach, and the books document essentially that era.