Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Do you find working on large distributed systems exhausting?

news.ycombinator.com

Post reply on HN
Print view View original

131–140 of 261 posts

Re: Ask HN: Do you find working on large distributed systems exhausting?

#131

Post by fxtentacle »

flyinglizard wrote: post #31
fxtentacle wrote: post #18

Yes, I used to, but No, I fixed it :) Among other things, I am team lead for a private search engine whose partner-accessible API handles roughly 500 mio requests per month. I used to feel powerless and stressed out by the complexity and the scale, because whenever stuff broke (and it always does at this scale), I had to start playing politics, asking for favors, or threatening people on the phone to get it fixed. Hi…

Care to share uptime metrics on AWS vs your own servers?

That wouldn't be much help because the AWS and Heroku metrics are always green, no matter what. If you can't push updates to production, they count that as a developer-only outage and do not deduct it from their reported uptime.

For me, the most important metric would be time that me and my team spent fixing issues. And that went down significantly. After a year of everyone feeling burned out, now people can take extended vacations again.

One big issue for example was the connectivity between EC2 servers degrading, so that instead of the usual 1gbit/s they would only get 10mbit/s. It's not quite an outage, but it makes things painfully slow and that sluggishness is visible for end users. Getting reliable network speeds is much easier if all the servers are in the same physical room.

Re: Ask HN: Do you find working on large distributed systems exhausting?

#132

Post by fxtentacle »

ddorian43 wrote: post #36
fxtentacle wrote: post #18

Yes, I used to, but No, I fixed it :) Among other things, I am team lead for a private search engine whose partner-accessible API handles roughly 500 mio requests per month. I used to feel powerless and stressed out by the complexity and the scale, because whenever stuff broke (and it always does at this scale), I had to start playing politics, asking for favors, or threatening people on the phone to get it fixed. Hi…

What are you using for aws alternatives? Example for S3?

docker + self-developed image management + CEPH

Re: Ask HN: Do you find working on large distributed systems exhausting?

#133

Post by fxtentacle »

NavinF wrote: post #52
nostrebored wrote: post #38

Earlier quoted context omitted.

You had problems with management of a cloud based api and executive visibility… so you bought a set of data centers to handle 500mio req per month? The visibility you will get after the capex when there’s a truly disastrous outage will be interesting.

Hmm that’s only 190Hz on average, but we don’t know what kind of search engine it is. For example if he’s doing ML inference for every query, it would make perfect sense to get a few cabinets at a data center. I’ve done so for a much smaller project that only needs 4 GPUs and saved a ton of money.

Nah, it's text-only requests returning JSON arrays of which newspaper article URLs mention which influencer or brand name keyword.

The biggest hardware price point is that you need insane amounts of RAM so that you can mmap the bloom hash for the mapping from word_id to document_ids.

Re: Ask HN: Do you find working on large distributed systems exhausting?

#134

Post by [deleted] »

ckdarby wrote: post #53
fxtentacle wrote: post #18

Yes, I used to, but No, I fixed it :) Among other things, I am team lead for a private search engine whose partner-accessible API handles roughly 500 mio requests per month. I used to feel powerless and stressed out by the complexity and the scale, because whenever stuff broke (and it always does at this scale), I had to start playing politics, asking for favors, or threatening people on the phone to get it fixed. Hi…

>I used to feel powerless and stressed out by the complexity and the scale, because whenever stuff broke (and it always does at this scale), I had to start playing politics, asking for favors, or threatening people on the phone to get it fixed. Higher management would hold me accountable for the downtime even when the whole S3 AZ was offline and there was clearly nothing I could do except for hoping that we'll someho…

[deleted]

Re: Ask HN: Do you find working on large distributed systems exhausting?

#135

Post by danny_taco »

fxtentacle wrote: post #111
aeyes wrote: post #65

Earlier quoted context omitted.

I feel that your problems aren't even remotely related to my problems with large distributed systems. My problems are all about convincing the company that I need 200 engineers to work on extremely large software projects before we hit a scalability wall. That wall might be 2 years in the future so usually it is next to impossible to convince anyone to take engineers out of product development. Even more so because w…

My impression has always been that FAANG need lots of engineers because the 10xers refuse to work there. I've seen plenty of really scalable systems being built by a small core team of people who know what they are doing. FAANG instead seem to be more into chasing trends, inventing new frameworks, rewriting to another more hip language, etc. I would have no idea how to coordinate 200 engineers. But then again, I have…

Your impression comes from the fact that you have not worked at larger teams, as you said so yourself. It's relatively easy to build something scalable from the beginning if you know what you need to build and if you are not already handling large amounts of traffic and data.

It's a whole different ballgame to build on top of an existing complex system already in production that was made to satisfy the needs at the time it was built but it now needs to support other features, bug fixes and supporting existing features but at scale while having 50+ engineers not step on each other and not break each others code in the process. 4 friends in the basement will not achieve more than 50+ engineers in this scenario, even when considering the inefficiencies of the difficulty in communication that come along with so many minds working on the same thing.

Re: Ask HN: Do you find working on large distributed systems exhausting?

#136

Post by sdoering »

Spooky23 wrote: post #89
benjamir wrote: post #78

Earlier quoted context omitted.

Which costs you more than $100k monthly to operate with the same level of manageability and reliability. We don't use AWS, because our use cases don't require that level of reliability and we simply cannot afford it, but if I needed a company to depend on IT that generates enough revenue... I probably wouldn't argue about the AWS bill. So long, prepaid at hetzner + in-house works good enough, but I know what I cannot…

This is a religious debate among many. The IT/engineering nerd stuff doesn’t matter at all. Cloud migration decisions are always made by accounting and tax factors. I run two critical apps, one on-prem and one cloud. There is no difference in people cost, and the cloud service costs about 20% more on the infrastructure side. We went cloud because customer uptake was unknown and making capital investments didn’t make…

> These things are tools and it doesn’t pay to be dogmatic.

I wish I would hear this line more often.

So many things today are (pseudo-) religious now. The right frsmework/language, cloud or on prem, x vs not x.

Especially bad imho when somebody tries to tell you how you could do better with 'not x' instead of x you are currently using without even trying to understand the context this decision resides in.

[Edit] typo

Re: Ask HN: Do you find working on large distributed systems exhausting?

#137

Post by islandert »

kortex wrote: post #119

It definitely can be. I'm constantly trying to push our stack away from anti-patterns and towards patterns that work well, are robust, and reduce cognitive load. It starts by watching Simple Made Easy by Rich Hickey. And then making every member of your team watch it. Seriously, it is the most important talk in software engineering. https://www.infoq.com/presentations/Simple-Made-Easy/ Exhausting patterns: - Mutable…

I agree with most of you points, but the one that stands out is "push for unit tests over integration/system tests wherever possible".

By integration/system tests, do you mean tests that you cannot run locally?

Re: Ask HN: Do you find working on large distributed systems exhausting?

#138

Post by LoveGracePeace »

kortex wrote: post #119

It definitely can be. I'm constantly trying to push our stack away from anti-patterns and towards patterns that work well, are robust, and reduce cognitive load. It starts by watching Simple Made Easy by Rich Hickey. And then making every member of your team watch it. Seriously, it is the most important talk in software engineering. https://www.infoq.com/presentations/Simple-Made-Easy/ Exhausting patterns: - Mutable…

Most of that I agree with, I'm curious why you'd recommend unit tests over integration tests? It seems at odds with the direction of overall software engineering best practices.

Re: Ask HN: Do you find working on large distributed systems exhausting?

#140

Post by macksd »

Mental / emotional burnout is certainly not uncommon in tech (probably in most other careers, I'd bet). Most people in Silicon Valley are changing jobs more often than 4-5 years. I don't like to constantly be the new guy, but there is a refreshing feeling to starting on something new and not carrying years of technical debt on your emotions. Maybe it's time to try something new, take a bigger vacation than usual, or talk to someone about new approaches you can try in your professional or personal life. But certainly don't let the fact that you feel like this add to the load - you're not alone, and it's not permanent.
Post reply on HN