Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Do you find working on large distributed systems exhausting?

news.ycombinator.com

Post reply on HN
Print view View original

101–110 of 261 posts

Re: Ask HN: Do you find working on large distributed systems exhausting?

#101

Post by jqgatsby »

fxtentacle wrote: post #18

Yes, I used to, but No, I fixed it :) Among other things, I am team lead for a private search engine whose partner-accessible API handles roughly 500 mio requests per month. I used to feel powerless and stressed out by the complexity and the scale, because whenever stuff broke (and it always does at this scale), I had to start playing politics, asking for favors, or threatening people on the phone to get it fixed. Hi…

@fxtentacle, I was curious which private search engine this is for. Is the system you are describing ImageRights.com?

Re: Ask HN: Do you find working on large distributed systems exhausting?

#102

Post by notimetorelax »

aeyes wrote: post #65
fxtentacle wrote: post #18

Yes, I used to, but No, I fixed it :) Among other things, I am team lead for a private search engine whose partner-accessible API handles roughly 500 mio requests per month. I used to feel powerless and stressed out by the complexity and the scale, because whenever stuff broke (and it always does at this scale), I had to start playing politics, asking for favors, or threatening people on the phone to get it fixed. Hi…

I feel that your problems aren't even remotely related to my problems with large distributed systems. My problems are all about convincing the company that I need 200 engineers to work on extremely large software projects before we hit a scalability wall. That wall might be 2 years in the future so usually it is next to impossible to convince anyone to take engineers out of product development. Even more so because w…

I usually move on to a different project/team/company when it gets to this. E.g. my new team builds a new product that grows like crazy and has its own set of challenges. I prefer to be deliver immediate customer value vs. long term hard to sell and hard to project the value work.

Re: Ask HN: Do you find working on large distributed systems exhausting?

#103

Post by jeffrallen »

glintik wrote: post #95

The most undervalued thing that forgot even highly skilled engineers - KISS principle. That’s why you are burning out supporting such systems.

Yes, it's amazing how much one modern high spec system running good code can do. Turn off all the distributed crap and just use a pair in leader/follower config with short ttl DNS to choose the leader and manual failover scripts. If your app/company/industry cannot accept the compromises from such a simple config, quit and work in one which can.

Re: Ask HN: Do you find working on large distributed systems exhausting?

#104

Post by ehnto »

robertlagrant wrote: post #44
nikhilsimha wrote: post #42

I used to lead teams that owned message bus, a stream processing framework and a distributed scheduler (like k8s) at Facebook. The oncall was brutal. At some point I thought I should work on something else, perhaps even switch careers entirely. However this also forced us to separate user issues and system issues accurately. That’s only possible because we are a platform team. Since then I regained my love for distri…

I like that the mantra went from "move fast and break things" to (paraphrased) "move fast and don't break things".

It's been a pretty poor mantra from the beginning anyway. How about we move at a realistic pace and deliver good features, without burning out, and without leaving a trail of half-baked code behind us?

Re: Ask HN: Do you find working on large distributed systems exhausting?

#105

Post by fxtentacle »

jqgatsby wrote: post #101
fxtentacle wrote: post #18

Yes, I used to, but No, I fixed it :) Among other things, I am team lead for a private search engine whose partner-accessible API handles roughly 500 mio requests per month. I used to feel powerless and stressed out by the complexity and the scale, because whenever stuff broke (and it always does at this scale), I had to start playing politics, asking for favors, or threatening people on the phone to get it fixed. Hi…

@fxtentacle, I was curious which private search engine this is for. Is the system you are describing ImageRights.com?

No, ImageRights is much more requests and mostly images. Also, at ImageRights I don't have management above me that I would need to convince :)

This one is text-only and used by influencers and brands to check which newspapers report about their events. As I said, it's internally used by a few partner companies who buy the API from my client and sell news alerts to their clients.

BTW, I'm hoping to one day build something similar as an open source search engine where people pay for the data generation and then effectively run their own ad-free Google clone, but so far interest has been _very_ low:

https://news.ycombinator.com/item?id=30374611 (1 upvote)

https://news.ycombinator.com/item?id=30361385 (5 upvotes)

EDIT: Out of curiosity I just checked and found my intuition wrong. The ImageRights API averages 316rps = 819mio requests per month. So it's not that much bigger.

Re: Ask HN: Do you find working on large distributed systems exhausting?

#106

Post by fxtentacle »

dekhn wrote: post #93
fxtentacle wrote: post #73

Earlier quoted context omitted.

You might be surprised. The performance equivalent of $100k monthly in EC2 spend fits into a 16m2 cage with 52HU racks.

that cage is a liability, not an asset. How is the networking in that rack? What's its connection to large-scale storage (IE, petabytes, since that's what I work with). What happens if a meteor hits the cage? Etc.

That depends on what contracts you have. You could have multiple of these cages in different locations. Also, 1 PB is only 56 large enterprise HDDs. So you just put storage into the cage, too.

But my point wasn't about how precisely the hardware is managed. My point was that with a large cloud, a mid-sized company has effectively NO SUPPORT. So anything that gives you more control is an improvement.

Re: Ask HN: Do you find working on large distributed systems exhausting?

#107

Post by angarg12 »

I don't find it exhausting, I find it *exhilarating*.

After years of proving myself, earning trust and strategical positioning I am finally leading a system that will support millions of requests per second. I love my job and this is the most intellectually stimulating activity I have done in a long while.

I think this is far from the expectation of the average engineer. You can find many random companies with very menial and low stake work. However if you work at certain companies you sign up for this.

BTW I don't think this is unreasonable. This is precisely why programmers get paid big bucks, definitely in the US. We have have a set of skills that require a lot of talent and effort, and we are rewarded for it.

Bottom line this isn't for everyone, so if you feel you are done with it that's fair. Shop around for jobs and be deliberate about where you choose to work, and you will be fine.

Re: Ask HN: Do you find working on large distributed systems exhausting?

#109

Post by randomsilence »

asim wrote: post #39

Yup. Spent more than a decade doing it. Got so frustrated that I started a company to try abstract it all away for everyone else. It's called M3O https://m3o.com . Everyone ends up building the same thing over and over. A platform with APIs either built in house or an integration to external public APIs. If we reuse code, why not APIs. I should say, I've been a sysadmin, SRE, software engineer, open source creator, m…

>Consume public APIs as simpler programmable building blocks

Is the 'r' in simpler there intentionally? In which way are the building blocks more simple than simple blocks?

Re: Ask HN: Do you find working on large distributed systems exhausting?

#110

Post by wreath »

jmyeet wrote: post #26

It's hard to answer this because you don't specify what exactly you find exhausting. Is it oncall? Deployment? Performance issues? Dealing with different teams? Failures and recovery? The right hand not knowing what the left hand is doing? Too many services? Something else? It's not even clear how big your service is. You mention billions of requests per month. Every 1B requests/month translates to ~400 QPS, which is…

> Every 1B requests/month translates to ~400 QPS, which isn't even that large

I said billions not one billion.

I guess what I find exhausting is the long feedback cycle. For example, Writing a simple script that makes two calls to different APIs requires tons of wiring for telemetry, monitoring, logging, error handling, integrating w/ two APIs, setting up the proper kubernetes manifests, setting up the required permissions to run this thing and have them available to k8s. I find all this to be exhausting. We're not even talking about operating this thing yet (on call, running in issues with the APIs owned by other teams etc)

Post reply on HN