Ask HN: Do you find working on large distributed systems exhausting?

> you were highly incentivized (and had the time) to build tools or submit patches to fix the problems that woke you at 2am.

Ah, so you worked on a team where the SRE needs were prioritized over the feature requests? Because in most companies where I've worked, Product + Customer Service + Sales + Marketing + Executives don't really have time or patience for the engineers to get their diamond polishing cloths out. They want to see feature development. They're willing to be forced to prioritize exactly which feature they'll get soonest, and they understand that engineering needs time to keep the systems running, but in most businesses I've worked, the business comes first.

> Documentation and runbooks are useless in an outage. Instead of runbooks, write code to do the thing. Instead of documentation, comment the code and build automation to make the documentation unnecessary

We do that too. If you could write code to Solve All The Problems then you'd never need to page a human in the first place ;)

I'll give you a simple example of where you can't write code to solve this sort of thing. Let's say that you have an autoscaler that will scale your server group up to X servers. You define an alert to page you if the autoscaler hits the maximum. The page goes off. Do you really want to write code to arbitrarily increase the autoscaler maximum whenever it hits the maximum? Why do you have the maximum in the first place? The entire reason why the autoscaler maximum exists is to prevent cost overruns from autoscaling run amok. You want a human being, not code, to look at the autoscaler and make the decision. Do you have steady-slow growth up to the maximum? Maybe it should be raised, if it represents natural growth. Maybe it shouldn't, if you just raised it last week and it shouldn't be anywhere near this busy. Do you have hockey-stick growth? Maybe the maximum is working as expected, looks like a resource leak hit production. Or maybe you have a massive traffic hit and you actually do want to increase the maximum. Maybe you'd prefer to take the outage from the traffic hit, let the 429s cool everyone off. But good luck trying to write code to handle that automatically, and correctly for you!

> or at least surface the right information automatically if you can't automate it.

Ah, well, that's exactly what the dedicated operations staff are doing, because when you have three follow-the-sun teams, you need standards, not three sets of people who each somehow telepathically share the same tribal knowledge?

Don't get me wrong, I'm not anti-automation or something. If your operations folks are click-clicking in consoles all day long, the same click-clicking every day, probably something's wrong. But the SRE model asks for operations automation to stick within operations teams, not development teams.

> The “on call” teams end up frustrated with the engineers who ship bugs and this results in added process that delays deploys, arbitrary demands for test coverage, capricious error budgets, etc.

This is poor operations culture. Software is no different from industrial manufacturing. You QA before you ship product to customers and you QA your raw materials before you start to process them. Operations is responsible for catching show-stopper bugs before they hit production. This means that operations is responsible for pushing to staging, not developers; operations stakeholders need to be looped into feature planning to ensure that feature work will easily integrate into the operations culture (somebody's got to tell the developers they can't adopt MySQL if it's a PostgreSQL shop, etc.). Fundamentally, Ops needs to be able to say No to Dev. The SRE take on it is to "hand the pager back to Dev", but the actual method of saying No is different from Ops culture to Ops culture.

> reliability people who tend to be systems experts but lack the business logic intuition to spot errors at a glance

If Dev didn't build the monitoring, the observability, put proper logging in place, etc., then honestly, Dev isn't going to spot the errors at a glance. Customer Service will when customers complain. @jedberg seems to think that Developers should write code to auto-solve their operations issues. If Developers can write code to auto-solve their operations issues, and Developers obviously anyway need to add telemetry etc., then why, pray tell, should it be so unreasonable to expect Developers to be able to succinctly add the kind of telemetry and documentation that explains the business logic, according to an Operations standard, such that Operations can thus keep the system running?

I wrote such a system. 6+ years, between end of '07 to beginning of '14. It grew organically, with more and more end points as time went by, and when I exited the project it had over 250 end points, each having hundreds of thousand of users requests per day. By your measurement, that would mean the system I wrote would've handled in a month a total of 250 (end points) x 30 (days) x ~400k (requests per day) == 3B user requests in a month.

To my knowledge the system is still used to this day and I think it grew 10x meanwhile, so I think it's serving over 30B requests each month.

That being said, to answer your question - Yes! I got tired of it, started to plateau and felt I was lagging behind in terms of keeping up with technology around me. So I exited but at the same time I also started to get involved in other projects as well. So in the end I was overworked and I ditched the biggest project of my entire career as freelancer because payment was not worth anymore. I wanted to feel excited and the additional projects eventually made up in terms of money, but boy oh boy! the variation is what made me not feeling burnout. Nowadays if I feel another project is going that route I discuss with client to replace me with a team once I deliver the project in a stable state and for horizontal scaling.

you seem to have a deeper knowledge of the business & organisational context that dictate the true requirements than someone working there. please share these details so we can all learn!

The end is near?