181–190 of 261 posts
Earlier quoted context omitted.
You could have used a sharded database like Mongo. Just throw up 10 shards, use "source" (influencer or brand name) as shard key?
Yes, I could have used Mongo, but it would have been 100x to 1000x slower than an mmap-ed look up table.
What do you find exhausting? One anti-pattern I've found is that most orgs ask a single team to handle on-call around the clock for their service. This rarely scales well, from a human standpoint. If you're getting paged at 2:00 in the morning on a regular basis you will start to resent it. There's not much you can do about that so long as only one team is responsible for uptime 24/7. The solution is to hire operatio…
Yes, you would get calls at 2am, sometimes multiple days in a row. But you were only on call once every six to eight weeks, and we scheduled out well in advance so you could plan your life accordingly.
As a bonus, for the five weeks you weren't on call, you were highly incentivized (and had the time) to build tools or submit patches to fix the problems that woke you at 2am.
> It requires (oh, the horror!) actually maintaining production standards, runbooks, and other documentation.
I disagree with this too. Documentation and runbooks are useless in an outage. Instead of runbooks, write code to do the thing. Instead of documentation, comment the code and build automation to make the documentation unnecessary, or at least surface the right information automatically if you can't automate it.
Earlier quoted context omitted.
You could have used a sharded database like Mongo. Just throw up 10 shards, use "source" (influencer or brand name) as shard key?
Yes, I could have used Mongo, but it would have been 100x to 1000x slower than an mmap-ed look up table.
Earlier quoted context omitted.
Your impression comes from the fact that you have not worked at larger teams, as you said so yourself. It's relatively easy to build something scalable from the beginning if you know what you need to build and if you are not already handling large amounts of traffic and data. It's a whole different ballgame to build on top of an existing complex system already in production that was made to satisfy the needs at the t…
GP said they have never work on something that truly needed 50+ engineers. Truly being the keyword here IMO. I have worked on a 1000+ engineer project and another that was 500+, but I'm on the same boat as GP. Both of those didn't needed 50+, and the presence of the extra 950/450 caused several communication, organisational and architectural issues that became impossible to fix on the long term. So I can definitely s…
I'm also admittedly extremely curious what (broadly) had 1000 (and 500) engineers dedicated to it, when arguably only 50 were needed. Abstractly speaking that sounds a lot like coordinational/planning micromanagement, where the manglement had final say on how much effort needed to be expended where instead of allowing engineering to own the resource allocation process :/
(Am I describing the patently impossible? Not yet had experience in these types of environments)
- Your micro service should be able to run independently. No shared data storage, no direct access into other microservices' storage.
- Your service should protect itself from other services, rejecting requests before it becomes overloaded.
- Your service should be lenient on the data it accepts from other services, but strict about what it sends.
- Your service should be a good citizen, employing good backoffs when other services it is calling appear overloaded.
- The API should be the contract and fully describe your service's relationship to the other services. You should absolutely collaborate with the engineers who make other services, but at the end of the day anything you agree on should be built into the API.
Generally if you follow these best practices, you shouldn't have to maintain a huge working knowledge of the system, only detailed knowledge of your part, which should be small enough to fit into your mental model.
There will be a small team of people responsible for the entire system and how it fits together, but ideally if everyone is following these practices, they won't need to know details of any system, only how to read the APIs and the call graph and how the pieces fit together.
We do billions of requests a day on one of the teams that I manage at work, and that team alone has sole operational and development responsibility for a large number of subsystems to be able to manage the complexity that a sustained QPS of that level requires. But those subsystems are in turn dependent on a whole suite of other subsystems which other teams own and maintain.
It requires a lot of coordination with a spirit of good-will and trust among the parties in order to be able to develop the organizational discipline and rigor needed to be able to handle those kinds of loads without things falling over terrible all the time and everybody pointing fingers at each other.
But! There are lots of great people out there who have spent a lot of time figuring out how to do these things properly and that have come up with general principals that can be applied in your specific circumstances (whatever they may be). And when executed properly I would argue that these principals can be used to mitigate the burnout you're talking about. It's possible to make it through those rough spots in an organization (that frequently, though not always, come from quick business scaling -- i.e. we grew from 1000 customers to 10,000 last year) etc.
If you're feeling this kind of feeling and the organization isn't taking steps to work on it, then there are things you can do as an IC to help, too. But this is all a much longer conversation :)
What do you find exhausting? One anti-pattern I've found is that most orgs ask a single team to handle on-call around the clock for their service. This rarely scales well, from a human standpoint. If you're getting paged at 2:00 in the morning on a regular basis you will start to resent it. There's not much you can do about that so long as only one team is responsible for uptime 24/7. The solution is to hire operatio…
I don’t think this is a stable long term solution. The “on call” teams end up frustrated with the engineers who ship bugs and this results in added process that delays deploys, arbitrary demands for test coverage, capricious error budgets, etc. It’s much better to have the engineers who wrote the code be responsible for running it, and if their operational burden becomes too high, to staff up the dev team to empower…
One of the challenges for larger companies in trying to make teams on-call 24/7 is that your most senior engineers often have enough money that they don't have to take on-call. Some variation of the following conversation happens in Big Tech more than most people seem to anticipate:
"hey, so I have 7 mil in the bank, a house, and kids; so I'm not taking on-call anymore"
"I understand on-call is a burden, but the practice is a big part of how we maintain operational excellence"
"Alright, I quit"
"Woah woah woah, uh, ok, what about we work on transitioning you out of on call over the next 6 months?"
"Nah, I'm done"
"This is going to be really disruptive to the team!"
"Yeah man it sucks, I really feel for you"
My understanding is a few famous outages at large cloud providers are a direct result of management not anticipating these conversations and assuming 24/7 on-call from a single geographically centered team of high powered engineers was sustainable.
It definitely can be. I'm constantly trying to push our stack away from anti-patterns and towards patterns that work well, are robust, and reduce cognitive load. It starts by watching Simple Made Easy by Rich Hickey. And then making every member of your team watch it. Seriously, it is the most important talk in software engineering. https://www.infoq.com/presentations/Simple-Made-Easy/ Exhausting patterns: - Mutable…
E.g. what exactly does it mean to: >> Don’t use an object to handle information. That’s not what objects were meant for. We need to create generic constructs that manipulate information. You build them once and reuse them. Objects raise complexity in that area.
What kind of generic constructs?
What do you find exhausting? One anti-pattern I've found is that most orgs ask a single team to handle on-call around the clock for their service. This rarely scales well, from a human standpoint. If you're getting paged at 2:00 in the morning on a regular basis you will start to resent it. There's not much you can do about that so long as only one team is responsible for uptime 24/7. The solution is to hire operatio…
I would respectfully say that you are wrong. I speak from experience. At Netflix we tried to hire for around the clock coverage. But what ended up working much better was taking that same team and having each person on call for a week at a time, all based in Pacific Time. Yes, you would get calls at 2am, sometimes multiple days in a row. But you were only on call once every six to eight weeks, and we scheduled out we…
Our support team does the same, and they seem to be quite happy with it. They also get the following Friday off (in addition to compensation).
They do their best to shield us developers from after-hour calls, usually one can get things moving enough that it can be handled properly in the morning.