241–250 of 284 posts
Earlier quoted context omitted.
Chaos monkey is a simulation, I believe.
It's not a simulation, it's real-world production technology. https://en.wikipedia.org/wiki/Technical_details_of_Netflix
Earlier quoted context omitted.
Why would you make that a separate service when e.g. one query on a PostGIS table can do that?
One query on a table is the exact same thing as a HTTP GET call on a service.
But since asking the question I've realized that if your application already needs a huge amount of servers because it simply gets that much traffic, then putting something like this in its own docker instance is probably the simplest way (it might even use postgres inside it), if those boundaries change now and then.
But most companies aren't near that scale.
"Uber is most reliable over the weekends when engineers don't change it" :)
We see this pattern at PagerDuty over the majority of our customers. There is a definite lull in alert volume over the weekends that picks up first thing Monday morning. It's led to my personal conclusion that most production issues are caused by people, not errant hardware or systems.
"Uber is most reliable over the weekends when engineers don't change it" :)
We see this pattern at PagerDuty over the majority of our customers. There is a definite lull in alert volume over the weekends that picks up first thing Monday morning. It's led to my personal conclusion that most production issues are caused by people, not errant hardware or systems.
Most of the other alerts are due to network issues, moving servers due to hardware intervention that change the topology of the network, etc etc.
It amazes me they have 1,700 services. It would be hilarious satire to actually see the description of each. And the debugging scenarios he listed make it sound like they have very poor engineers working for them. Who on earth lets an application get to prod that iterates through a list of items and makes a request for each thing? When did we loose our heads and think such an architecture is sane? The UNIX philosophy…
One thing to possibly consider is that once you have set up tooling for a platform; logging, request tracing, alerting, orchestration, common libraries, standards and deployment. Deploying and operating new services becomes relatively straight forward.
Saying that, 1700 is a lot, which makes me intrigued to see inside uber.
Earlier quoted context omitted.
I can see the argument that if releasing causes things to break then don't release so frequently, but in practice the end result of that is lots of things breaking at once and having to unpick everything. Debugging is much easier if you're debugging a single change fresh in your mind.
You are right. Making small changes that can be isolated for debugging purposes is a good approach. What I mean is that we should always question "best" practices and how we apply them to our development process. These days there is a tendency to drink the kool-aid (guilty of this as well). DevOps is something that we are still learning and developing as a profession.
Not really. DevOps is simply a modernish label for system administration that we have for dozens years already.
It amazes me they have 1,700 services. It would be hilarious satire to actually see the description of each. And the debugging scenarios he listed make it sound like they have very poor engineers working for them. Who on earth lets an application get to prod that iterates through a list of items and makes a request for each thing? When did we loose our heads and think such an architecture is sane? The UNIX philosophy…
While I'd concede that 1,700 services is a lot, and probably a function of the fact that they're hiring too many engineers too quickly (there's no possible way to indoctrinate people into sane and standardized engineering practices at this sort of hiring growth), I don't actually think what he's describing is that unusual for a company at Uber's scale or experiencing Uber's rate of growth. I work at Airbnb and while…
I didn't see the video. But given that a cab-service has a natural sharding point (i.e., per city), I don't get why scaling is such an issue.
Cab service on a planetary scale :) They have 2000 engineers, 800 microservices and 8000 GIT repositories. Does it still seem trivial to you? EDIT: to downvoters: why shoot the messenger? :)
It amazes me they have 1,700 services. It would be hilarious satire to actually see the description of each. And the debugging scenarios he listed make it sound like they have very poor engineers working for them. Who on earth lets an application get to prod that iterates through a list of items and makes a request for each thing? When did we loose our heads and think such an architecture is sane? The UNIX philosophy…
Earlier quoted context omitted.
We see this pattern at PagerDuty over the majority of our customers. There is a definite lull in alert volume over the weekends that picks up first thing Monday morning. It's led to my personal conclusion that most production issues are caused by people, not errant hardware or systems.
That doesn't surprise me at all. If you're not changing software, you can't create any additional bugs.