Earlier quoted context omitted.
> crazy monkey Literally lol'd. It's "chaos monkey".
Chaos monkey is a simulation, I believe.
It's not a simulation, it's real-world production technology.
What I Wish I Had Known Before Scaling Uber [video]
161–170 of 284 posts
Re: What I Wish I Had Known Before Scaling Uber [video]
#162Earlier quoted context omitted.
I've come to question releasing often as a result.
It seems like so many "best practices" are really thinly veiled attempts at exploding complexity with only tenuous potential business advantages. We create ourselves so many of the problems we are paid to solve.
> It seems like so many "best practices" are really thinly veiled attempts at exploding complexity
I'll bite. :D
It's a bit more nuanced than that IMO, the "deploy often" mantra is only as good as the process around release + deploy. If you half-ass testing and push to production without without a process for verification -- or if, say, your deploy process is half-baked, or your staging environment is worthless -- you can probably expect "interesting" production deploys on a pretty regular basis.
As much as we'd like to pretend we're all good engineers, this happens more often than you'd expect -- even with good engineers: at some point a company transitions from scrappy startup to a shambling beast, and the things that used to work for a scrappy startup (like skimping on testing and dealing with failures in production) are insufficient when you've got more eyes on the product. Further, the engineering culture remains stuck in "scrappy startup" mode long after the shambling has begun.
And all that's ignoring the fact that less frequent deploys with more changes have their own set of problems. We actually got to a point with deploys of a certain distributed system such that we were terrified if we had more than a few days worth of changes queued up. So many things that could go wrong! :)
> We create ourselves so many of the problems we are paid to solve.
This, on the other hand, I completely 100% agree with: if not us, then who? :)
EDIT: minor formatting change
Re: What I Wish I Had Known Before Scaling Uber [video]
#163Re: What I Wish I Had Known Before Scaling Uber [video]
#164Earlier quoted context omitted.
When the Facebook IPO crashed the NASDAQ, I suspect it's because NASDAQ was sharding by ticker symbol. That was rational until one ticker became 1/3 of the trading volume.
No, that had absolutely nothing to do with it. It was based on a bad assumption about volume of trade entries/cancellations during the initial price calculation.
Sounds like you know more. Care to explain what the assumption was and how it caused the malfunction?
Re: What I Wish I Had Known Before Scaling Uber [video]
#165I think these "alarming trends" are highlighting that operational complexity is an easier pill to take than conceptual complexity, for most workers in the field. Microservices address that gap. And in the process the field is transformed from one of software developers to software operators. This generation is witnessing the transfer of the IT crew from the "back office" to the "boiler room".
Do you think that's it's that people would rather deal with operational complexity, or just that it's easier to not think about operational complexity early on ("running locally, with low volume, everything works together great!")? Personally, I vastly prefer debugging and building on non-microservice architectures than on something split willy-nilly into dozens of services (because most implementations I've seen don…
>>> Do you think that's it's that people would rather deal with operational complexity, or just that it's easier to not think about operational complexity early on
That's one way to scale development projects.
Have multiple teams of dev who work on separate stuff. They can develop in their corner however they want and that makes them happy. The final thing is a clusterfuck of services with little cohesion [YET they ALWAYS manage to put the shit together in production in a mostly working state].
The alternative implies to have a consistent and cohesive view of the components. For that, you'll need to have people with experience in architecturing scalable and sane production systems [so they understand what are the consequences and tradeoffs of ALL their decisions]. I know of very few people on the planet who can design systems. (We're talking past unicorn-rarity here). Plus, the developers must actually listen AND care about the long term maintenance AND the people involved (i.e. not want to do shit that will hit either them in 6 months or the team in the next office).
The amount of collaboration AND communication AND skills required to operate this strategy is beyond the realm of most people and organisation. There are very few individuals who can execute at this level.
Re: What I Wish I Had Known Before Scaling Uber [video]
#166Earlier quoted context omitted.
And one to aggregate its logs.
And another to monitor whether the logs are being aggregated. And yet another one to ensure that the log-aggregation-monitoring-service is working.
Our grandpa's were running banks with 20 cobol programmers, they were not even calling themselves engineers
Re: What I Wish I Had Known Before Scaling Uber [video]
#167I think the world of service architecture is roughly divided in two camps: (1) people who still naively think that Rest/JSON is cool and schemas and databases should be flexible and "NoSQL" is nice and (2) people who (having gone through pains of (1)) realized that strong schemas, things like Thrift, Protobufs, Avro are a good thing, as is SQL and relational databases, because rigid is good. (Camp 1 is more likely to…
> people who still naively think that Rest/JSON is cool and schemas and databases should be flexible and "NoSQL" is nice and Yes, and no. Yes: REST / JSON is nice. I've used them widely as a kind of cross-platform compatibility layer. i.e. instead of exposing SQL or something similar, the API is all REST / JSON. That lets everyone use tools they're familiar with, without learning about implementation details. The RES…
"Once you've inserted crap into the DB, it's hard to fix it."
Agreed, it is hard to fix. NoSQL databases can be really hard to fix when they are full of crap, too.
Re: What I Wish I Had Known Before Scaling Uber [video]
#168Just to confirm, 1000 microservices in this case is 1000 different apps (e.g.different docker images) running simultaneously? 1000 microservices in this case not 1000 microservice instances (e.g. docker instances)? If it is 1000 microservices as in different apps, then they must have at least 2000 running apps (at least 2 instances per app for HA). Maybe uber only have 200 "active" microservice app running at the sam…
Our number is closer to 1,700 now, but yes this means 1,700 distinct applications. Each application has many instances, some have thousands of instances.
>>> Our number is closer to 1,700 now, but yes this means 1,700 distinct applications. Each application has many instances, some have thousands of instances.
Time to forbid adding more stuff and start cleaning.
(12) [...] perfection has been reached not when there
is nothing left to add, but when there is nothing
left to take away.
Re: What I Wish I Had Known Before Scaling Uber [video]
#169Earlier quoted context omitted.
I work at a microservice company. An example microservice is our geoservice which simply takes a lat/lon and tells you what service region the user is in (e.g. New York, San Francisco, etc..). You can see how dozens of these services might be needed when handling a single request coming in from the front end or mobile apps. The service may eventually gain another related function or two as we work on tearing down our…
Why would you make that a separate service when e.g. one query on a PostGIS table can do that?
One query on a table is the exact same thing as a HTTP GET call on a service.
Re: What I Wish I Had Known Before Scaling Uber [video]
#170I think these "alarming trends" are highlighting that operational complexity is an easier pill to take than conceptual complexity, for most workers in the field. Microservices address that gap. And in the process the field is transformed from one of software developers to software operators. This generation is witnessing the transfer of the IT crew from the "back office" to the "boiler room".
Do you think that's it's that people would rather deal with operational complexity, or just that it's easier to not think about operational complexity early on ("running locally, with low volume, everything works together great!")? Personally, I vastly prefer debugging and building on non-microservice architectures than on something split willy-nilly into dozens of services (because most implementations I've seen don…
The inherent greater complexity of designing extensible and cohesive systems turns away a large subset right away and it never gets to a comparative matrix to inform the decision making process. It's been reduced to "monoliths bad, microservices good" level of thinking.
I honestly think that the rise of microservices as "the technical cure for cancer and everything else" is a very interesting surfacing of the various systemic disfunctions of our beloved software industry. The disfunctions have been present from day 1 but the environment has somewhat radically changed.