101–110 of 284 posts
Earlier quoted context omitted.
We see this pattern at PagerDuty over the majority of our customers. There is a definite lull in alert volume over the weekends that picks up first thing Monday morning. It's led to my personal conclusion that most production issues are caused by people, not errant hardware or systems.
I've come to question releasing often as a result.
Earlier quoted context omitted.
that sounds nightmarish
Why? I'm assuming their highish engineer/service ratio is because their services do less individually. Anecdotally, I've worked on services that ran tens of thousands of instances across the world. You build the tools to manage them and it works very well.
Microservices can go to far. I'm very thankful for this video.
Highly recommended video. Lots of stuff he spoke are very relatable. Like having many repos , storing configs as a separate repo, politics by people, having a tracking system
Sorry, don't think so. I went through the slides and didn't find anything really interesting.
"Uber is most reliable over the weekends when engineers don't change it" :)
Facebook noticed the same thing. http://www.darkcoding.net/software/facebooks-code-quality-pr... - "Exhibit C: Our site works when the engineers go on holiday"
Earlier quoted context omitted.
Which isn't necessarily a problem. Just put X shard on the same machine. If the largest city can be handled by a single server, I generally agree with your parent. I've thought about this problem before, both for a related problem space and with friends working in this specific space. The short version is to create a grid where each square holds car data (id, status, type, x, y, ...) in-memory. Any write-lock of conc…
I always wonder in these cases about giving each car an actor in Erlang/Elixir and having complete network transparency, message handling and crashes handled for free. The routing is very complex too but as you note scales well, until you want to start routing/pickups based on the realtime location of other cars.
That describes no system, ever, including Erlang. TANSTAFL
Just to confirm, 1000 microservices in this case is 1000 different apps (e.g.different docker images) running simultaneously? 1000 microservices in this case not 1000 microservice instances (e.g. docker instances)? If it is 1000 microservices as in different apps, then they must have at least 2000 running apps (at least 2 instances per app for HA). Maybe uber only have 200 "active" microservice app running at the sam…
Our number is closer to 1,700 now, but yes this means 1,700 distinct applications. Each application has many instances, some have thousands of instances.
Earlier quoted context omitted.
Or until the power is cycled. That's why it's best to do the crazy monkey or whatever it's called, walk into the production offline datacenter, online for testing (you do have geographically redundant datacenters right?) and pull out a half dozen random NIC cables. You can simulate with software tools but nothing beats the real test.
> crazy monkey Literally lol'd. It's "chaos monkey".
Earlier quoted context omitted.
We see this pattern at PagerDuty over the majority of our customers. There is a definite lull in alert volume over the weekends that picks up first thing Monday morning. It's led to my personal conclusion that most production issues are caused by people, not errant hardware or systems.
After working at various enterprises over the years (where deployments are slower in some cases) I've noticed you'd do a Thursday/Friday/weekend deployment, everything "looks good" and you'll still have a bunch of issues Monday morning due to users finally using the system en masse.
"Uber is most reliable over the weekends when engineers don't change it" :)
Just to confirm, 1000 microservices in this case is 1000 different apps (e.g.different docker images) running simultaneously? 1000 microservices in this case not 1000 microservice instances (e.g. docker instances)? If it is 1000 microservices as in different apps, then they must have at least 2000 running apps (at least 2 instances per app for HA). Maybe uber only have 200 "active" microservice app running at the sam…
Our number is closer to 1,700 now, but yes this means 1,700 distinct applications. Each application has many instances, some have thousands of instances.