Live data from Hacker News

What I Wish I Had Known Before Scaling Uber [video]

youtube.com

251–260 of 284 posts

Re: What I Wish I Had Known Before Scaling Uber [video]

#251
post #246

Earlier quoted context omitted.

You are right. Making small changes that can be isolated for debugging purposes is a good approach. What I mean is that we should always question "best" practices and how we apply them to our development process. These days there is a tendency to drink the kool-aid (guilty of this as well). DevOps is something that we are still learning and developing as a profession.

> DevOps is something that we are still learning and developing as a profession. Not really. DevOps is simply a modernish label for system administration that we have for dozens years already.

Isn't that just "Ops"?

Re: What I Wish I Had Known Before Scaling Uber [video]

#252

Earlier quoted context omitted.

Are you implying that software purpose is to provide developers with puzzles to tinker with... in detriment of their paying customers?

I see how one could interpret it that way. I meant to imply rather that software generally needs to be regularly changed in order to continue providing value to its paying customers. At minimum, you've got keeping up with security patches, library/framework deprecations etc. Software which has literally not been changed in years is often an insecure timebomb, with unpatched vulnerabilities, and as soon as you do need…

Ok, that's a reasonable position, though not one I completely agree with.

It would be nice if we as customers had the freedom to pick up which upgrades to install, and to limit ourselves to security patches and (maybe, some) library deprecations. Windows upgrades let you do just that, even if it is a pain to manage. In my previous job, we held monthly meetings to discuss which windows patches to install in our dozens of business critical Win2k3 servers.

For most software, upgrades are opaque. In the best case, you are given a binary choice: install now or install later. In many cases you dont even get that, either the product stops working until you apply the patch or the patch gets automatically applied without your explicit consent.

Which would itself not be a problem if most upgrades did what they are ment to do. Instead, as the GP suggest, patches introduce new (sometimes critical) bugs all the time. With pure repair patches, the balance is mostly positive siince they fix more bug than they break, but with every new feature (which usually is required by a minority of very vocal customers) comes a risk that some other more important feature is going to stop working as expected.

Industry wide, we lack the discipline to change things in a responsible way. Change management is today what source version control was back in the 90s: something that most people has heard good things about, but most people is not doing correctly, and a significan minority not at all.

Re: What I Wish I Had Known Before Scaling Uber [video]

#253
post #246

Earlier quoted context omitted.

> DevOps is something that we are still learning and developing as a profession. Not really. DevOps is simply a modernish label for system administration that we have for dozens years already.

Isn't that just "Ops"?

My point exactly.

Re: What I Wish I Had Known Before Scaling Uber [video]

#254

It amazes me they have 1,700 services. It would be hilarious satire to actually see the description of each. And the debugging scenarios he listed make it sound like they have very poor engineers working for them. Who on earth lets an application get to prod that iterates through a list of items and makes a request for each thing? When did we loose our heads and think such an architecture is sane? The UNIX philosophy…

Saying something like '...they have very poor engineers working for them.' is pretty unfounded, and naive. It's easy to say that having 1700 services is overkill from our point of view, but we don't know the complete architecture, problems being solved and environment that they operate in. One thing to possibly consider is that once you have set up tooling for a platform; logging, request tracing, alerting, orchestra…

Yeah, that's completely ridiculous.

I'm currently working with a very (very) large UK-based retailer, with various outlets globally, and I can tell you from first hand experience that their selection process is ruthless. There are no, or at least very few, lamers in the tech team.

And yet, when you look at the systems, and the way things are built, some of it just seems crazy. Only it's not, and for many reasons, of which here are only a couple:

- The business has grown and evolved over time - with significant restructuring occurring either gradually, or more quickly and in a more intentionally managed way; systems have grown and evolved with it, often organically (which is something I see at every company I've worked with).

- Legacy systems that make it really hard to change things, and migrate to newer and better ways of doing things because they're still in production and depended upon by a host of other systems and services.

- The degree of interconnectedness between systems and services is high across the board; this isn't a reflection of bad design, so much as a reflection of the complexity of the business.

In any large organisation these kinds of things are apparent. I would imagine that in a large organisation that has become large very quickly, like Uber, if anything the problems would be magnified.

To say that they have poor engineers is therefore very unfair: these are people who would have needed to ship RIGHT NOW on a regular basis in order to facilitate the growth of the business. That's going to lead to technical compromises along the way, and there's little to be done to avoid it. There's also a difference between being aware of a problem and having time, or it being a priority, to fix it (e.g., iterating and retrieving).

Re: What I Wish I Had Known Before Scaling Uber [video]

#255
post #6

"Uber is most reliable over the weekends when engineers don't change it" :)

We see this pattern at PagerDuty over the majority of our customers. There is a definite lull in alert volume over the weekends that picks up first thing Monday morning. It's led to my personal conclusion that most production issues are caused by people, not errant hardware or systems.

Couldn't this also just as easily be a change in usage patterns?

Re: What I Wish I Had Known Before Scaling Uber [video]

#256

Earlier quoted context omitted.

You seem to be missing the point though. Most developers working on microservices projects don't have to deal with the operational complexity. They just need to work on their microservice and don't have to know what is going on elsewhere. Personally I much prefer microservices for debugging. You can quickly identify which one is the problem then test the APIs in isolation pretty quickly. Sure beats having to wait 20…

So what do these people do when they get data back from one of their dependency services, and it looks wrong? Or a bug is reported that somewhere in the chain, something is being dropped on the floor? You say you can quickly identify which one is the problem, but if that's a chain that spans 5 teams, how does that actually work in practice? (My experience is that it doesn't.) That's the sort of thing I was including…

Exactly. Testing an API in isolation doesn't help you a thing when the bug arises because of, say, subtle inconsistencies between API implementations of interacting services on a long chain.

This sounds a lot like the code coverage fallacy. (to which I usually answer "call me when you have 100% data coverage").

Re: What I Wish I Had Known Before Scaling Uber [video]

#257
post #71
post #65

Earlier quoted context omitted.

I don't agree with this. I'm just a small developer working for a medium-sized company, but I'm a python guy and I love RDBs and REST/JSON. Sometimes it's easier for me to alter the JSON payload a certain way in the frontend, then the python backend handles it and saves it to the DB. The RDB helps not to stray too far into crazy-land, while python and JSON gives you the flexibility to prototype and experiment.

In the absence of a good reason not to use Python (and there are many case where one should not), I use it quite a bit myself. And usually (absent good reason) I use it with PostresQL. SQLAlchemy is a wonderful thing. Granted, I am neither a database nor ORM savant, but I find that it makes explicit almost as easy as implicit - but with more safety! I haven't seen that elsewhere, but I haven't looked very hard either…

You should check out what the Elixir folks are doing with Ecto. I'm a huge fan so far!

Re: What I Wish I Had Known Before Scaling Uber [video]

#258
post #71

Earlier quoted context omitted.

In the absence of a good reason not to use Python (and there are many case where one should not), I use it quite a bit myself. And usually (absent good reason) I use it with PostresQL. SQLAlchemy is a wonderful thing. Granted, I am neither a database nor ORM savant, but I find that it makes explicit almost as easy as implicit - but with more safety! I haven't seen that elsewhere, but I haven't looked very hard either…

You should check out what the Elixir folks are doing with Ecto. I'm a huge fan so far!

Thanks for the pointer. I'm already using BEAM in a project, so that wouldn't be a bridge too far for that one...

Re: What I Wish I Had Known Before Scaling Uber [video]

#259

Earlier quoted context omitted.

Is your first example really "naive" though? In my experience, loose, flexible schemas and dynamic languages are very well suited to rapid early-stage development, much more so than rigid languages and schemas. Sure, in the long term things should be refactored, structured, and optimized. But if you do that too soon you risk locking yourself out of potential value, as well as gold-plating things that aren't critical.

> Sure, in the long term things should be refactored, structured, and optimized. How often does that really happen though? Once you've amassed enough technical/data debt, resistance to refactoring increases until it never happens at all. Having well defined, coherent data models and schemas from the start will pay off in the long run. Applications begin and end with data, so why half-ass this from the get go?

In my experience the likelihood of your scenario (increasing resistance to refactoring) is almost always indirectly proportional to the amount and quality of refactoring tools available.

500KLOC JVM/.NET application? No big deal.

50KLOC JS/HTML-based SPA? Pfooooh. That could take a while...do we really need to?

Re: What I Wish I Had Known Before Scaling Uber [video]

#260
post #164

Earlier quoted context omitted.

Sounds like you know more. Care to explain what the assumption was and how it caused the malfunction?

http://www.system-logic.com/commentary/2013/06/04/NASDAQ-and...

That doesn't contradict my conjecture that the issue was improper sharding. The jump in duration of cancellation detection from 2ms to 20ms could have been because they were running that calculation on a single machine.

Although... Ethernet latency would probably make it tough to stick to 2ms.

Post reply on HN