Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling lessons learned at Dropbox, part 1

eranki.tumblr.com

Post reply on HN
Print view View original

71–80 of 92 posts

Re: Scaling lessons learned at Dropbox, part 1

#71

Post by peterwwillis »

eranki wrote: post #70
peterwwillis wrote: post #63

Earlier quoted context omitted.

Can you imagine if they loaded 1000lbs of lead onto an airplane before they took off, just to see if the plane takes off with the headroom filled? "Oh crap, the plane is falling, let's dump the lead." OP's "extra reads" is dumb because he could have had normal metrics for memcached load and planned to only support like 70% capacity or somesuch, and when load hit that number, he would immediately increase capacity. In…

It's actually more like dumping fuel for an emergency landing instead of dumping passengers, if we really insist on using completely irrelevant analogies.

Are planes in the habit of carrying much more fuel than they need, just in case they will weigh too much when landing, in which case they dump the fuel? No. In the inverse example, we don't put extra load on servers in anticipation that load goes up, just so we can dump the extra load and put it back to where it would have been had we not had extra load. The idea is moronic. It is literally the same as putting lead in your servers until you have to dump it, just because you can. Real admins have alerts on their servers to tell them when they will need additional capacity.

Incidentally, fuel dump systems were initially added due to a rule by the FAA that a plane's structural landing weight not be exceeded by its takeoff weight. Many commercial planes never had this problem, so dumping systems were not installed. As a result, most planes just circle until they've burned up enough fuel, or land anyway overweight. You could dump fuel to lessen the chance of explosion, but only if your plane is equipped with a fuel dump system, and such incidents are so rare it's not even a safety consideration.

Re: Scaling lessons learned at Dropbox, part 1

#72

Post by eranki »

peterwwillis wrote: post #71
eranki wrote: post #70

Earlier quoted context omitted.

It's actually more like dumping fuel for an emergency landing instead of dumping passengers, if we really insist on using completely irrelevant analogies.

Are planes in the habit of carrying much more fuel than they need, just in case they will weigh too much when landing, in which case they dump the fuel? No. In the inverse example, we don't put extra load on servers in anticipation that load goes up, just so we can dump the extra load and put it back to where it would have been had we not had extra load. The idea is moronic. It is literally the same as putting lead i…

lol why are you so mad -- perhaps you couldn't make this out through your tears of rage:

"Why not just plan ahead? Because most of the time, it was a very abrupt failure that we couldn’t detect with monitoring."

Re: Scaling lessons learned at Dropbox, part 1

#73

Post by dools »

thezilch wrote: post #66
dools wrote: post #53

but I really hate ORM’s and this was just a giant nuisance to deal with I like object relational mapping as a theory (ie. I have an object of type Author which has 1 or more books I can loop over), but I hate ActiveRecord implementations. Eventually, they just end up implementing almost all of SQL but in some arcane bullshit syntax or sequence of method calls that you have to spend a bunch of time learning. I also se…

All I want from an ORM is to manage caching intelligently. I'll learn some arcane bits to assist the ORM's pathfinding, but I simply can't imagine ORM's strong suits being in writing less tedious queries. Granted, the less-strokes and less-verbose nature of an ORM query is still a nice benefit. There are only two hard things in Computer Science: cache invalidation and naming things. -- Phil Karlton

The simplification of writing join queries where the primary key relationships are obvious also has the nice side effect of meaning that if you change relationships between entities (for example from 1-to-many to many-to-many the code won't have to be changed.

At any rate what I'm saying really is that reducing the amount of keystrokes writing and maintaining joins is the only part of SQL where I see there can be significant gains in productivity through automation of the task.

Most ORMs implement where clauses, from clauses, aggregate functions, grouping, having, etc. etc. etc. ie. they wind up basically re-implementing SQL and abstracting it so that your previous knowledge of SQL is basically obsoleted and in order to debug problems or create complex queries you either have to switch entirely to SQL (in which case you lose all query building functionality) or map in your head the SQL you want to achieve, to the arbitrary syntax provided by the ORM software.

Re: Scaling lessons learned at Dropbox, part 1

#74

Post by jaylevitt »

brc wrote: post #11

The idea of running extra load - it sounds good in theory but I can't help thinking that it's a bit like setting your watch forwards to try and stop being late for things. Eventually you know your watch is 5 minutes fast so start compensating for it. I wonder if this strategy starts to have the same effect - putting fixes off because you know you can pull the extra load before it becomes critical. In the same way you…

I think (I assume) his point was that when you have a complex system that's growing rapidly, it's difficult to predict what the next breaking point will be. Building in extra load gives you a window of opportunity: You discover the breaking point, turn off the extra load, re-engineer, and turn back on the canary so you can find the next breaking point.

In theory, you'd think you can do load-testing and simulations and capacity planning, and find these breaking points ahead of time. In practice, it's not always feasible, and this seems like a simple-enough hack that gets you much of the way there.

Re: Scaling lessons learned at Dropbox, part 1

#75

Post by jaylevitt »

peterwwillis wrote: post #63
Game_Ender wrote: post #24

Earlier quoted context omitted.

This is just an adaption of the "margin" system commonly used in systems engineering. Basically you never spec everything out to perfectly match the load it's expected to bare, always leave some headroom. For example if you are designing an aircraft your first design is never perfect. So you when you do the initial design, you do it as if the aircraft has to weight 70% of what it really will. As errors are corrected…

Can you imagine if they loaded 1000lbs of lead onto an airplane before they took off, just to see if the plane takes off with the headroom filled? "Oh crap, the plane is falling, let's dump the lead." OP's "extra reads" is dumb because he could have had normal metrics for memcached load and planned to only support like 70% capacity or somesuch, and when load hit that number, he would immediately increase capacity. In…

If your system is complex, you don't know what your capacity is, or whether the curve has a sharp knee. Think of disk I/O; if 100K simultaneous users keep your disks 50% busy, how many simultaneous users can you actually support? The answer is not 200K; the answer is "it depends".

Re: Scaling lessons learned at Dropbox, part 1

#76

Post by zzzeek »

dools wrote: post #53

but I really hate ORM’s and this was just a giant nuisance to deal with I like object relational mapping as a theory (ie. I have an object of type Author which has 1 or more books I can loop over), but I hate ActiveRecord implementations. Eventually, they just end up implementing almost all of SQL but in some arcane bullshit syntax or sequence of method calls that you have to spend a bunch of time learning. I also se…

ORMs aren't about saving one the need to know how to write SQL, they are about automating the task of constructing all the redundant SQL needed to marshal data to and from domain objects, as well as all the redundant automation involved in the actual marshaling of data from database client libraries into domain objects.

I think you are exaggerating quite a bit when you refer to SQLAlchemy's patterns adding "mind-bending complexity". Object relational mapping is a complex affair to start with. Have you much experience with modern versions of SQLAlchemy directly (and if not, how fair are comments like that) ?

Re: Scaling lessons learned at Dropbox, part 1

#77

Post by zzzeek »

acslater00 wrote: post #6

For the record, I use sqlalchemy 0.6.6 regularly under fairly heavy load, and have never had a problem with it. Any 'sqlalchemy bugs' are inevitably coding mistakes on my part.

We got a dozen or so email list requests for support from people who I know to be from Dropbox in late 2008. At that time, we were at version 0.4.8. That is an extremely old version and the codebase was quite immature at that time - I personally didn't use SQLAlchemy in production until 0.5 (which of course is because Python was hardly used at all in the early 2000's outside of the zope community, so I was still stuck with Java/Perl gigs). However I am still quite skeptical of the claim that it returned the wrong results. You're expected to watch the SQL you're telling it to generate during development. It will always be true that pushing an ORM will not always generate the SQL you want - which is why you have to make sure those queries are how they should be, before pushing to production. The ORM will of course stick to the plan you've given it - it isn't "deciding" anything, and at worst it can only misinterpret your intent - just like any library.

My strategy with SQLAlchemy has always been to under-promote it. If you have lots of big players early adopting you and hitting all the pointy edges, it can damage your rep. There's a group of major folks out there who will never use my library due to old experiences. Others like Reddit and Yelp have hung on, and apparently dropbox is still using the core, hooray !

That's why I'm always amazed at how aggressively MongoDB is promoted, when it seems like they're still going through a lot of growing pains. I guess they sort of have to, given that they're a business and all.

Re: Scaling lessons learned at Dropbox, part 1

#78

Post by arohner »

dools wrote: post #53

but I really hate ORM’s and this was just a giant nuisance to deal with I like object relational mapping as a theory (ie. I have an object of type Author which has 1 or more books I can loop over), but I hate ActiveRecord implementations. Eventually, they just end up implementing almost all of SQL but in some arcane bullshit syntax or sequence of method calls that you have to spend a bunch of time learning. I also se…

> I also seriously doubt that anyone has ever written a production system of any reasonable complexity and been able to use the exact same ORM code with absolutely any backend (if you have an example please correct me on this).

You're entirely right here, because databases are different. For example, (I forget the exact details), "select count(*)..." in MySQL is O(1), but it's O(log n) or O(n) in Postgres, depending on indices. That's a detail no ORM is going to save you from.

> SQL is easy to learn and very expressive.

Strongly disagree. The reason everyone keeps trying to write ORMs is because 1) SQL is a shitty language and 2) it's not the language that programmers want to use. Write a better frontend language for Postgres, and the ORMs would disappear.

I strongly suspect that would take some of the wind out of the NoSQL crowd. There are certainly NoSQL deployments that would have a hard time on traditional RDMBS, but there are a lot of other places that use Mongo just because they don't like SQL-the-language, rather than Postgres-the-DB.

Re: Scaling lessons learned at Dropbox, part 1

#80

Post by batista »

peterwwillis wrote: post #63
Game_Ender wrote: post #24

Earlier quoted context omitted.

This is just an adaption of the "margin" system commonly used in systems engineering. Basically you never spec everything out to perfectly match the load it's expected to bare, always leave some headroom. For example if you are designing an aircraft your first design is never perfect. So you when you do the initial design, you do it as if the aircraft has to weight 70% of what it really will. As errors are corrected…

Can you imagine if they loaded 1000lbs of lead onto an airplane before they took off, just to see if the plane takes off with the headroom filled? "Oh crap, the plane is falling, let's dump the lead." OP's "extra reads" is dumb because he could have had normal metrics for memcached load and planned to only support like 70% capacity or somesuch, and when load hit that number, he would immediately increase capacity. In…

>Can you imagine if they loaded 1000lbs of lead onto an airplane before they took off, just to see if the plane takes off with the headroom filled? "Oh crap, the plane is falling, let's dump the lead."

They actually do this kind of stuff (except for the "lets dump the lead" part), in stress tests, especially in cargo and millitary planes. And they do similar tests not only in aviation, but in most kinds of engineering.

So maybe misplaced sarcasm?

Post reply on HN