Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling lessons learned at Dropbox, part 1

eranki.tumblr.com

Post reply on HN
Print view View original

81–90 of 92 posts

Re: Scaling lessons learned at Dropbox, part 1

#81

Post by batista »

arohner wrote: post #78
dools wrote: post #53

but I really hate ORM’s and this was just a giant nuisance to deal with I like object relational mapping as a theory (ie. I have an object of type Author which has 1 or more books I can loop over), but I hate ActiveRecord implementations. Eventually, they just end up implementing almost all of SQL but in some arcane bullshit syntax or sequence of method calls that you have to spend a bunch of time learning. I also se…

> I also seriously doubt that anyone has ever written a production system of any reasonable complexity and been able to use the exact same ORM code with absolutely any backend (if you have an example please correct me on this). You're entirely right here, because databases are different. For example, (I forget the exact details), "select count(*)..." in MySQL is O(1), but it's O(log n) or O(n) in Postgres, depending…

No, actually the only reason is "its not a language programmers want to use".

It is very much non shitty.

Its just that lots of programmers, especially OO minded cannot get into its mindset, and use it for what it is, they have to put a lame OO abstraction on top.

Functional programmers shoud fare better in this regard (or Prolog programmers, if they still exist).

If you really want to abstract it, something like LINQ is a better way.

Re: Scaling lessons learned at Dropbox, part 1

#82

Post by batista »

akent wrote: post #14

I noticed that a particular “FUUUCCKKKKKasdjkfnff” wasn’t getting printed where it should have Why not take the extra half a second to make those random strings meaningful and hidden behind a DEBUG log level?

Because a half and second here and a half a second there, and soon we're talking weeks.

Plus, the statement is not only meaningful, but also very expressive.

Re: Scaling lessons learned at Dropbox, part 1

#83

Post by batista »

vosper wrote: post #37
ephemeralgomi wrote: post #21

Earlier quoted context omitted.

Probably most of their logging _is_ meaningful, but deciding how to professionally phrase each and every log message will eventually get you to decision fatigue. The point that he was making with this was that over-logging is a good thing - this probably wasn't something the initial author thought was going to be terribly informative, hence the random string. And yet it ended up diagnosing a real world problem. In a…

I don't know how many new logging statements you commit to production code every day, but I can't imagine it averages out to more than one or two. If you can't take the time to phrase them both professionally and meaningfully then you're doing yourself and your team a disservice.

I think Dropbox did well enough financially and technically, so that the team doesnt need pedantic advice on professionalism...

Re: Scaling lessons learned at Dropbox, part 1

#84

Post by andrewflnr »

pestaa wrote: post #60
chipsy wrote: post #46

Earlier quoted context omitted.

Were you the college roommate I had who suspended the alarm on a string over the bed so that standing was required to turn it off?

I saw an alarm that had a propeller on top of it. The clock made it fly and you had to hunt it down (even if it falls behind the furniture) to shut the alarm.

I've heard of alarms on wheels that run away from you but that takes the cake.

Re: Scaling lessons learned at Dropbox, part 1

#85

Post by jmathai »

batista wrote: post #81
arohner wrote: post #78

Earlier quoted context omitted.

> I also seriously doubt that anyone has ever written a production system of any reasonable complexity and been able to use the exact same ORM code with absolutely any backend (if you have an example please correct me on this). You're entirely right here, because databases are different. For example, (I forget the exact details), "select count(*)..." in MySQL is O(1), but it's O(log n) or O(n) in Postgres, depending…

No, actually the only reason is "its not a language programmers want to use". It is very much non shitty. Its just that lots of programmers, especially OO minded cannot get into its mindset, and use it for what it is, they have to put a lame OO abstraction on top. Functional programmers shoud fare better in this regard (or Prolog programmers, if they still exist). If you really want to abstract it, something like LIN…

I agree. I see SQL similarly to regular expressions. There's a handful of commands which let you do a lot of stuff.

The hard part in SQL is optimization which requires really understanding how the underlying database engine optimizes and executes the query.

Optimizing complex queries is no joke. It's one of the reasons noSQL seems nice at glance. You can do the optimizations by adding lots of indexes or using application logic. In reality, it's a tradeoff for other problems.

Re: Scaling lessons learned at Dropbox, part 1

#86

Post by lucian1900 »

dools wrote: post #53

but I really hate ORM’s and this was just a giant nuisance to deal with I like object relational mapping as a theory (ie. I have an object of type Author which has 1 or more books I can loop over), but I hate ActiveRecord implementations. Eventually, they just end up implementing almost all of SQL but in some arcane bullshit syntax or sequence of method calls that you have to spend a bunch of time learning. I also se…

I've found SQLAlchemy to be very nice, actually. It first provides very basic abstractions on top of SQL, things like defining tables/columns and querying without having to mess around with strings.

That alone is most of the usefulness of SQLAlchemy, as it lets you write subqueries and joins extremely easily.

On top of that, the (optional) ORM is built as models on top of SQLAlchemy's table/relationship API. These models can be queried almost exactly like the raw tables.

Re: Scaling lessons learned at Dropbox, part 1

#87

Post by wizard_2 »

kevinburke wrote: post #50

MySQL has a huge network of support and we were pretty sure if we had a problem, Google, Yahoo, or Facebook would have to deal with it and patch it before we did. :) I am fairly certain Google is running its own (patched) version that's fairly different than the off-the-shelf MySQL.

And hopefully they're pushing important stuff upstream, it wouldn't make sense to not leverage the community.

Re: Scaling lessons learned at Dropbox, part 1

#88

Post by peterwwillis »

eranki wrote: post #72
peterwwillis wrote: post #71

Earlier quoted context omitted.

Are planes in the habit of carrying much more fuel than they need, just in case they will weigh too much when landing, in which case they dump the fuel? No. In the inverse example, we don't put extra load on servers in anticipation that load goes up, just so we can dump the extra load and put it back to where it would have been had we not had extra load. The idea is moronic. It is literally the same as putting lead i…

lol why are you so mad -- perhaps you couldn't make this out through your tears of rage: "Why not just plan ahead? Because most of the time, it was a very abrupt failure that we couldn’t detect with monitoring."

lol why are you retarded? What does monitoring have to do with adding extra load onto your servers? If there's a failure there's a failure. What does adding or removing load have to do with detecting it?

So you have a system, and you have monitoring in place. Let's say the monitors were set up for 1 minute polls, because somebody thought that was a good idea. Suddenly you find out one of your servers is down. Oh noes! There's 45 seconds until the monitor finds this out, which would be horrible.

Since we have doubled the reads on the existing servers, we now no longer have capacity and connections are stacking up. Shit :'( But not to worry! Let's just quickly kill the extra reads - now we have more capacity! Hooray!

Except, if the extra reads weren't happening, they would have already had extra fucking capacity and not had to flip a switch in the first place.

Now you see why i'm mad, bro?

Re: Scaling lessons learned at Dropbox, part 1

#89

Post by peterwwillis »

batista wrote: post #80
peterwwillis wrote: post #63

Earlier quoted context omitted.

Can you imagine if they loaded 1000lbs of lead onto an airplane before they took off, just to see if the plane takes off with the headroom filled? "Oh crap, the plane is falling, let's dump the lead." OP's "extra reads" is dumb because he could have had normal metrics for memcached load and planned to only support like 70% capacity or somesuch, and when load hit that number, he would immediately increase capacity. In…

> Can you imagine if they loaded 1000lbs of lead onto an airplane before they took off, just to see if the plane takes off with the headroom filled? "Oh crap, the plane is falling, let's dump the lead." They actually do this kind of stuff (except for the "lets dump the lead" part), in stress tests, especially in cargo and millitary planes. And they do similar tests not only in aviation, but in most kinds of engineeri…

That's stress and load testing. You don't do that on every single flight. That's the stupid part.

Re: Scaling lessons learned at Dropbox, part 1

#90

Post by peterwwillis »

jaylevitt wrote: post #75
peterwwillis wrote: post #63

Earlier quoted context omitted.

Can you imagine if they loaded 1000lbs of lead onto an airplane before they took off, just to see if the plane takes off with the headroom filled? "Oh crap, the plane is falling, let's dump the lead." OP's "extra reads" is dumb because he could have had normal metrics for memcached load and planned to only support like 70% capacity or somesuch, and when load hit that number, he would immediately increase capacity. In…

If your system is complex, you don't know what your capacity is, or whether the curve has a sharp knee. Think of disk I/O; if 100K simultaneous users keep your disks 50% busy, how many simultaneous users can you actually support? The answer is not 200K; the answer is "it depends".

First of all, it's pointless to monitor how much of your system is in use by 'number of users'. That's not a metric. You look at your iops metrics to figure out how loaded it is. Once you've gathered trending data you can then come up with an average iop load for a given number of users.

Secondly, you should know what your capacity is. Stress testing exists for a reason.

Post reply on HN