There's a talk about Dropbox scaling at http://www.stanford.edu/class/ee380/winter-schedule-20112012... .
Scaling lessons learned at Dropbox, part 1
51–60 of 92 posts
Re: Scaling lessons learned at Dropbox, part 1
#52The idea of running extra load - it sounds good in theory but I can't help thinking that it's a bit like setting your watch forwards to try and stop being late for things. Eventually you know your watch is 5 minutes fast so start compensating for it. I wonder if this strategy starts to have the same effect - putting fixes off because you know you can pull the extra load before it becomes critical. In the same way you…
it sounds good in theory
Don't you mean "it sounds good in practice"? This entire post is about practical experience.
I don't think this is like setting your watch forward 5 mins. I think it's more like RAID. When you get a warning that one of your drives has died, you know you have to get in and replace it.
Depending on how critical the machine is, the cost of getting to the data centre etc. you might leave now, in the middle of the night and drive like a bat out of hell, or you might leave it til next week when you'll be in there anyway.
Either way you know your risk just went up a hell of a lot. Depending on how risk averse you are, you will act accordingly.
Re: Scaling lessons learned at Dropbox, part 1
#53but I really hate ORM’s and this was just a giant nuisance to deal with
I like object relational mapping as a theory (ie. I have an object of type Author which has 1 or more books I can loop over), but I hate ActiveRecord implementations. Eventually, they just end up implementing almost all of SQL but in some arcane bullshit syntax or sequence of method calls that you have to spend a bunch of time learning.
I also seriously doubt that anyone has ever written a production system of any reasonable complexity and been able to use the exact same ORM code with absolutely any backend (if you have an example please correct me on this). This barely even works with something like PDO in PHP which is a bare bones abstraction across multiple SQL backends.
When it comes down to it, the benefits of ActiveRecord are all but dead on about the third day of development. The data mapper pattern adopted by SQLAlchemy (et. al.) takes all of the shitness of ActiveRecord and adds mind bending complexity to it.
SQL is easy to learn and very expressive. Why try and abstract it?
I spent years working with an ActiveRecord ORM I wrote myself in my feckless youth and thought that it was the answer to the world's problems. I didn't really understand why it was so terrible until I did a large project in Django and had to use someone else's ORM.
When I really analysed it, there were only three things that I really wanted out of an ORM:
1) Make the task of writing complex join statements a bit less tedious
2) Make the task of writing a sub-set of very basic where clauses slightly less tedious
3) Obviate the need for me to detect primary key changes when iterating over a joined result set to detect changes in an object (for example, looping over a list of Authors and their Books)
To that end, I wrote this:
https://github.com/iaindooley/PluSQL
It's written in PHP because I like and use PHP but it's a very simple pattern that I would like to see elaborated upon/taken to other languages as I think it provides just the bare minimum amount of functionality to give some real productivity gains without creating a steep learning curve, performance trade-off or any barrier to just writing out SQL statements if that's the fastest way to solve the problem at hand.
Re: Scaling lessons learned at Dropbox, part 1
#54Earlier quoted context omitted.
So, so true. When the problem occurs, it's pretty quick for the guy who needs to fix it at 2am, to find where it exploded in the code base, while the original developer is (maybe) passed out in a bar somewhere. Not much else matters. He could of just done :( x 10 and had the same result. The main thing is, it's easily traceable!
Sure, for fatal errors, random (greppable) strings aren't so bad, but the OP made it sound like his FFFFFFFFFFFUUUUUU message was expected all the time rather than an exception. If you're going to print something all the time in normal operation, make it meaningful.
That isn't how I took it. I read it as: there was an error scenario that was clearly happening, which should have resulted in a particular log message. Except it wasn't, which meant that something was racing.
Re: Scaling lessons learned at Dropbox, part 1
#55MySQL has a huge network of support and we were pretty sure if we had a problem, Google, Yahoo, or Facebook would have to deal with it and patch it before we did. :) I am fairly certain Google is running its own (patched) version that's fairly different than the off-the-shelf MySQL.
You mean using the Google Mysql5patches[1]?
[1] http://code.google.com/p/google-mysql-tools/wiki/Mysql5Patch...
Re: Scaling lessons learned at Dropbox, part 1
#56I believe that the section on "The security-convenience tradeoff" is fundamentally flawed.
A username and password represent a pair. Neither one has meaning in terms of authentication without the other.
Take the example where I have forgotten my username (JohnGB), but try with what I think it is (Say JohnB), and enter the correct password for my actual username. The system would then tell me that my username is fine, but that my password isn't. From then on, I would be trying to reset the password for a different user as the system has already told me that my username was correct.
Please, for the sake of sane UX, don't do this!
Re: Scaling lessons learned at Dropbox, part 1
#57I believe that the section on "The security-convenience tradeoff" is fundamentally flawed. A username and password represent a pair. Neither one has meaning in terms of authentication without the other. Take the example where I have forgotten my username (JohnGB), but try with what I think it is (Say JohnB), and enter the correct password for my actual username. The system would then tell me that my username is fine,…
No way, sir. Saying 'you entered the wrong password' in that case is not any more confusing than the ambiguous error that says 'you got one of them wrong but I'm not gonna tell you which.' most reset password systems are keyed to your email address anyway.
Re: Scaling lessons learned at Dropbox, part 1
#58The idea of running extra load - it sounds good in theory but I can't help thinking that it's a bit like setting your watch forwards to try and stop being late for things. Eventually you know your watch is 5 minutes fast so start compensating for it. I wonder if this strategy starts to have the same effect - putting fixes off because you know you can pull the extra load before it becomes critical. In the same way you…
[deleted]
Re: Scaling lessons learned at Dropbox, part 1
#59'Even memcached, which is the conceptually simplest of these technologies and used by so many other companies, had some REALLY nasty memory corruption bugs we had to deal with, so I shudder to think about using stuff that’s newer and more complicated'
Does anyone know what memory corruption bugs they are referring to?
Re: Scaling lessons learned at Dropbox, part 1
#60Earlier quoted context omitted.
I only have one alarm. If it fails I am late. I found out that depending on complex systems work against you. Once I had three wake up alarms, at different points at the bedroom. Didn't work. Being late is lame. Suffering its consequences is the best teacher one can have.
Were you the college roommate I had who suspended the alarm on a string over the bed so that standing was required to turn it off?
I saw an alarm that had a propeller on top of it. The clock made it fly and you had to hunt it down (even if it falls behind the furniture) to shut the alarm.