The idea of running extra load - it sounds good in theory but I can't help thinking that it's a bit like setting your watch forwards to try and stop being late for things. Eventually you know your watch is 5 minutes fast so start compensating for it. I wonder if this strategy starts to have the same effect - putting fixes off because you know you can pull the extra load before it becomes critical. In the same way you leave for the train a couple of minutes later because you know your watch is actually running fast.
Scaling lessons learned at Dropbox, part 1
11–20 of 92 posts
Re: Scaling lessons learned at Dropbox, part 1
#12Rajiv is awesome, you should listen to him
Says an ex "Product Manager at Dropbox".
Edit: Thanks for the downvotes. My point is, just make it unambiguous to everyone in your comment so we don't have to click through your profile. Context matters. e.g.:
"I was Product Manager at Dropbox and worked with Rajiv (the OP). He's awesome, you should listen to him."
Much better.
Re: Scaling lessons learned at Dropbox, part 1
#13For the record, I use sqlalchemy 0.6.6 regularly under fairly heavy load, and have never had a problem with it. Any 'sqlalchemy bugs' are inevitably coding mistakes on my part.
Yeah, I found that bit quite vague. Are they using SQLAlchemy's object layer, but just not the high level query stuff? Or are they using only the low-level query stuff and nothing else?
I'd love to know more about how their system works, if they are indeed not using an ORM.
Every time I've tried to build something without an ORM, I just end up writing my own shitty one accidentally.
Re: Scaling lessons learned at Dropbox, part 1
#14I noticed that a particular “FUUUCCKKKKKasdjkfnff” wasn’t getting printed where it should have
Why not take the extra half a second to make those random strings meaningful and hidden behind a DEBUG log level?
Re: Scaling lessons learned at Dropbox, part 1
#15Great post, but this part scares me a bit... I think a lot of services (even banks!) have serious security problems and seem to be able to weather a small PR storm. So figure it out if it really is important to you (are you worth hacking? do you actually care if you’re hacked? is it worth the engineering or product cost?) before you go and lock down everything. Just because you can "afford" to be hacked, doesn't mean…
That's an externality; shareholders can sue the board of directors if they find out it is company policy to waste resources on it beyond its potential costs to the company: PR + legal liability + probability of causing new regulation * the cost of adhering to said regulation.
Sucks, but that's capitalism. However, there are a few states now which allow you to have some charitable clauses in your corporate charter.
Re: Scaling lessons learned at Dropbox, part 1
#16I wish he'd left the security advice out.
The whole post was excellent, but all the useful points will now be overshadowed by the armchair quarterbacking about security by people who mostly don't understand that ALL security is a compromise, and it is as important to understand and make deliberate decisions about your security as it is to try to make a secure system in the first place.
Re: Scaling lessons learned at Dropbox, part 1
#17Can you explain the nginx/HAproxy config a little more?
agree - i see a lot of start ups putting haproxy behind nginx for load balancing but i have never figured out why they wouldnt just stick with nginx. does anyone have an example of how the configuration looks on github?
Re: Scaling lessons learned at Dropbox, part 1
#18Rajiv is awesome, you should listen to him
Says an ex "Product Manager at Dropbox". Edit: Thanks for the downvotes. My point is, just make it unambiguous to everyone in your comment so we don't have to click through your profile. Context matters. e.g.: "I was Product Manager at Dropbox and worked with Rajiv (the OP). He's awesome, you should listen to him." Much better.
Even more reason to listen to him.
Re: Scaling lessons learned at Dropbox, part 1
#19Rajiv is awesome, you should listen to him
Says an ex "Product Manager at Dropbox". Edit: Thanks for the downvotes. My point is, just make it unambiguous to everyone in your comment so we don't have to click through your profile. Context matters. e.g.: "I was Product Manager at Dropbox and worked with Rajiv (the OP). He's awesome, you should listen to him." Much better.
I've found working with people to be a reasonable way of finding out whether their opinions are worth listening to...
Re: Scaling lessons learned at Dropbox, part 1
#20Fabulous post. Thanks for writing. One point it misses though is to test your backup strategy often. When you scale fast things break very often and it's good to be in practice of restoring from backups every now and then.
Just started reading a book called "High Performance MySQL" and in one of the early pages, the following advice appears: "It's an excellent idea to run a realistic load simulation on a test server and then literally pull the power plug. The firsthand experience of recovering from a crash is priceless. It saves nasty surprises later." Same goes for testing network connectivity and failover. I can't tell you how many t…
Reading High Performance MySQL as well. Loving it so far!