Scaling lessons learned at Dropbox, part 1
eranki.tumblr.com
Scaling lessons learned at Dropbox, part 1
1–10 of 92 posts
Re: Scaling lessons learned at Dropbox, part 1
#2[deleted]
Re: Scaling lessons learned at Dropbox, part 1
#3Fabulous post. Thanks for writing.
One point it misses though is to test your backup strategy often. When you scale fast things break very often and it's good to be in practice of restoring from backups every now and then.
Re: Scaling lessons learned at Dropbox, part 1
#4Nice, love the idea of running with extra load to predict breaking points.
Re: Scaling lessons learned at Dropbox, part 1
#5Rajiv is awesome, you should listen to him
Re: Scaling lessons learned at Dropbox, part 1
#6For the record, I use sqlalchemy 0.6.6 regularly under fairly heavy load, and have never had a problem with it. Any 'sqlalchemy bugs' are inevitably coding mistakes on my part.
Re: Scaling lessons learned at Dropbox, part 1
#7Fabulous post. Thanks for writing. One point it misses though is to test your backup strategy often. When you scale fast things break very often and it's good to be in practice of restoring from backups every now and then.
Just started reading a book called "High Performance MySQL" and in one of the early pages, the following advice appears:
"It's an excellent idea to run a realistic load simulation on a test server and then literally pull the power plug. The firsthand experience of recovering from a crash is priceless. It saves nasty surprises later."
Same goes for testing network connectivity and failover. I can't tell you how many times I've heard things like "The automatic recovery _should_ have kicked in but..."
Having a recovery procedure and backup strategy is completely different from having actually restored a backup and recovered from a failure.
Re: Scaling lessons learned at Dropbox, part 1
#8Fabulous post. Thanks for writing. One point it misses though is to test your backup strategy often. When you scale fast things break very often and it's good to be in practice of restoring from backups every now and then.
Thanks! Good point. We actually repurposed our offsite database recovery to clone slaves off a master (after LVM was no longer performing), so that's a great way to get more testing in.
Re: Scaling lessons learned at Dropbox, part 1
#9Can you explain the nginx/HAproxy config a little more?
Re: Scaling lessons learned at Dropbox, part 1
#10Great post, but this part scares me a bit...
I think a lot of services (even banks!) have serious security problems and seem to be able to weather a small PR storm. So figure it out if it really is important to you (are you worth hacking? do you actually care if you’re hacked? is it worth the engineering or product cost?) before you go and lock down everything.
Just because you can "afford" to be hacked, doesn't mean you shouldn't take all the steps necessary to proactively protect your data. In the end, security is not about you, it is about your users. This is exactly the type of attitude that leads to all the massive breaches we have been seeing recently. Sure your company is "hurt" with bad PR, but really your users are the ones who are the real victims. You should consider their risk (especially with something as sensitive as people's files!) before you consider your own company's well being.
Edit: formatting