I quickly got distracted by the F-16 flight manual in the "More from this user" sidebar. Downloading...
Oh Shit: How to Break a Large Website (and how not to) -- PDF version
11–20 of 30 posts
Re: Oh Shit: How to Break a Large Website (and how not to) -- PDF version
#12Am I the only person who's shocked by the idea that one of Scribd's developers was repeatedly taking down the website by pushing code live without testing it or understanding the queries it ran? I guess maybe he's just exaggerating for entertainment value... It's good to see that he's sharing his mistakes for others to learn from, and going into detail on why things went wrong. However, it's really depressing to not…
Half of these are things that I could see doing quite easily -- for example, although I happen to know that "select count(*) from users where lower(users.user_name) = 'patio11'" will result in a full table scan on MySQL, that category of mistake is easy to make and has bitten me a time or three. And it is easy to miss if your testing methodology doesn't include "Now run it against a data set about as big as the production DB." So that would be a pattern to look out for both when coding and, critically, when designing processes to make sure this sort of thing doesn't happen to you.
Then there's "code that throws an exception anywhere you execute it every time you execute it was pushed to production." This points to a rather different type of process error.
Re: Oh Shit: How to Break a Large Website (and how not to) -- PDF version
#13Am I the only person who's shocked by the idea that one of Scribd's developers was repeatedly taking down the website by pushing code live without testing it or understanding the queries it ran? I guess maybe he's just exaggerating for entertainment value... It's good to see that he's sharing his mistakes for others to learn from, and going into detail on why things went wrong. However, it's really depressing to not…
My vote goes to entertainment value, because the stories don't completely add up. When he was implementing the case-insensitive logins he would, of course, first have to run a query to see if there are any users whose login names collide without case sensitivity. That query would run (according to the slides) really slowly, because it would have to essentially do something like:
SELECT a.id, b.id, a.username FROM users AS a, users AS B
WHERE a.id != b.id AND a.username = b.username AND
BINARY(a.username) != BINARY(b.username)
Edit: I realize that most likely the username column had a UNIQUE index already, making this query unnecessary.
--
In his example of find_in_batches where he has to make some changes that applies to all users he makes it sound like the User.all is the big problem because it does a `SELECT * FROM users`. But even when you break up the work in parts you have to be careful, because by the time you're done processing a few of the existing users may have changed already. So as your database becomes too large to do your updates in one big atomic transaction block you almost always have to follow a process like this:
1. find set of rows that match the problem (in batches)
2. apply the (expensive) fix/migration on those rows
3. go back to 1 until the result set is empty
4. do (1) and (2) in a big transaction (but since there are only a few rows, that's fine)
Of course there are a hundred ways to upgrade a live database safely, but there always must be some kind of process in place otherwise your database is going to go down all the time.
Re: Oh Shit: How to Break a Large Website (and how not to) -- PDF version
#14I'm surprised any website running at scale is letting an ORM generate SQL queries.
Just out of interest - why? Do you not trust developers to keep an eye out for bad generated queries, or do you have a problem with ORM generated SQL in general?
Re: Oh Shit: How to Break a Large Website (and how not to) -- PDF version
#15Am I the only person who's shocked by the idea that one of Scribd's developers was repeatedly taking down the website by pushing code live without testing it or understanding the queries it ran? I guess maybe he's just exaggerating for entertainment value... It's good to see that he's sharing his mistakes for others to learn from, and going into detail on why things went wrong. However, it's really depressing to not…
[deleted]
Re: Oh Shit: How to Break a Large Website (and how not to) -- PDF version
#16Am I the only person who's shocked by the idea that one of Scribd's developers was repeatedly taking down the website by pushing code live without testing it or understanding the queries it ran? I guess maybe he's just exaggerating for entertainment value... It's good to see that he's sharing his mistakes for others to learn from, and going into detail on why things went wrong. However, it's really depressing to not…
Half of these are things that I could see doing quite easily -- for example, although I happen to know that "select count(*) from users where lower(users.user_name) = 'patio11'" will result in a full table scan on MySQL, that category of mistake is easy to make and has bitten me a time or three. And it is easy to miss if your testing methodology doesn't include "Now run it against a data set about as big as the produ…
I know you're making a general point here, but for the example given, I think it's generally best to refresh your dev data from production regularly (maybe after each large release). That way you're always testing on data 'close' to the current production state, which would help you avoid any data-related issues like this.
Re: Oh Shit: How to Break a Large Website (and how not to) -- PDF version
#17Somewhat OT: is it just me or does anyone else find it utterly ironic that scribd, a site dedicated to helping people avoid dealing with the PDF browser plugin mess, is intentionally going through the trouble of converting their source document to PDF just so they can display it in their viewer? There's dogfooding, and then there's this. I'm not sure what to make of it.
Why is it ironic? I wouldn't talk about the trouble of converting their source document to PDF, since PDF already is the standard format for publishing presentations. Every scientist I know who makes presentations in PowerPoint exports to PDF when transmitting them to other people, mainly to minimize version conflicts and people ripping off their content. PS and PDF are also the standard output of LaTeX.
scribd exists because PDF browser plugins kind of suck, not because PDF itself sucks.
Re: Oh Shit: How to Break a Large Website (and how not to) -- PDF version
#18Earlier quoted context omitted.
Half of these are things that I could see doing quite easily -- for example, although I happen to know that "select count(*) from users where lower(users.user_name) = 'patio11'" will result in a full table scan on MySQL, that category of mistake is easy to make and has bitten me a time or three. And it is easy to miss if your testing methodology doesn't include "Now run it against a data set about as big as the produ…
I know you're making a general point here, but for the example given, I think it's generally best to refresh your dev data from production regularly (maybe after each large release). That way you're always testing on data 'close' to the current production state, which would help you avoid any data-related issues like this.
I think that is a good idea if you're able to get away with it. Scribd can. We can't do it at my day job.
There are hundreds of different ways to break the personal information privacy law if we screw up anonymization of test data. For example, supposing we just take the naive approach and overwrite all names, addresses, emails, phone numbers, etc etc etc. Should be fine, right? Except, uh oh, the data is still personally identifiable: the data will tell you that a female student who took CS103 and English 101 last year was given a semester of medical leave. If a copy of that dataset leaks and someone in the department realizes "Hey wait, the only person in that double major is Hanako... medical leave... Hanako was pregnant last year?!?", then our company just made the front page news, we made our customer look horrible (and likely owe them and Hanako several tens of millions of yen in we're-so-sorry money), and we just broke the information privacy law something fierce.
Incidentally, engineers not treating test data with the same "This CD is nuclear waste" precautions we treat the production data set is a frequent cause of breaches like this. Somebody decides to work from home for the day, gets his laptop stolen, bam front page news. I nearly got in severe trouble for leaving a printout of the student roster on the printer fifteen feet from a door somebody could tailgate through -- the only thing that saved my keister was that I could show that the student roster I printed out was fake. (Lesson learned about producing good test data: don't produce too good test data.)
Re: Oh Shit: How to Break a Large Website (and how not to) -- PDF version
#19Somewhat OT: is it just me or does anyone else find it utterly ironic that scribd, a site dedicated to helping people avoid dealing with the PDF browser plugin mess, is intentionally going through the trouble of converting their source document to PDF just so they can display it in their viewer? There's dogfooding, and then there's this. I'm not sure what to make of it.
Why is it ironic? I wouldn't talk about the trouble of converting their source document to PDF, since PDF already is the standard format for publishing presentations. Every scientist I know who makes presentations in PowerPoint exports to PDF when transmitting them to other people, mainly to minimize version conflicts and people ripping off their content. PS and PDF are also the standard output of LaTeX. scribd exist…
The epic shittiness of Acrobat Reader 7 is just how scribd got traction -- they don't really give a shit about PDF viewers -- their raison d'têre is to be the Youtube of documents.
Their Flash viewer is terrible (doesn't work at all on Linux) and their site is largely hostile to the idea of you actually reading the content. I had an involved email conversation with one of the founders after slamming their iPaper flash viewer here before, but nothing ultimately came of it.
Choose life. Choose Google's pure-JS document viewer: http://docs.google.com/viewer
Re: Oh Shit: How to Break a Large Website (and how not to) -- PDF version
#20Somewhat OT: is it just me or does anyone else find it utterly ironic that scribd, a site dedicated to helping people avoid dealing with the PDF browser plugin mess, is intentionally going through the trouble of converting their source document to PDF just so they can display it in their viewer? There's dogfooding, and then there's this. I'm not sure what to make of it.
Why is it ironic? I wouldn't talk about the trouble of converting their source document to PDF, since PDF already is the standard format for publishing presentations. Every scientist I know who makes presentations in PowerPoint exports to PDF when transmitting them to other people, mainly to minimize version conflicts and people ripping off their content. PS and PDF are also the standard output of LaTeX. scribd exist…
> PS and PDF are also the standard output of LaTeX.
Erm, I thought that DVI was the default output medium.