Dribbble is Back With a Day of Data Lost

#1

Post by 19_ploT » Wed, May 15, 2013, 11:53 AM UTC

Dribbble is Back With a Day of Data Lost
blog.dribbble.com

Re: Dribbble is Back With a Day of Data Lost

#2

Post by jakerocheleau » Wed, May 15, 2013, 12:20 PM UTC

It could have been a lot worse, for sure. I'm just happy they resolved the issue and it's back online with minimal damage.

Building & maintaining a website is always a learning lesson because there are so many different areas to study.

Re: Dribbble is Back With a Day of Data Lost

#3

Post by hijinks » Wed, May 15, 2013, 12:44 PM UTC

Not trying to be an ass here or anything but something doesn't add up. I understand the memory corruption idea but I wouldn't think that would replicate to the other postgresql server. So am I right in thinking there was no slave ever here?

Re: Dribbble is Back With a Day of Data Lost

#4

Post by pfg » Wed, May 15, 2013, 12:55 PM UTC

hijinks wrote: post #3

Not trying to be an ass here or anything but something doesn't add up. I understand the memory corruption idea but I wouldn't think that would replicate to the other postgresql server. So am I right in thinking there was no slave ever here?

I don't know exactly how the replication in postgres works, but I can think of scenarios where bit flipping in RAM gets propagated to the slave(s) (e.g., newly-generated data which currently resides only in RAM gets corrupted and then fsync'd to disk, at which point it gets replicated).

The real question is, why on earth would they use non-ECC memory on their database server.

Re: Dribbble is Back With a Day of Data Lost

#5

Post by pilif » Wed, May 15, 2013, 1:21 PM UTC

hijinks wrote: post #3

Not trying to be an ass here or anything but something doesn't add up. I understand the memory corruption idea but I wouldn't think that would replicate to the other postgresql server. So am I right in thinking there was no slave ever here?

It really depends on how you have configured replication and what the exact issue was. Postgres replication either works by directly streaming the WAL archive or by manually shipping older archived WAL files. If these files were corrupted on the master, then the slave would also get the corrupted files.

Now the files (and when streaming directly, the packets) have a header containing some metadata and the actual WAL log entries have a fixed formatting, so it's likely that the slave would have detected this corruption (unless you were really unlucky which would then easily replicate the corruption over to the slave).

But that would just lead to the slave stopping to actually replicate. Unless you watch your clients whether they are still ok, streaming from the master and the replication lag is reasonably low, you would not notice the replication stopping. When you fail over, you get to the state which the database was in when the first corrupted packet arrived.

So either you check your slaves, or you use two-phase commit, ensuring that your data has reached the slaves, but that has some serious performance costs.

BTW: I would assume this was far more likely an issue with their storage, not with RAM.

Re: Dribbble is Back With a Day of Data Lost

#6

Post by replicatorblog » Wed, May 15, 2013, 2:05 PM UTC

There is a lot of arm chair sysadmining going on, but remember, the team that built Dribbble is essentially 4 total people, 2.5 engineers, working with no outside funding. The fact that they've built the designer's equivalent of Github and keep it running as smoothly as it does is amazing. It's fine to provide suggestions, but this is a minor blip in an otherwise impeccable record of performance.

Re: Dribbble is Back With a Day of Data Lost

#7

Post by benjaminwootton » Wed, May 15, 2013, 2:25 PM UTC

Perhaps we will start to see posts about people abandoning Postgres and moving back to MongoDB, completing the circle?

Re: Dribbble is Back With a Day of Data Lost

#8

Post by hijinks » Wed, May 15, 2013, 2:39 PM UTC

pilif wrote: post #5

hijinks wrote: post #3

Not trying to be an ass here or anything but something doesn't add up. I understand the memory corruption idea but I wouldn't think that would replicate to the other postgresql server. So am I right in thinking there was no slave ever here?

It really depends on how you have configured replication and what the exact issue was. Postgres replication either works by directly streaming the WAL archive or by manually shipping older archived WAL files. If these files were corrupted on the master, then the slave would also get the corrupted files. Now the files (and when streaming directly, the packets) have a header containing some metadata and the actual WAL…

thanks for the explanation

Re: Dribbble is Back With a Day of Data Lost

#9

Post by NelsonMinar » Wed, May 15, 2013, 2:45 PM UTC

Good disclosure on the part of Dribbble.

I have some sympathy; I've seen a Linux server randomly corrupt its file cache, no idea why. Google's study found 8% of DIMMs experienced at least one memory error a year. If you can't trust your RAM, what can you trust? http://research.google.com/pubs/pub35162.html

Re: Dribbble is Back With a Day of Data Lost

#10

Post by manmal » Wed, May 15, 2013, 2:50 PM UTC

NelsonMinar wrote: post #9

Good disclosure on the part of Dribbble. I have some sympathy; I've seen a Linux server randomly corrupt its file cache, no idea why. Google's study found 8% of DIMMs experienced at least one memory error a year. If you can't trust your RAM, what can you trust? http://research.google.com/pubs/pub35162.html

In ECC we trust.