Love Your Bugs
akaptur.com
Love Your Bugs
1–10 of 19 posts
Re: Love Your Bugs
#2If you like bug postmortems like this you may also enjoy the Jepsen presentations by @aphyr. A video summary is available here:
https://www.youtube.com/watch?v=eSaFVX4izsQ
and if you want to get into the weeds with any of them these are largely published publicly on aphyr.com; see for example:
There's a lot of "oh, here's how this dirty read from the underlying system became a much bigger bug in the system we built on top of it!" but what I like most about Kyle is that he is generally pretty great about having an attitude of "these are actually really hard problems and it's not surprising that there are implementation bugs when you let me mess with the clocks and cut off nodes and whatnot."
Re: Love Your Bugs
#3If you like bug postmortems like this you may also enjoy the Jepsen presentations by @aphyr. A video summary is available here: https://www.youtube.com/watch?v=eSaFVX4izsQ and if you want to get into the weeds with any of them these are largely published publicly on aphyr.com; see for example: http://jepsen.io/analyses There's a lot of "oh, here's how this dirty read from the underlying system became a much bigger bu…
Also, u/luu started a list of interesting bug post-mortems. It hasn't been updated in awhile but contains a lot of the classics (many which have made the rounds on HN):
Re: Love Your Bugs
#4My take on why bugs are good for us (mostly because we learn from them):
https://henrikwarne.com/2012/10/21/4-reasons-why-bugs-are-go...
Re: Love Your Bugs
#5Good article, although I'm a bit surprised at the difficulty the author proposes for rolling back the log bug. Why not simply serve 500s to 95% of your clients? That way, you get logs sent to you gradually.
Re: Love Your Bugs
#6> Bitflips are real!
And yet some people still claim that they don’t need ECC memory.
Re: Love Your Bugs
#7Good article, although I'm a bit surprised at the difficulty the author proposes for rolling back the log bug. Why not simply serve 500s to 95% of your clients? That way, you get logs sent to you gradually.
I think they expected it wouldn't have much effect, since the client was also updated to delete the old (corrupted) log. Because the logs were always deleted after a success, and updates started with the oldest, the corrupted log would necessarily become the oldest.
By the time the DDoS was in effect, the corrupted logs had been deleted by the client. They would now always succeed (even with the old server code, or old client code) until they got a new corrupted log.
Re: Love Your Bugs
#8Good article, although I'm a bit surprised at the difficulty the author proposes for rolling back the log bug. Why not simply serve 500s to 95% of your clients? That way, you get logs sent to you gradually.
I think they expected it wouldn't have much effect, since the client was also updated to delete the old (corrupted) log. Because the logs were always deleted after a success, and updates started with the oldest, the corrupted log would necessarily become the oldest. By the time the DDoS was in effect, the corrupted logs had been deleted by the client. They would now always succeed (even with the old server code, or o…
Yes, I'm saying that they could have just served 500s to clients (even ones with regular log files), which would have backed off and retried later. Essentially what the "chillout" method does too, but it doesn't sound like the author had considered it.
Re: Love Your Bugs
#9Earlier quoted context omitted.
I think they expected it wouldn't have much effect, since the client was also updated to delete the old (corrupted) log. Because the logs were always deleted after a success, and updates started with the oldest, the corrupted log would necessarily become the oldest. By the time the DDoS was in effect, the corrupted logs had been deleted by the client. They would now always succeed (even with the old server code, or o…
Yes, I'm saying that they could have just served 500s to clients (even ones with regular log files), which would have backed off and retried later. Essentially what the "chillout" method does too, but it doesn't sound like the author had considered it.
I'm guessing the answer is that inserting a feature to serve 500 to a fraction of the incoming requests would have required a similar amount of effort as just enabling the chillout feature.
Re: Love Your Bugs
#10I really doubt they were legitimate bit flips and not just software bugs, to be honest.
The likelihood that you’ve seen multiple flips on the same piece of data .. sounds like a typical threading bug.