Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Uncovering a 24-year-old bug in the Linux Kernel (2021)

engineering.skroutz.gr

Post reply on HN
Print view View original

31–40 of 84 posts

Re: Uncovering a 24-year-old bug in the Linux Kernel (2021)

#31

Post by stjohnswarts »

trasz wrote: post #16
c0mptonFP wrote: post #5

Earlier quoted context omitted.

> We need more people and companies like this, who are willing to go beyond "oh it fails randomly sometimes" and track down the underlying issues. I absolutely disagree. Most capable engineers I know have this urge to go down rabbit holes and fix any issue, this is nothing special. Everyone wants to be the hero that found a bug deep in the stack, make a glorious pull request, and be celebrated in the community. I muc…

This _is_ the meaningful stuff. Engineers might have the urge, but most don’t have the opportunity, because they need to focus on the currently fashionable framework. A good rule of thumb regarding meaningful battles is to ignore everything promoted by companies like Google or Facebook - everything they do is either going to be abandoned in five years, or makes sense only in the context of solving problems nobody els…

seems like something an engineer might fix on their own time if they were feeling feisty about the matter. Something tells me if it went on for 20 years it was an edge case that only very rarely came up and was mostly a non-issue.

Re: Uncovering a 24-year-old bug in the Linux Kernel (2021)

#33

Post by simtel20 »

sponaugle wrote: post #21

This was a cool example of a class of bugs that are both hard to find with no active example, and hard to prevent in complex systems. The optimization that was added many years ago for performance didn't update something that had a use case that was incompatible with not being updated in a very small number of circumstances. It is an interesting thought experiment to consider what kind of tool or automated detection…

It's a great question! Thinking back...

At the time this bug was introduced it would probably have been cost prohibitive to create a test case. We were proud of 100mbit networks, had flaky nics the vendors didn't help maintain much of the time (and which were often broken in hardware) and the filesystem max file size was something like 2tb, and most drives wee're in the handful of gbs. Conceiving of testing for something like this would have been expensive. And none of the big system vendors took Linux seriously then.

Though perhaps flooding zeros across a TCP socket could work, I really think that a kernel hacker would have found a lot of other hardware and driver issues before ever being able to trigger this.

Re: Uncovering a 24-year-old bug in the Linux Kernel (2021)

#34

Post by didgetmaster »

myself248 wrote: post #24
didgetmaster wrote: post #6

As someone who thrives on tracking down rare but annoying bugs in a debugger, I love stories like this. It is not just bugs that cause real failures which can be headaches; but also bugs that just slow things down unexpectantly. They can sometimes go undetected for decades like this one. I wrote an article this past year that talks about silent bugs that slowly eat resources and collectively can be very expensive in…

Okay but where's the bug story? Did I miss the story?

I wrote the article right after I fixed a huge inefficiency problem in a function within my own project. I neglected to give the specifics in the article, but here they are since you asked.

My Didgets tool lets you create pivot tables against relational database tables, even very large ones. For the pivot values, you can choose to just count the occurrence of each value or if it is a number type you can add them up. You can also add up the values in a separate number column. Here is a quick demo video: https://www.youtube.com/watch?v=2ScBd-71OLQ

When adding up numbers in a separate column, I had just a few lines of unnecessary code that ended up being called exponentially. For smaller tables it was barely noticeable, but for tables with 30 million+ rows it really bogged down.

A simple fix to the affected lines caused a certain test against a large table to go from over 10 minutes down to under 20 seconds. The effects of just a few lines of code when applied to a big enough data set can really impact performance. It is the old Einstein equation E=mc2 in effect which is discussed here: https://didgets.substack.com/p/musings-from-an-old-programme...

Re: Uncovering a 24-year-old bug in the Linux Kernel (2021)

#35

Post by rrss »

c0mptonFP wrote: post #5
MatthiasPortzel wrote: post #2

I remember when this was originally posted, but I voted it up again because I think it's such an excellent story, and excellent programming. We need more people and companies like this, who are willing to go beyond "oh it fails randomly sometimes" and track down the underlying issues. => https://news.ycombinator.com/item?id=26102241 Previous Discussion (497 points - 41 comments)

> We need more people and companies like this, who are willing to go beyond "oh it fails randomly sometimes" and track down the underlying issues. I absolutely disagree. Most capable engineers I know have this urge to go down rabbit holes and fix any issue, this is nothing special. Everyone wants to be the hero that found a bug deep in the stack, make a glorious pull request, and be celebrated in the community. I muc…

In my experience, the “oh it fails randomly sometimes” bugs are often in some random dull legacy infrastructure component where there is zero attention or celebration for fixing them, and so engineers tend to tolerate losing a bit of time once a week due to them for years rather than someone spending half a day to fix it for everyone.

Re: Uncovering a 24-year-old bug in the Linux Kernel (2021)

#36

Post by halukakin »

mauvehaus wrote: post #29
halukakin wrote: post #11

Could someone provide link(s) on how regular snapshots of databases can be taken like this? (Googling didn't help much, maybe I'm googling for the wrong keywords.) For me, backing up the database is a few-hour-long process. Restoring it for a developer again is a few hours process. I read about snapshots before but haven't realized they could be this effective.

Because it isn't a backup. They put the database into a quiescent state on disk, take a file system snapshot, let the dbms resume working, and send the snapshot data via rsync. This requires the cooperation of the dbms software to get the on-disk data quiesced. Then your snapshot has to go fast enough that the dbms doesn't end up with too many spinning plates before you let it start writing normally.

Got it. Thank you!

Re: Uncovering a 24-year-old bug in the Linux Kernel (2021)

#37

Post by halukakin »

nick__m wrote: post #22
halukakin wrote: post #11

Could someone provide link(s) on how regular snapshots of databases can be taken like this? (Googling didn't help much, maybe I'm googling for the wrong keywords.) For me, backing up the database is a few-hour-long process. Restoring it for a developer again is a few hours process. I read about snapshots before but haven't realized they could be this effective.

for mariadb : 0) make sure the the database data volume is on lvm or zfs in a sql prompt: 1) BACKUP STAGE START; BACKUP STAGE BLOCK_COMMIT; 2) \! the shell command to take the snapshot 3) BACKUP STAGE END; you can now mount your snapshot, copy it offsite and delete it. The restore procedure is left as an exercise!

Very helpful. Thank you!

Re: Uncovering a 24-year-old bug in the Linux Kernel (2021)

#38

Post by layer8 »

mdaniel wrote: post #30
layer8 wrote: post #26

This is a good case for formal verification.

I struggle because I want to upvote these comments, because that's the world I want to live in. But the opposite side of that coin is who is going to author the incredibly arcane specification of TCP against which any such implementation is formally verified? Maybe TCP stacks are one of the few cases where that make sense, but I'd suspect if it was "worth the cost" it would have already been done

There are certain guarantees you want such a formal specification to give, like for example not getting permanently stuck in some state as with the present bug. You can formalize the proofs for those guarantees and have their correctness machine-checked. Something like TLA+/PlusCal is likely suitable for that.

A formal specification is less ambiguous than a prose specification. Formalizing the TCP specification will, if anything, expose aspects where the specification is unclear, or corner cases where the specification actually leads to unwanted behavior and doesn’t provide the desired guarantees.

So, while you can’t prove that the formal specification matches the prose specification a 100%, you can prove that it provides all the guarantees the original prose specification was aiming for (once you’ve formalized those desired guarantees), which is something you can’t do for the prose specification.

Re: Uncovering a 24-year-old bug in the Linux Kernel (2021)

#39

Post by charcoalhobo »

Love deep dive troubleshooting like this. I haven't heard of systemtap before; looks nice. When I had to troubleshoot a kernel bug [1] I used perf [2] probes which are also really nice for this kind of debugging.

[1] https://www.spinics.net/lists/xdp-newbies/msg01231.html

[2] https://www.brendangregg.com/perf.html

Re: Uncovering a 24-year-old bug in the Linux Kernel (2021)

#40

Post by freedomben »

c0mptonFP wrote: post #5
MatthiasPortzel wrote: post #2

I remember when this was originally posted, but I voted it up again because I think it's such an excellent story, and excellent programming. We need more people and companies like this, who are willing to go beyond "oh it fails randomly sometimes" and track down the underlying issues. => https://news.ycombinator.com/item?id=26102241 Previous Discussion (497 points - 41 comments)

> We need more people and companies like this, who are willing to go beyond "oh it fails randomly sometimes" and track down the underlying issues. I absolutely disagree. Most capable engineers I know have this urge to go down rabbit holes and fix any issue, this is nothing special. Everyone wants to be the hero that found a bug deep in the stack, make a glorious pull request, and be celebrated in the community. I muc…

This opinion is a popular one these days (particularly since it complements the demands of business nicely by maximizing personal/company profit), but it is a big part of the reason why the majority of software these days is so unreliable and buggy. It results in hacks on top of hacks to paper over problems in the lower levels of the abstraction tower that is modern software, and it results in tons of "WTF" bugs that are just accepted and never fixed.
Post reply on HN