I've frequently run into problems with Postgres streaming replication that looks exactly like the issues encountered here. I was never able to find the source of the issue, so I'm very curious if this fix will also fix the issues I encountered.
Uncovering a 24-year-old bug in the Linux Kernel (2021)
71–80 of 84 posts
Re: Uncovering a 24-year-old bug in the Linux Kernel (2021)
#72Earlier quoted context omitted.
> We need more people and companies like this, who are willing to go beyond "oh it fails randomly sometimes" and track down the underlying issues. I absolutely disagree. Most capable engineers I know have this urge to go down rabbit holes and fix any issue, this is nothing special. Everyone wants to be the hero that found a bug deep in the stack, make a glorious pull request, and be celebrated in the community. I muc…
This opinion is a popular one these days (particularly since it complements the demands of business nicely by maximizing personal/company profit), but it is a big part of the reason why the majority of software these days is so unreliable and buggy. It results in hacks on top of hacks to paper over problems in the lower levels of the abstraction tower that is modern software, and it results in tons of "WTF" bugs that…
When you need working software at the end of the sprint, there's often no time to carefully control for all edge cases.
Re: Uncovering a 24-year-old bug in the Linux Kernel (2021)
#73I've frequently run into problems with Postgres streaming replication that looks exactly like the issues encountered here. I was never able to find the source of the issue, so I'm very curious if this fix will also fix the issues I encountered.
Please keep us updated.
Re: Uncovering a 24-year-old bug in the Linux Kernel (2021)
#74This is a good case for formal verification.
I struggle because I want to upvote these comments, because that's the world I want to live in. But the opposite side of that coin is who is going to author the incredibly arcane specification of TCP against which any such implementation is formally verified? Maybe TCP stacks are one of the few cases where that make sense, but I'd suspect if it was "worth the cost" it would have already been done
It seems to me that we should design for (formal) verification. Much like we should design for testability. But am noob, so what do I know?
That said, my quick search shows some academic efforts to formally verify QUIC, both in whole and in parts.
I would hope that bespoke (boutique?) TCP replacements, like Homa (specifically for datacenters), are verified as part of the design process. From a quick scan, I gleaned that Homa, and other aspirants, are simulated, compared, and benchmarked against each other. Maybe that's sufficient.
https://homa-transport.atlassian.net/wiki/spaces/HOMA/overvi...
Re: Uncovering a 24-year-old bug in the Linux Kernel (2021)
#75Earlier quoted context omitted.
> We need more people and companies like this, who are willing to go beyond "oh it fails randomly sometimes" and track down the underlying issues. I absolutely disagree. Most capable engineers I know have this urge to go down rabbit holes and fix any issue, this is nothing special. Everyone wants to be the hero that found a bug deep in the stack, make a glorious pull request, and be celebrated in the community. I muc…
I think this was well prioritized; they struggled with the issue at times, found a temporary workaround, but when that workaround stöd being efficient and the bug hit them everyday, they decided to track down the source. Then they reported upstream, it was reproduced, and someone patched it, and rolled out new, fixed kernels. That is a perfect example of how things works and should work. They contributed to the commu…
[Author here] Your comment resonates with me. Indeed, we had been working around the issue for a number of years before we decided to tackle it head-on and even then we set a deadline for debugging and had a fallback plan as well: if we didn't manage to figure something out in a couple of days, we would switch transports or ditch rsync altogether.
In the end I believe we struck a good balance between time spent and result achieved: we gathered enough information for someone more familiar with the code to identify and fix the root cause without the need for a reproducer. We could have spent more time trying to patch it ourselves (and to be honest I would probably have gone down that route 10 years ago), but it would be higher risk in terms of both, time invested and patch quality.
Finally, I'm always encouraging our teams to contribute upstream whenever possible, for three reasons:
a) minimizing the delta vs upstream pays off the moment you upgrade without having to rebase a bunch of local patches
b) doing a good write-up and getting feedback on a fix/patch/PR from people who are more familiar with the code will help you understand the problem at hand better (and usually makes you a better engineer)
c) everyone gets to benefit from it, the same way we benefit from patches submitted by others
Re: Uncovering a 24-year-old bug in the Linux Kernel (2021)
#76Which kernel version has this patch?
it says it was put in to 5.10-rc1; however i noticed weird network issues beginning in june or july. I am wondering if it was put in to the kernel for 5.10 and then left "off by default" until this year.
5.15.32 was around the kernel where i noticed the issues start. If i'm just connected via SSH and streaming video from a LAN server, everything is great. if i go on youtube.com (or whatever), i'll get "network unreachable" on ping within a minute. I swapped NICs to make sure my NIC wasn't the issue; now youtube doesn't cause this issue, but i tested rsync oddly enough and the NIC goes AWOL after a few gigabytes of transfer. I have to physically unplug and replug the NIC (or a reboot if it was PCI).
I haven't had time to track down why, but it has stayed with newer kernels, too: 5.15.41, 5.15.59 also have this issue. I compiled 5.15.72 last night but i haven't rebooted yet.
Re: Uncovering a 24-year-old bug in the Linux Kernel (2021)
#77Earlier quoted context omitted.
It's popular because these war stories you find in blog posts are pure survivorship bias. If I'd let every fucking team member go on an exploratory bug hunt whenever they feel like it (hint: that would be always) we would never get anything done. What if they don't find anything? Is this issue really worth 2 weeks of dev time? That's 15k down the drain for a senior engineer, if not more.
From a short-term business perspective, sure, it doesn't make financial sense. As a user of software, though, I want someone to fix the bug. I want software that doesn't have bugs. So let me repeat my original statement. We need more like this that are willing to spend engineer time fixing bugs, even upstream bugs in open source projects. Instead of prioritizing shoving half-baked features out the door for next week'…
[Author here] Even from a short-term business perspective, it actually might make sense to fix things and contribute upstream. When you face a problem with something you built using FOSS, essentially you have three choices:
- Work around it, most likely creating technical debt inside your organization in the process
- Invest the time to fix it yourself
- Pay someone else to fix it for you (e.g. the original authors via a support contract)
None of these options is for free, and which one is the most cost-effective depends largely on the complexity of the issue at hand, the skillset and availability of the people involved and the criticality of the impacted system.
Re: Uncovering a 24-year-old bug in the Linux Kernel (2021)
#78I had never heard about systemtap, but it sounds magical, and extremely useful. I would actually need this in my day job, but for a regular but huge embedded C++ application.
I have no idea how to "hot-patch" a C++ application though, are there libraries for this?
Re: Uncovering a 24-year-old bug in the Linux Kernel (2021)
#79Which kernel version has this patch?
it says it was put in to 5.10-rc1; however i noticed weird network issues beginning in june or july. I am wondering if it was put in to the kernel for 5.10 and then left "off by default" until this year. 5.15.32 was around the kernel where i noticed the issues start. If i'm just connected via SSH and streaming video from a LAN server, everything is great. if i go on youtube.com (or whatever), i'll get "network unreac…
>I am wondering if it was put in to the kernel for 5.10 and then left "off by default" until this year.
Any idea how we might check that?
>I haven't had time to track down why, but it has stayed with newer kernels, too: 5.15.41, 5.15.59 also have this issue. I compiled 5.15.72 last night but i haven't rebooted yet.
I'm fairly certain we've seen it on `Linux 5.15.0-1017-aws x86_64`.
Re: Uncovering a 24-year-old bug in the Linux Kernel (2021)
#80Earlier quoted context omitted.
it says it was put in to 5.10-rc1; however i noticed weird network issues beginning in june or july. I am wondering if it was put in to the kernel for 5.10 and then left "off by default" until this year. 5.15.32 was around the kernel where i noticed the issues start. If i'm just connected via SSH and streaming video from a LAN server, everything is great. if i go on youtube.com (or whatever), i'll get "network unreac…
>I am wondering if it was put in to the kernel for 5.10 and then left "off by default" until this year. Any idea how we might check that? >I haven't had time to track down why, but it has stayed with newer kernels, too: 5.15.41, 5.15.59 also have this issue. I compiled 5.15.72 last night but i haven't rebooted yet. I'm fairly certain we've seen it on `Linux 5.15.0-1017-aws x86_64`.
I could, i guess, try to load an old kernel. i tend to clean out /boot, and if the package manager deletes my kernel source all i have is a .config. I would lean toward .config diffs, but it is possible that my distribution kernel maintainers made specific patches that made this networking stuff "optional" - as is spectre and rowhammer and other speculative execution mitigation "optional". I just know my torment with networking started about 5 months ago at the closest. I didn't upgrade kernels between February and May 31st of 2022, so if it happened in there, i wouldn't be able to track that down without some sort of distro-specific archive of kernel releases.