Live data from Hacker News

Log4j: The pain just keeps going

thenewstack.io

121–130 of 291 posts

Re: Log4j: The pain just keeps going

#121

Earlier quoted context omitted.

It's funny how every Go project seems to statically compile in the same libraries. If just one of them has a serious vuln, we're talking nearly every Go project in the world having to be patched and recompiled, or upgraded with potentially breaking changes. And as we know from Log4j, that can be incredibly difficult. Just finding all the affected Go apps will be a nightmare, and patching will not be as simple as "rep…

> nearly every Go project in the world having to be patched and recompiled Is that a particular problem? What's the difficulty with recompiling?

Imagine you are a random person in charge of a computer. You hear there's this big vuln affecting Go programs.

1. How do I find out which programs on my system are Go programs? I have a lot of programs scattered all over. Someone will have to come up with a method (probably search through $PATH for files, run `strings` on it, grep for "Go", pray that's enough) to identify them.

2. They don't list a website, so I have to google the name of every program and find their website.

3. How do I tell which ones are affected by the vuln? I can hope their websites tell me, but maybe they're no longer maintained, or they haven't updated their READMEs.

4. If they don't provide a patched version (most probably won't), I might have to upgrade, which may break things (probably will), assuming they even have an upgradeable version that isn't affected.

5. If I can't upgrade and have to patch, am I even a software developer and know how to do that?

6. If I luckily happen to be one, what system compiled this app 8 years ago, with what dependencies, with what version of Go, etc? Can I recreate that system and learn their build tools to finally patch & recompile? (I am not a Go developer and it has taken me days to figure out how to get different Go programs to compile, for reasons I don't understand)

7. Once I apply the patch how do I test it? (Log4j had multiple iterations of patches/mitigations, some didn't work)

If I can't find the source code, can't upgrade it, & can't patch it, I am just stuck with a vulnerable version until I can somehow replace the program. For the programs I was able to identify as vulnerable, anyway.

Log4j was "easy" compared to this, because you could just find all the jars on your system and unzip them, find a file, replace it, zip it back up. Log4j was actually a nightmare though, and Go apps will be worse.

Re: Log4j: The pain just keeps going

#122

If you automatically update your dependencies all the time, you will constantly get new bugs, issues and sometimes even malware. If you don't update your dependencies all the time, you will be vulnerable to old bugs and issues. The current software engineering paradigm has no meaningful answer to this , no matter what "security experts" tell you. In a sane industry this realization would lead to a change of the parad…

We actually are running into this exact problem with running production ML workloads and despite anyone's claims it's not a solved problem. In fact, this log4j mess is a cake walk in comparison.

The issue: ML workloads are very sensitive to the exact version of a framework/library used and these frameworks/libraries are not stable despite what their semantic version says.

So what do you do when you have a python vulnerability? Best case, an update "just works" but in many cases you either need to retrain the model (can be extremely expensive or inconvenient), rewrite it if APIs changed, or hunt down a difference in model output if some downstream dependency broke something. Stability of software in the python world is abysmal.

Re: Log4j: The pain just keeps going

#123

Earlier quoted context omitted.

It's funny how every Go project seems to statically compile in the same libraries. If just one of them has a serious vuln, we're talking nearly every Go project in the world having to be patched and recompiled, or upgraded with potentially breaking changes. And as we know from Log4j, that can be incredibly difficult. Just finding all the affected Go apps will be a nightmare, and patching will not be as simple as "rep…

> nearly every Go project in the world having to be patched and recompiled Is that a particular problem? What's the difficulty with recompiling?

The advantage of being able to build to a binary is that you can then distribute the binary directly instead of distributing source and forcing people to rebuild it.

When 98% of your users only have the binary and not the source code it means 98% of your users won't even be aware that they need a new version that isn't vulnerable.

Remember the vulnerability disclosure will be something like "Log4Go has a vulnerability!!", not that the binary apps you've downloaded have a vulnerability. So if it affects 200 projects, every one of those projects will also have to publish vulnerability notifications independently.

And frankly 80% of those projects will likely be abandoned or at least neglected after a couple of years, leaving many people with vulnerable binaries they don't even know to check for vulnerabilities.

In other words, they very advantage of Go (easy to distribute binaries directly!) is a security disadvantage. If you instead installed the app using a package manager, the package manager could at least notify you that you have a package with a security issue. But that's nothing new; security and ease-of-use are often at odds.

Re: Log4j: The pain just keeps going

#124

If you automatically update your dependencies all the time, you will constantly get new bugs, issues and sometimes even malware. If you don't update your dependencies all the time, you will be vulnerable to old bugs and issues. The current software engineering paradigm has no meaningful answer to this , no matter what "security experts" tell you. In a sane industry this realization would lead to a change of the parad…

Yup, you’re outsourcing your decisions in exchange for not doing the work. You have to understand your risks and needs. Your boring business systems that will live for 50 years need to have boring system level libraries or stuff you maintain. Your fast time to market or non-critical systems are optimized by speed and cost. If your service depends on a bunch of downstream stuff, your processes need to support upgrade…

System level libraries are a huge risk. It makes updating much harder leading to many developers just foregoing updates unless absolutely necessary. Libraries your software uses should be decoupled as much as possible from system level libs.

Re: Log4j: The pain just keeps going

#125
> Welcome to how we use open source code. That’s a big reason Software Bills of Materials (SBOM) have become essential for securing the software supply chain.

I tend to do this, but it's pretty damn easy for me. I have a "2-level-dependency" rule that I generally go by, these days. i.e., Each dependency can have only one dependency, and that dependency can't have any others.

The main way that I deal with it, is to write all my own dependencies, and I don't count toolsets, standard libraries, or platform frameworks (for better, or for ill).

I really, really want dependencies to work. They are a great idea, and I have been promoting modular architecture for decades.

It's just that we are in a "wild west" situation, where we can't trust dependencies.

I'm tired of hearing the phrase "It's a solved problem," with a reference to a dependency.

I don't have the answer. I just have my own rule, which elicits scorn from most modern programmers (I'm painted as a luddite, refusing to "get with the times," or as a "tinfoil-hatter").

WFM. YMMV.

Re: Log4j: The pain just keeps going

#126

Had to deal with some automated scan tool telling a client "you have log4j - you're vulnerable!" when... we didn't. A project was including a dependency on SLF4J and that project pulled in slf4j-log4j (IIRC). But it was not configured, wasn't compiled in to the final jar, and ... even it was, it was very old log4j (1.1 or 1.2 IIRC?). The vulnerability didn't affect that older version. We had to spend a week back and…

Typically there's a way to suppress specific warnings in systems like these. In your company's situation, I would look at moving away from a scanning system if it didn't allow overrides like this.

So far this is the best approach I've found. The scanning tools rarely include that ability but if you build tooling around them you can maintain exclusion lists, for particular vulnerabilities, library/version pairs, etc.

Unfortunately it does mean there's no getting around having someone manually deal with false positives.

Re: Log4j: The pain just keeps going

#127

Sometimes people ask me why I'm so skeptical about software delivery techniques involving bundling and static compilation. This is a thing that people sometimes ask.

It's not a question of bundling / static builds, it's a question of dependencies and updates. If you build a static binary from maintained inputs (say, nix, Gentoo+portage, whatever) in CI and automatically distribute it, you're golden. If you completely separate a library out and... heck, make the user supply their own copy! but don't monitor it for updates, don't notify users, don't ship patches ever, then you're i…

The beauty of the unbundled approach is, though, that you don't need to rebuild anything except the vulnerable thing. It scales. Having to rebuild the whole world, not so much, even if you can do it. Maybe you have access to an infinite supply of zero-cost energy, but I don't, and neither does the world at large.

When they find a critical bug in a Go module or a Rust crate, you usually end up with a litany of packages that had to be updated as well because everyone and their crab relies on that crate/module.

Count in release managers (or whatever that's called at your shop) who pin all dependencies to the most exact minor version before shipping, because they have learned to do it like a mantra without actually understanding all the "whys" of it and just keep religiously applying that hammer everywhere.

People keep complaining about how distributions work and how the traditional package management sucks all the time (of thousands complaining, I estimate that maybe 20 people actually know what they are talking about and thus are qualified to whine), but they somehow keep forgetting that those tools exist for a reason and that they are about to rediscover it the hard way. Chesterton's fence and all that.

Re: Log4j: The pain just keeps going

#128

If you automatically update your dependencies all the time, you will constantly get new bugs, issues and sometimes even malware. If you don't update your dependencies all the time, you will be vulnerable to old bugs and issues. The current software engineering paradigm has no meaningful answer to this , no matter what "security experts" tell you. In a sane industry this realization would lead to a change of the parad…

> So the exact software you compile will depend on whether or not you can connect to a 3d party service? Do you understand the actual implications of this?

This is already true with the vast majority of software using any an online software repository, e.g. Go, NodeJS, Java, Rust, etc.

Re: Log4j: The pain just keeps going

#130

Earlier quoted context omitted.

packageblabla-1.2.3.deb is a single file.

That will only work on Debian. And that "single file" requires also the libfoo-1.5.deb file (must be >=1.5, not 1.4 and not higher than 2.0!) etc.

The thing about "single file deployment" is that it shouldn't have external requirements, otherwise it's not a single file deployment anymore.
Post reply on HN