Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Log4j: The pain just keeps going

thenewstack.io

Post reply on HN
Print view View original

21–30 of 291 posts

Re: Log4j: The pain just keeps going

#21

Post by thayne »

jacobyoder wrote: post #12

Had to deal with some automated scan tool telling a client "you have log4j - you're vulnerable!" when... we didn't. A project was including a dependency on SLF4J and that project pulled in slf4j-log4j (IIRC). But it was not configured, wasn't compiled in to the final jar, and ... even it was, it was very old log4j (1.1 or 1.2 IIRC?). The vulnerability didn't affect that older version. We had to spend a week back and…

I thought Log4j 1.1 was vulnerable, just to a lesser extent, and there wasn't a lot of information on it because 1.x is end of life

Re: Log4j: The pain just keeps going

#22

Post by hedora »

shagie wrote: post #14
0xbadcafebee wrote: post #5

Earlier quoted context omitted.

It's funny how every Go project seems to statically compile in the same libraries. If just one of them has a serious vuln, we're talking nearly every Go project in the world having to be patched and recompiled, or upgraded with potentially breaking changes. And as we know from Log4j, that can be incredibly difficult. Just finding all the affected Go apps will be a nightmare, and patching will not be as simple as "rep…

I'm not as familiar with Go... I do know that we use a dependency scan that looks at the docker images and recognizes vulnerable versions of various Java libraries. With Go, can you look at a compiled binary in an image on the artifact repo and identify what libraries it used and if those libraries have vulnerabilities? (and it appears that my question was answered in a sibling comment)

Java's "statically link then scan" approach doesn't really work though. (Some sibling threads explain why.)

Even Linux distributions where rebuilding the world is easy (like Debian) insist on breaking dependencies out into shared libaries, since tracking ad hoc vendored dependencies is just too difficult.

For example, sometimes people play dynamic loading and renaming tricks to allow multiple versions of the same Java library to coexist. That can usually be detected by vulnerability scanners, but not always.

Re: Log4j: The pain just keeps going

#23

Post by Nursie »

jacobyoder wrote: post #12

Had to deal with some automated scan tool telling a client "you have log4j - you're vulnerable!" when... we didn't. A project was including a dependency on SLF4J and that project pulled in slf4j-log4j (IIRC). But it was not configured, wasn't compiled in to the final jar, and ... even it was, it was very old log4j (1.1 or 1.2 IIRC?). The vulnerability didn't affect that older version. We had to spend a week back and…

Been in similar situations, it is not fun.

"You have a vulnerable version of netty"

"No, we don't, that's a false positive"

"No see, the tool says you need netty 4.x and that's version 1.x, you need to update"

"OK, but the tool is wrong, it's just picking up anything with 'netty' in the name, and that component is a wrapper around netty for some other thing, it only goes up to 1.8"

"You have to update it to 4.x, this is a vulnerability"

"Do you understand this isn't part of the thing you're concerned about"

"I am only concerned about shipping a product without vulnerabilities"

"Well this isn't one, because it's a false positive, here's the CVE, here's what it applies to, here's the link to this package on maven central, it's not part of netty, back off"

"But it's insecure, the tool says so"

We went round and round for about 3 hours until I told the guy (an infosec 'pro') to leave me alone until he'd figured out how to do his job... he came back to me the next morning for another 3 hour slack argument along the exact same lines at which point I told him to leave me alone permanently and communicate through management if he had anything to say. I'm not massively impressed with infosec as a profession after a few similar encounters.

Re: Log4j: The pain just keeps going

#24

Post by hedora »

bgro wrote: post #16

When are they going to release Log5j? Or even Log4k to address these issues?

The log4j author already did this about a decade ago. No one could understand log4j back then, so the replacement is much, much simpler, and API compatible for 99+% of use cases.

Few projects seem to have bothered moving to it, probably becuase log4j is such a collosal pain to set up or modify.

Re: Log4j: The pain just keeps going

#25

Post by er4hn »

jewel wrote: post #20
skybrian wrote: post #11

Earlier quoted context omitted.

The situation for Go seems to have gotten a bit better with Go 1.18. Go binaries now include the versions of all the modules in them, and this can be listed with the "go version" command. That won't help you rebuild the binary, but at least you can find them.

That's fantastic. Since Go has gone all-in on static compiling, it'd be awesome if they made it possible to update a dependency without recompiling (or even having the source code for) the rest of the program. In addition to making it easier to do security updates in situations like log4j, it'd also make it easy to comply with LGPL license terms.

Wouldn't that require the ABI, or at least the API, to be the same across versions of the affected library? That feels like it may be possible with enough tooling, but still very hard to enforce.

Re: Log4j: The pain just keeps going

#26

Post by hedora »

upupandup wrote: post #8

It's probable that there are more log4j type of exploits thats just waiting to be discovered. It's also becoming hard to tell which are private enterprises vs state sponsored cyber attack. It just seems like there's no solution. If Java is this bad imagine how much more attack surface a npm heavy project. Not only that the seeming hardware knowledge of state sponsored attackers seem sophisticated enough to gain acces…

We could learn to write auditably correct software.

Reflections on Trusting Trust explains why writing correct software is inadequate:

https://dl.acm.org/doi/10.1145/358198.358210

Re: Log4j: The pain just keeps going

#27

Post by hyperman1 »

I never understood where log4j 2 was supposed to fit in the ecosystem.

* We had log4j 1, which was good enough for most use cases and everywhere.

* We had slf4j/logback, which was from the same authors, and broke backward compatibility but gave us a fundamentally better design.

* We had java.util.logging, which while braindead was builtin to Java and hence available everywhere.

*We had commons logging, which was never supposed to escape from tomcat, but got overused everywhere anyway.

All of this was more logging frameworks than was healthy for 1 language, and there are some more smaller ones.

Then apache decides to put new people on log4j, do a backward incompatible v2 design that nevertheless is worse than slf4j. Why?

Re: Log4j: The pain just keeps going

#28

Post by d4mi3n »

Nursie wrote: post #23
jacobyoder wrote: post #12

Had to deal with some automated scan tool telling a client "you have log4j - you're vulnerable!" when... we didn't. A project was including a dependency on SLF4J and that project pulled in slf4j-log4j (IIRC). But it was not configured, wasn't compiled in to the final jar, and ... even it was, it was very old log4j (1.1 or 1.2 IIRC?). The vulnerability didn't affect that older version. We had to spend a week back and…

Been in similar situations, it is not fun. "You have a vulnerable version of netty" "No, we don't, that's a false positive" "No see, the tool says you need netty 4.x and that's version 1.x, you need to update" "OK, but the tool is wrong, it's just picking up anything with 'netty' in the name, and that component is a wrapper around netty for some other thing, it only goes up to 1.8" "You have to update it to 4.x, this…

Security engineer here. This is sadly common. The grim fact seems to be that we have a dearth of information security analysts with engineering experience. If they don't have "Engineer" or similar in their title, odds are they haven't had the pleasure of building or maintaining a non-trivial unit of software over more than a quarter or so.

That said, I've seen the inverse problem: engineering staff that either don't understand how their dependencies are managed (not unreasonable for less experienced teams, NPM for example has a tendency for huge dependency trees and this can be a hard problem to manage for a big project) or whom are for a plethora of reasons incentivized to push back on requests for work they don't perceive as important.

The middle ground here requires trust between both parties (InfoSec and Engineering org functions). Sadly, not all security programs are well managed and not all engineering teams have mature practices.

If you have time or capacity, you can get a lot of leverage here by educating your security analysts on how your technology works (e.g. how the build system functions, how your languages of choice share and bundle code). Trust can be built between engineering and infosec teams by educating where possible. Done well such practices can up-level both sides of these discussions and save you stress in the long run.

Re: Log4j: The pain just keeps going

#29

Post by commandlinefan »

Nursie wrote: post #23
jacobyoder wrote: post #12

Had to deal with some automated scan tool telling a client "you have log4j - you're vulnerable!" when... we didn't. A project was including a dependency on SLF4J and that project pulled in slf4j-log4j (IIRC). But it was not configured, wasn't compiled in to the final jar, and ... even it was, it was very old log4j (1.1 or 1.2 IIRC?). The vulnerability didn't affect that older version. We had to spend a week back and…

Been in similar situations, it is not fun. "You have a vulnerable version of netty" "No, we don't, that's a false positive" "No see, the tool says you need netty 4.x and that's version 1.x, you need to update" "OK, but the tool is wrong, it's just picking up anything with 'netty' in the name, and that component is a wrapper around netty for some other thing, it only goes up to 1.8" "You have to update it to 4.x, this…

I sometimes feel that my job description could be "arguing with people who feel passionately about something they don't understand".

Re: Log4j: The pain just keeps going

#30

Post by bragr »

thayne wrote: post #21
jacobyoder wrote: post #12

Had to deal with some automated scan tool telling a client "you have log4j - you're vulnerable!" when... we didn't. A project was including a dependency on SLF4J and that project pulled in slf4j-log4j (IIRC). But it was not configured, wasn't compiled in to the final jar, and ... even it was, it was very old log4j (1.1 or 1.2 IIRC?). The vulnerability didn't affect that older version. We had to spend a week back and…

I thought Log4j 1.1 was vulnerable, just to a lesser extent, and there wasn't a lot of information on it because 1.x is end of life

Log4j 1.x is vulnerable, but to different things and not as badly.
Post reply on HN