Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Log4j: The pain just keeps going

thenewstack.io

Post reply on HN
Print view View original

161–170 of 291 posts

Re: Log4j: The pain just keeps going

#161

Post by chrisseaton »

0xbadcafebee wrote: post #121
chrisseaton wrote: post #103

Earlier quoted context omitted.

> nearly every Go project in the world having to be patched and recompiled Is that a particular problem? What's the difficulty with recompiling?

Imagine you are a random person in charge of a computer. You hear there's this big vuln affecting Go programs. 1. How do I find out which programs on my system are Go programs? I have a lot of programs scattered all over. Someone will have to come up with a method (probably search through $PATH for files, run `strings` on it, grep for "Go" , pray that's enough) to identify them. 2. They don't list a website, so I hav…

> How do I find out which programs on my system are Go programs?

Well how are these programs getting onto your system?

Presumably via some package manager, or an image builder?

Don't those manage the update for you?

If you're running code that you don't know who wrote, with no source code, that you can't patch, that nobody is taking responsibility for, then yeah you've got issues. But you had issues already if that's the predicament you're in. Some of this applies to Java too - obfuscation breaks the fixes you described using!

Re: Log4j: The pain just keeps going

#162

Post by dotnet00 »

xmprt wrote: post #139
alfalfasprout wrote: post #122

Earlier quoted context omitted.

We actually are running into this exact problem with running production ML workloads and despite anyone's claims it's not a solved problem. In fact, this log4j mess is a cake walk in comparison. The issue: ML workloads are very sensitive to the exact version of a framework/library used and these frameworks/libraries are not stable despite what their semantic version says. So what do you do when you have a python vuln…

Is this an issue of code stability or do these libraries have leaky abstractions that cause issues with models trained on previous version, because the library implementation changed and since ML has low explainability, it's hard to tell how the change impacts the model performance? eg. making a library run faster or use a slightly different parameter internally causes some floating point imprecision that cascades in…

There's also the issue that subtle bugs are a lot harder to find in ML since often the model just adapts (although potentially with worse performance). This then breaks models when the framework updates.

One recent example that comes to mind is that PyTorch now disables Ampere GPUs' TF32 units by default since there were some hard to find edge cases where the matrix multiply results were completely wrong compared to the expected result. This went mostly unnoticed except in a few cases where users were trying something a bit more sensitive to such issues.

I had a similar issue in one of my own models where there was an error in a normalization layer's implementation in Tensorflow (it was computing the average over the wrong axis iirc). It only became noticeable to me when I came back to retrain the model a few months later with an updated Tensorflow version and found that the results from training were not even comparable.

Re: Log4j: The pain just keeps going

#163

Post by ptx »

smarx007 wrote: post #156
vbezhenar wrote: post #88

Earlier quoted context omitted.

Log4j2 does have some legitimate improvements over logback. These days you don't need anything fancy, just spew logs into stdout and something will capture it to your ELK, so those loggers seem obsolete indeed. Java need standard logging with slightly improved ergonomics and that's about it.

> Java need standard logging with slightly improved ergonomics That's exactly what JDK9+ java.lang.System.Logger is. See https://stackoverflow.com/questions/59976828/difference-betw...

It seems to be mainly intended for use internally in the JDK, though. From the JEP[1]: "It is not a goal to define a general-purpose interface for logging. The service interface contains only the minimal set of methods that the JDK needs for its own usage."

And from the motivation section: "The proposed service enables applications to configure the JDK to use the same logging framework as the application: It would only need to provide an implementation of the service that returns platform loggers that wrap the loggers of the preferred logging framework."

[1] https://openjdk.org/jeps/264

Re: Log4j: The pain just keeps going

#164

Post by ClumsyPilot »

gnulinux wrote: post #151
ClumsyPilot wrote: post #115

Earlier quoted context omitted.

'software written in garbage languages' well, back to the mainframe it is then

Quite the opposite, we have very safe languages now in 2022. Any software that's not written in Rust or Haskell has to explain the trade-off of being written in an unsafe language. Of course the industry will still use unsafe languages, but we lack the awareness that we're intentionally writing garbage software in garbage languages that should be treated as radioactive material in any context where safety matters. If…

We've had safe lnaguages for decades - for example Ada

https://www.adacore.com/about-ada

Re: Log4j: The pain just keeps going

#165

Post by layer8 »

hyperman1 wrote: post #27

I never understood where log4j 2 was supposed to fit in the ecosystem. * We had log4j 1, which was good enough for most use cases and everywhere. * We had slf4j/logback, which was from the same authors, and broke backward compatibility but gave us a fundamentally better design. * We had java.util.logging, which while braindead was builtin to Java and hence available everywhere. *We had commons logging, which was neve…

> Then apache decides to put new people on log4j, do a backward incompatible v2 design that nevertheless is worse than slf4j. Why?

Because the original Log4j author wanted to do his own thing for a Log4j successor (slf4j/logback), and the Apache Log4j maintainers also had ideas about a next-gen framework and wanted to do that.

That’s Open Source, everyone can do their own thing if they like, and some things will get adopted more than others. Log4j2 got the most adoption because they had the Log4j “brand” and the Apache backing and more company-backed(?) maintainers.

I don’t consider java.util.logging to be braindead. It can in principle be used as a logging facade like slf4j, but not too many do that, and Log4j 1 had come first and kept that momentum.

The real mistake was for Java to not have a standard logging API earlier. (It took six years after Java 1.0.)

Re: Log4j: The pain just keeps going

#166

Post by spc476 »

tekchip wrote: post #33
hedora wrote: post #17

Earlier quoted context omitted.

Once you realize this sort of shallow, automated audit exists to grease the wheels of some security theater operation, this sort of back and forth makes sense. False positives don't waste the security team's budget, and are the only surefire way to justify ongoing expenditures on the scanning tool.

So somehow false positives negate all the actual positives caught and corrected? The only true solution is what? Manual audit of everything by some perfect human security practitioner? I suppose the same applies to automated development tools then. I will concede there probably are some firms out there acting poorly that way. When aren't there? Humans sigh . But by in large automation and the problems inherent are re…

Back in the late 90s, I was working at a small web hosting company (take note). One day, a 500+ page report of a recent PCI compliancy check landed on my desk. It was nothing but "OMFG! DOMAIN1 RESPONDED TO PING! YOU WILL BE PWOWNED! OMFG! DOMAIN1 HAS DNS RECORDS! YOU WILL BE PWONED! OMFG! DOMAIN1 HAS A WEB SERVER RUNNING! YOU WILL BE PWONED!". Over and over again. For every domain we have. Complete and utter garbage report.

Better---just summarize the IPs scanned and report back which services were found running on said IPs. Then in an appendix, list why each service is (or might be) problematic. "Ping? Attackers might be able to figure out your network topology and that is bad because blah blah blah blah."

But 500 pages of this automatic breathless garbage? Utter trash.

Re: Log4j: The pain just keeps going

#167

Post by dotnet00 »

jacobyoder wrote: post #12

Had to deal with some automated scan tool telling a client "you have log4j - you're vulnerable!" when... we didn't. A project was including a dependency on SLF4J and that project pulled in slf4j-log4j (IIRC). But it was not configured, wasn't compiled in to the final jar, and ... even it was, it was very old log4j (1.1 or 1.2 IIRC?). The vulnerability didn't affect that older version. We had to spend a week back and…

I recall that NVIDIA had to release an update to their CUDA Toolkit around when the Log4j vulnerability was announced. They weren't using it but the file was being distributed, presumably for a similar reason. I'm guessing they might've had similar complaints coming in.

Re: Log4j: The pain just keeps going

#168

Post by stonemetal12 »

gnulinux wrote: post #151
ClumsyPilot wrote: post #115

Earlier quoted context omitted.

'software written in garbage languages' well, back to the mainframe it is then

Quite the opposite, we have very safe languages now in 2022. Any software that's not written in Rust or Haskell has to explain the trade-off of being written in an unsafe language. Of course the industry will still use unsafe languages, but we lack the awareness that we're intentionally writing garbage software in garbage languages that should be treated as radioactive material in any context where safety matters. If…

These Log4j issues aren't real hardcore exploits. They wrote a logger that could run arbitrary code, then were surprised when attackers figured out how to run arbitrary code.

It isn't a language problem, it is a developer practices problem. Developers code like security isn't a thing then are confused when their programs aren't secure.

Re: Log4j: The pain just keeps going

#169

Post by paulmd »

paulmd wrote: post #100
mrbonner wrote: post #7

At this point I’m asking if it is ok to go nuclear: just slab aspectJ compiler as the last step and do comptime patch the log4j remote logging API. Have anyone thought of this? Btw who would use remote logging to begin with?

> Btw who would use remote logging to begin with? It's not about remote logging like "we log to somewhere else" but rather about including information from remote sources (specifically JDNI) in your log messages. If you are ingesting logs from a bunch of microservices, it would obviously be desirable to know which instance you're in, and just include that in the logger pattern. Or if you have multiple containers in a…

Also, btw, just for performance reasons - I wouldn't put a JNDI call into a log message anyway - even calling to somewhere local, it's not entirely free, and logging is highly performance-sensitive. I'm not sure if it's HTTP-based or what, maybe it's memory-based if local, but whatever it is, doing it thousands of times a second is wasteful.

It's not a "get the container name" call, it's a generic JNDI call, those values could change and the output probably can't be cached... so just for performance's sake I would get those values at application startup and then put them into the Mapped Diagnostic Context, which can be accessed via log4j pattern formatters. That way you're not calling out for every log message.

Re: Log4j: The pain just keeps going

#170

Post by taeric »

SomeCallMeTim wrote: post #117
BrainVirus wrote: post #37

If you automatically update your dependencies all the time, you will constantly get new bugs, issues and sometimes even malware. If you don't update your dependencies all the time, you will be vulnerable to old bugs and issues. The current software engineering paradigm has no meaningful answer to this , no matter what "security experts" tell you. In a sane industry this realization would lead to a change of the parad…

>"Use a build tool to do dependency checks!" > >And then what? > >"Only update libraries if they have vulnerabilities." > >So, should this be a manual process where I have to dig through obscure warnings every time I build something? > >"No, you can automate it!" > > Congrats, you've just outsourced the decision making process about which versions of libraries to use to some 3d party. You've also made your build proc…

I don't see how you have really fixed the break much. Especially if you can "push back" and say no to an update, you are just setting yourself up for when the update is even harder to accept. (That is, the further behind on an update you are, the harder to take it. Typically.)

And then there is the "I want the fix, but I don't want the added attack vector of features that comes with it."

Post reply on HN