Live data from Hacker News

Log4j: The pain just keeps going

thenewstack.io

71–80 of 291 posts

Re: Log4j: The pain just keeps going

#71
post #23

Had to deal with some automated scan tool telling a client "you have log4j - you're vulnerable!" when... we didn't. A project was including a dependency on SLF4J and that project pulled in slf4j-log4j (IIRC). But it was not configured, wasn't compiled in to the final jar, and ... even it was, it was very old log4j (1.1 or 1.2 IIRC?). The vulnerability didn't affect that older version. We had to spend a week back and…

Been in similar situations, it is not fun. "You have a vulnerable version of netty" "No, we don't, that's a false positive" "No see, the tool says you need netty 4.x and that's version 1.x, you need to update" "OK, but the tool is wrong, it's just picking up anything with 'netty' in the name, and that component is a wrapper around netty for some other thing, it only goes up to 1.8" "You have to update it to 4.x, this…

I run a security team in a large company.

What you described is usually the result of some "consulting company" (in our case big ones) that drop stuff with zero actual knowledge.

Every year we "review" these with the board and it is annoying as fuck.

Recently they got a new manager who understands we are in the same boat. They have to provide a report with some findings and I need to have a secure environment. We talked this over and suddenly the board presentation was cool.

As for internal teams, I have a few extraordinary people. They are very young and I had to coach them on how the vulnerability is actually critical or not depending on the context, data, etc.

So not all cybersecurity teams suck :)

Re: Log4j: The pain just keeps going

#72
post #25
post #20

Earlier quoted context omitted.

That's fantastic. Since Go has gone all-in on static compiling, it'd be awesome if they made it possible to update a dependency without recompiling (or even having the source code for) the rest of the program. In addition to making it easier to do security updates in situations like log4j, it'd also make it easy to comply with LGPL license terms.

Wouldn't that require the ABI, or at least the API, to be the same across versions of the affected library? That feels like it may be possible with enough tooling, but still very hard to enforce.

The library being patched needs to use the same ABI, yes. So you can usually only patch the same version of a library, not upgrade it.

Afaik, the ELF format allows you to replace sections in a binary. The code (or data) is packed as "Sections" and mapped to memory locations. You use a tool to unpack the ELF sections and replace them. The application accesses those sections of memory and uses whatever ABI the application and library are designed to interact with. As long as you patch a section with the same ABI that an app/library used before, nobody will notice. And the ELF ABI never changes. But of course, not every system uses ELFs.

But all this depends on the compiler/linker/etc and how they generate sections and store them in the binary (for example, is it all compiled into one blob-section, or does it store libraries as separate sections, which might be inefficient?). I somehow doubt Go does things in such a way to allow this kind of patching. They probably generate a giant blob full of random functions and record versions in a header without the ability to extricate a whole library.

Re: Log4j: The pain just keeps going

#73

Does anyone know if log4net the .net version of log4j is effected?

Hey, English is not my first language. Is "effected" here the right word to use or would "affected" be the correct term? I see English speaking people use these two interchangeably so much online that I am starting to doubt what I learned in my English courses.

Re: Log4j: The pain just keeps going

#74
post #23

Earlier quoted context omitted.

Been in similar situations, it is not fun. "You have a vulnerable version of netty" "No, we don't, that's a false positive" "No see, the tool says you need netty 4.x and that's version 1.x, you need to update" "OK, but the tool is wrong, it's just picking up anything with 'netty' in the name, and that component is a wrapper around netty for some other thing, it only goes up to 1.8" "You have to update it to 4.x, this…

I sometimes feel that my job description could be "arguing with people who feel passionately about something they don't understand".

Like I usually say, everyone is a doctor and a football coach, and recently a security expert.

I am coming to a point in my 25+ years career where I stopped to argue. I just tell them that they are wrong but I do not care until they put the company on danger.

Unfortunately this leaves me with fat powerpoint presentations which well to be the modern documentation of a company.

Re: Log4j: The pain just keeps going

#76

If you automatically update your dependencies all the time, you will constantly get new bugs, issues and sometimes even malware. If you don't update your dependencies all the time, you will be vulnerable to old bugs and issues. The current software engineering paradigm has no meaningful answer to this , no matter what "security experts" tell you. In a sane industry this realization would lead to a change of the parad…

Better idea: stop logging.

It's not just logging. L4J is so extensible that people have used it for all kinds of things, way waaaaaayyyyy away from just logs. So disabling logs won't necessarily cut it.

I am no expert. I ended up indexing all the open source kruft I use to hold this ship of fools together, then verified that the Log4J pieces were definitely disabled with a bunch of monitoring while I tossed stuff at it. I did mention I am not an expert, right? I am sure there is a more Pro way to do this.

Re: Log4j: The pain just keeps going

#77

I never understood where log4j 2 was supposed to fit in the ecosystem. * We had log4j 1, which was good enough for most use cases and everywhere. * We had slf4j/logback, which was from the same authors, and broke backward compatibility but gave us a fundamentally better design. * We had java.util.logging, which while braindead was builtin to Java and hence available everywhere. *We had commons logging, which was neve…

> Then apache decides to put new people on log4j, do a backward incompatible v2 design that nevertheless is worse than slf4j. Why?

slf4j itself isn't a logging framework. It's a facade to logging frameworks.

Simple Logging Facade for Java ( https://www.slf4j.org )

It needs a logging framework behind it - log4j, log4j2, logback, commons, JUL.

The question is "why do log4j2?"

Logback went from the log4j1.x path ( https://logback.qos.ch )

Log4j2 has a lot of features that weren't present when the project started ( https://en.wikipedia.org/wiki/Log4j#Apache_Log4j_2 ).

There is a licensing difference between Logback (LGPL) and Log4jx (Apache Commons).

Re: Log4j: The pain just keeps going

#78
post #33
post #17

Earlier quoted context omitted.

Once you realize this sort of shallow, automated audit exists to grease the wheels of some security theater operation, this sort of back and forth makes sense. False positives don't waste the security team's budget, and are the only surefire way to justify ongoing expenditures on the scanning tool.

So somehow false positives negate all the actual positives caught and corrected? The only true solution is what? Manual audit of everything by some perfect human security practitioner? I suppose the same applies to automated development tools then. I will concede there probably are some firms out there acting poorly that way. When aren't there? Humans sigh . But by in large automation and the problems inherent are re…

I was in this industry for a while (I did 5 years full time for a security company).

> I will concede there probably are some firms out there acting poorly that way.

This is a hilarious take. Here's mine: The vast majority of these firms are here for liability. They do not provide security, they provide security theater so that if/when something goes wrong, the client can claim to have followed best practices, and that it's not their fault.

This style of theater is RAMPANT in the industry. Literally most of them come in with some version of a shitty automated tool, often with false positives in the 95%+ range, and then you check off checkboxes to make legal happy, and ensure that your insurance (or your customer's insurance) will pay if you get compromised.

This is not limited to small companies, but is instead mostly how large banks, credit firms, and large enterprises work.

The entire damn show is for liability, and half of these "Security professionals" can't do anything other than read the message coming out of their tool. They are utterly incompetent when it comes to applying logic to figure out whether a specific use of a "risky" tool/language feature a genuine problem, vs a god damned fucking regex match from their tool on something completely unrelated.

I went in as a naive young dev, I came out with ZERO respect for this industry, and a healthy dose of skepticism around anything these folks are saying.

Security researchers? Generally fine (although there's a new breed of them that simply submits inane non-vulnerabilities over and over to attempt to get bug bounties from large companies)

Security advice from open source devs? Generally top notch, listen to it.

Security advice from that contractor your company paid? Expect to have 95% false positives, and they'll miss the 2 places you know actually have issues. But it's ok because you'll check all the boxes on their sheet and legal is happy.

---

I'm waiting for insurance companies to wise up and stop covering companies who are breached. Prices are already shooting way up, since it turns out security theater does a very bad job stopping real threats, and that's what this industry is right now. Might was well be the TSA of software, gonna frisk you real good, and then flunk every fucking test.

Re: Log4j: The pain just keeps going

#79

If you automatically update your dependencies all the time, you will constantly get new bugs, issues and sometimes even malware. If you don't update your dependencies all the time, you will be vulnerable to old bugs and issues. The current software engineering paradigm has no meaningful answer to this , no matter what "security experts" tell you. In a sane industry this realization would lead to a change of the parad…

Yup, you’re outsourcing your decisions in exchange for not doing the work. You have to understand your risks and needs.

Your boring business systems that will live for 50 years need to have boring system level libraries or stuff you maintain. Your fast time to market or non-critical systems are optimized by speed and cost.

If your service depends on a bunch of downstream stuff, your processes need to support upgrade cycles. If you can’t keep up with that, you need to refactor.

Re: Log4j: The pain just keeps going

#80

Had to deal with some automated scan tool telling a client "you have log4j - you're vulnerable!" when... we didn't. A project was including a dependency on SLF4J and that project pulled in slf4j-log4j (IIRC). But it was not configured, wasn't compiled in to the final jar, and ... even it was, it was very old log4j (1.1 or 1.2 IIRC?). The vulnerability didn't affect that older version. We had to spend a week back and…

Yup, that's us. Same deal exactly. They already stoppered our entire team for eight months with a tools audit a few years ago, so I hung that dead fish above the conversation when this came up. "Uh, ok, don't want to do that, sooooo . . recommendations?" "Let's isolate it, toss stuff at it, and see if it lights up" "Sounds good to me. FIXED"
Post reply on HN