Log4j: The pain just keeps going • Hacker News Board

Re: Log4j: The pain just keeps going

#231

Post by kelnos » Wed, Jul 20, 2022, 9:14 PM UTC

preseinger wrote: post #192

0xbadcafebee wrote: post #121

Earlier quoted context omitted.

Imagine you are a random person in charge of a computer. You hear there's this big vuln affecting Go programs. 1. How do I find out which programs on my system are Go programs? I have a lot of programs scattered all over. Someone will have to come up with a method (probably search through $PATH for files, run `strings` on it, grep for "Go" , pray that's enough) to identify them. 2. They don't list a website, so I hav…

> You hear there's this big vuln affecting Go programs. This isn't relevant. What's relevant is if any of the specific programs I am responsible for have a vulnerability. If so, I need to update those specific programs. The notion of implementation language, shared library, dependency, these are all abstraction leaks. The atomic unit of responsibility, at the machine level, is program/service.

I think that's the parent's point. The hypothetical big announcement that this hypothetical "Log4Go" library has a vulnerability will not include a comprehensive list of all applications that use the library, so then we have to hope that the developers of every single application that uses Log4Go hears about the vulnerability, updates their software, makes a new release, and then notifies all their users somehow.

Alternatively, a more-savvy user has to hear about this Log4Go vulnerability and see if any of their applications use it and might need an update (which might involve contacting the maintainer to inform them that they need to update their software). But the vast majority of users aren't even going to hear about the Log4Go vulnerability, let alone understand what it means or what they need to do to protect themselves.

The safest thing would be for every application to dynamically link to a system-provided version of Log4Go, and OS maintainers would be the ones to send out an update via their OS's auto-update service. This does create other problems, but at least they are problems that technically-unsophisticated users do not need to solve or concern themselves with.

(Of course, there is no mechanism to do this with Go modules, so the entire thing is moot.)

This is why I get annoyed when developers say "I have to vendor library X into my application, because I'm afraid that the system copy will get updated to a version that breaks my app". Well then, maybe you shouldn't use a library whose author can't abide by semver and provide a reasonably-strong promise that minor and patch version updates won't break things. Otherwise you are just outsourcing your problems to your users.

Re: Log4j: The pain just keeps going

#232

Post by kelnos » Wed, Jul 20, 2022, 9:14 PM UTC

chrisseaton wrote: post #161

0xbadcafebee wrote: post #121

Earlier quoted context omitted.

Imagine you are a random person in charge of a computer. You hear there's this big vuln affecting Go programs. 1. How do I find out which programs on my system are Go programs? I have a lot of programs scattered all over. Someone will have to come up with a method (probably search through $PATH for files, run `strings` on it, grep for "Go" , pray that's enough) to identify them. 2. They don't list a website, so I hav…

> How do I find out which programs on my system are Go programs? Well how are these programs getting onto your system? Presumably via some package manager, or an image builder? Don't those manage the update for you? If you're running code that you don't know who wrote, with no source code, that you can't patch, that nobody is taking responsibility for, then yeah you've got issues. But you had issues already if that's…

> If you're running code that you don't know who wrote, with no source code, that you can't patch, that nobody is taking responsibility for, then yeah you've got issues.

This is the default state of affairs for nearly everyone who uses a computer.

We can either throw our hands up and futilely say "well they shouldn't do that!", or we can try to solve the problem.

Re: Log4j: The pain just keeps going

#233

Post by bostik » Wed, Jul 20, 2022, 9:16 PM UTC

horsawlarway wrote: post #86

paulmd wrote: post #56

Earlier quoted context omitted.

The problem isn't that audits are inherently worthless, it's that most of these tools are very low-quality implementations of the concept. In one of my past jobs I was an early-mover on doing a lot of our ops on Linux and the audit tools had no concept of the backport security model whatsoever. If you were running on some kind of LTS distro rather than a current distro, it would see "gosh you're 3 minor versions behi…

> The problem isn't that audits are inherently worthless, it's that most of these tools are very low-quality implementations of the concept. I think the problem is that the current audits are inherently worthless. I've never seen another industry that would accept a 95% false positive rate from a tool. But that's on the low end from my experiences (I've done 4 major codebase audits, and worked in the security industr…

> current audits are inherently worthless

And that's not helped by the warped incentives. Security has to be, fundamentally, assessed as a holistic thing and even highly skilled technical people can't really do that with modern systems. Now you add in auditors, a profession populated by glorified accountants, working off of gargantuan checklists.

You invite and reward mediocrity. At best.

Anyone good enough to actually understand and able to work with the technology will find a job in the industry, for twice the pay. And the same applies to regulators. They can't retain technically skilled staff, so they are filled with accountants. As a result we get rules and regulations, written by technically incompetent accountants, aimed at technically incompetent accountants.

To fill the gaps we have shitty, false-positive ridden, noisy, outright useless tools that generate reports intended to satisfy accountants. Security has next to nothing to do with that.

And to top it off, we have an entire industry riddled with so-called security engineers who know how to run a tool (with some marginally helpful StackOverflow pointers) and export a report, but can't for their life actually interpret, let alone VERIFY, any of it.

That has as much to do with security engineering as burning ants with a magnifying glass has to do with biology.

Re: Log4j: The pain just keeps going

#234

Post by Kalium » Wed, Jul 20, 2022, 9:18 PM UTC

secabeen wrote: post #221

Kalium wrote: post #209

Earlier quoted context omitted.

> The current software engineering paradigm has no meaningful answer to this, no matter what "security experts" tell you. In a sane industry this realization would lead to a change of the paradigm, but people in our industry seem to be only doubling down. On the contrary, I believe we do. The answer is ongoing maintenance . The basic problem is that people persist in looking at software as a fundamentally mechanistic…

So then what do you do with software that doesn't meet this standard? Software that works, that meets a need you have, but that is a fixed artifact, or at least will be in the foreseeable future. Do you refuse to use said software, and instead choose a less useful alternative that is actively developed?

That's a risk management problem. For how long will the world around this fixed artifact remain constant? How far away is your "foreseeable future" horizon? What happens when you're wrong? I have, in fact, chosen to use actively developed and maintained systems with fewer features than unmaintained EOL'd ones.

When you choose software to solve a problem, you're not just choosing software. You're encoding expectations around the problem, the environment, and your needs. If all three of those can be assured to never change then choosing a fixed artifact might be the way forward. Should any of those ever change, then your software will either adapt or become a less suitable solution.

How often can an organization really be that certain?

Re: Log4j: The pain just keeps going

#235

Post by beAbU » Wed, Jul 20, 2022, 9:18 PM UTC

horsawlarway wrote: post #78

tekchip wrote: post #33

Earlier quoted context omitted.

So somehow false positives negate all the actual positives caught and corrected? The only true solution is what? Manual audit of everything by some perfect human security practitioner? I suppose the same applies to automated development tools then. I will concede there probably are some firms out there acting poorly that way. When aren't there? Humans sigh . But by in large automation and the problems inherent are re…

I was in this industry for a while (I did 5 years full time for a security company). > I will concede there probably are some firms out there acting poorly that way. This is a hilarious take. Here's mine: The vast majority of these firms are here for liability. They do not provide security, they provide security theater so that if/when something goes wrong, the client can claim to have followed best practices, and th…

My project's client, a major local bank, mandates pen testing before we go live with the product that we're building.

Due to time and resourcing constraints, and a fixed launch date (don't ask), we had to significantly reduce scope. We are currently launching with probably 30% of the original scope. Once live we'll iterativley add features and grow our product to bring it back to the original vision.

Will we have to do pen testing again in the future? Nope. Even though a lot of our planned post launch features involve integrations with 3rd parties and the exchange of sensitive personal data.

Pen testing is 100% a box ticking exercise in my opinion. And most of the shops that offer this service are set up to provide the service such that large corporates to appease legal.

I really don't like pen testing as a practice and seriously dislike when it's a go live impediment.

Re: Log4j: The pain just keeps going

#236

Post by kelnos » Wed, Jul 20, 2022, 9:22 PM UTC

shagie wrote: post #77

hyperman1 wrote: post #27

I never understood where log4j 2 was supposed to fit in the ecosystem. * We had log4j 1, which was good enough for most use cases and everywhere. * We had slf4j/logback, which was from the same authors, and broke backward compatibility but gave us a fundamentally better design. * We had java.util.logging, which while braindead was builtin to Java and hence available everywhere. *We had commons logging, which was neve…

> Then apache decides to put new people on log4j, do a backward incompatible v2 design that nevertheless is worse than slf4j. Why? slf4j itself isn't a logging framework. It's a facade to logging frameworks. Simple Logging Facade for Java ( https://www.slf4j.org ) It needs a logging framework behind it - log4j, log4j2, logback, commons, JUL. The question is "why do log4j2?" Logback went from the log4j1.x path ( https…

> It needs a logging framework behind it - log4j

Minor nit: log4j doesn't implement the slf4j interface, though. If you do want log4j to do your logging, but use slf4j in your application, you need to include a "bridge" library that adapts slf4j to log4j. And, conversely, if you have dependencies that use log4j, but want things to end up going through your slf4j-supporting backend (say, logback), you need to exclude the "real" log4j jar and include a different library that implements log4j's interface and forwards calls to slf4j.

Re: Log4j: The pain just keeps going

#237

Post by kelnos » Wed, Jul 20, 2022, 9:27 PM UTC

marginalia_nu wrote: post #118

vbezhenar wrote: post #88

Earlier quoted context omitted.

Log4j2 does have some legitimate improvements over logback. These days you don't need anything fancy, just spew logs into stdout and something will capture it to your ELK, so those loggers seem obsolete indeed. Java need standard logging with slightly improved ergonomics and that's about it.

Yes, Java logging is ridiculously complicated for largely historical concerns that just don't make much sense in 2022. Like what you most likely need is a wrapper for System.out that maybe does some clever buffering and backpressure management, but the rest is sensibly managed outside of the application; whether by logrotate, ELK, or whatever. Long story short, no modern Java logging framework has any sensible reason…

I don't 100% agree with this, because even when you're doing containers and just dumping your log output to stdout, you still might be shipping that to a log aggregation system that will work a lot better with structured logging. So you might want to output JSON or some other structured format, and you might want your logger to support tagging, markers, stuff like that.

Even then, of course, this kind of logging should be able to be done in far fewer lines than most Java logging frameworks.

Re: Log4j: The pain just keeps going

#238

Post by tetha » Wed, Jul 20, 2022, 9:29 PM UTC

d4mi3n wrote: post #28

Nursie wrote: post #23

Earlier quoted context omitted.

Been in similar situations, it is not fun. "You have a vulnerable version of netty" "No, we don't, that's a false positive" "No see, the tool says you need netty 4.x and that's version 1.x, you need to update" "OK, but the tool is wrong, it's just picking up anything with 'netty' in the name, and that component is a wrapper around netty for some other thing, it only goes up to 1.8" "You have to update it to 4.x, this…

Security engineer here. This is sadly common. The grim fact seems to be that we have a dearth of information security analysts with engineering experience. If they don't have "Engineer" or similar in their title, odds are they haven't had the pleasure of building or maintaining a non-trivial unit of software over more than a quarter or so. That said, I've seen the inverse problem: engineering staff that either don't…

This is for example why we, as a company, rather took 2 hours to escalate Log4Shell internally. Which was like 2-3 hours before it went globally holy shit.

We took the time to analyze the vulnerability, and multiple competent people concluded: Yup. This is serious. Then we figured out plans, mitigations, and communication paths to update the plans and mitigations together with ops, dev and other folks. And then we shoved all of that as a large document into the larger technical organization as "Holy fuck the sky is falling, start patching, here's how to see when you have to, here's how you do. Go".

And that's also something I'm currently telling our new full-time security engineer. Be a bit mindful in the communication of criticality and vunlerability. Like, with my team of admins and SREs, you can be like "We're all doomed" and someone will be "but we don't deploy that feature, calm down. Would you like some tea? What evil thing did the CVE say to you?". Or they might escalate accordingly, if it looks bad to them, too.

But if you do that to most of our dev-teams? That'll be just an unstructured mess, which will be worse than waiting a moment to structure a proper response to the threat.

Re: Log4j: The pain just keeps going

#239

Post by joebob42 » Wed, Jul 20, 2022, 9:30 PM UTC

recursive wrote: post #91

throwaway23234 wrote: post #70

Earlier quoted context omitted.

What counts as a garbage language? Java guys will tell you it's rust or go, go guys will tell you it's java or rust, rust guys will tell you it's java and go.

> What counts as a garbage language? I assume it's one with GC.

Wouldn't it be one without gc? Gc collects the garbage and disposes of it, whereas in other languages it just piles up :D

Re: Log4j: The pain just keeps going

#240

Post by staticassertion » Wed, Jul 20, 2022, 9:30 PM UTC

Kalium wrote: post #234

secabeen wrote: post #221

Earlier quoted context omitted.

So then what do you do with software that doesn't meet this standard? Software that works, that meets a need you have, but that is a fixed artifact, or at least will be in the foreseeable future. Do you refuse to use said software, and instead choose a less useful alternative that is actively developed?

That's a risk management problem. For how long will the world around this fixed artifact remain constant? How far away is your "foreseeable future" horizon? What happens when you're wrong? I have, in fact, chosen to use actively developed and maintained systems with fewer features than unmaintained EOL'd ones. When you choose software to solve a problem, you're not just choosing software. You're encoding expectations…

That's exactly right - it's about risk. No one has "perfect" operations/ security. They have a threat model and a risk tolerance.