Live data from Hacker News

Log4j: The pain just keeps going

thenewstack.io

271–280 of 291 posts

Re: Log4j: The pain just keeps going

#271
post #178

Earlier quoted context omitted.

I have a very simple solution, in three parts: 1) We use very few dependencies. We err on the side of writing our own function, library, or UI component rather than installing a new dependency, even if a "better" open-source version might theoretically be available. 2) We use a monorepo. We have two Go backends sharing the same go.mod, and three Typescript/React frontends sharing the same package.json, all in the sam…

> 1) We use very few dependencies. We err on the side of writing our own function, library, or UI component rather than installing a new dependency, even if a "better" open-source version might theoretically be available. This just means you likely have a lower quality version with the same problems but without the benefit of an ecosystem to find them.

It doesn't mean that at all. When we need one feature we write it, rather than exposing the 1000 features of an open-source library to the internet when we only need one feature.

We won't write it ourselves if we aren't sure we can do it well, i.e. we're not going to roll our own encryption. Those are the types of dependencies we do use, and upgrade once a month like clockwork.

The point is not "don't use someone else's encryption library," the point is "don't use someone else's LeftPad() function that takes 3 lines to write". That gives us fewer dependencies, which makes it easier to upgrade all dependencies regularly, which is good for security.

Re: Log4j: The pain just keeps going

#272

If you automatically update your dependencies all the time, you will constantly get new bugs, issues and sometimes even malware. If you don't update your dependencies all the time, you will be vulnerable to old bugs and issues. The current software engineering paradigm has no meaningful answer to this , no matter what "security experts" tell you. In a sane industry this realization would lead to a change of the parad…

Isn't that just software though? If you fix bugs in your code, you'll introduce new bugs. If you don't fix bugs, you'll have bugs.

Re: Log4j: The pain just keeps going

#273
post #242

Earlier quoted context omitted.

The problem I have with adopting something like this is that (other than the added time it would take to write and maintain all my own dependencies), I don't trust myself to avoid writing the same kinds of security bugs that a third-party dependency might also write. But those bugs are much more likely to be discovered and fixed in the widely-used third-party library than in my bespoke library, even if I publish that…

I don't claim to "never write security bugs." However, it appears that the norm in the industry, is to write lash-up code, as quickly as possible, in the hopes on an MVP that will get brought up by a FAANG. And I am doubtful that most OSS actually gets all that "other eyeballs vetting" that everyone talks about. Most folks that I know, that use a lot of these dependencies, would be unable to understand what goes on i…

> I don't claim to "never write security bugs."

Sorry if I gave that impression; I wasn't trying to put those words in your mouth.

> And I am doubtful that most OSS actually gets all that "other eyeballs vetting" that everyone talks about.

Agreed, but a reasonably popular OSS project has a much higher chance of getting that vetting (or any vetting at all, really) than a bespoke library that the public never sees. The chances are a little higher for a nice / little-used bespoke library that you end up putting on GitHub or wherever, but it's still pretty low.

> In fact, I remember a founder, on this very forum, declare that "If you don't get physically sick, looking at your MVP code, then you are spending too much time on code quality."

I would never want to work for such a founder. I get the tension between getting a MVP out before you run out of money, but experience shows that prototype code ends up living in production a lot longer than anyone plans for. And I've personally seen those prototypes cause 6 and 7 figures of damage to a company later, when they haven't been properly replaced or at least improved when they should be.

I'm glad what you're doing works for you (you sound like you probably actually are an above-average developer, unlike -- by definition -- most people), but I don't think I could in good conscience recommend your approach as a general practice.

Re: Log4j: The pain just keeps going

#274

Earlier quoted context omitted.

I was sure reading your first post that you will propose rust. Rust users never cease to amaze me.

To be fair there was also a significant chance he would think Rust was garbage and only assembly (or if you are lucky, C) are acceptable languages.

You still don't know how to spot them :). The borrow checker is a great invention, but for them it's the holy grail that will save us from bugs. Like in the past people thought that GC was the solution to every problems. Youngsters must learn, we have to be patient.

Re: Log4j: The pain just keeps going

#275
post #273

Earlier quoted context omitted.

I don't claim to "never write security bugs." However, it appears that the norm in the industry, is to write lash-up code, as quickly as possible, in the hopes on an MVP that will get brought up by a FAANG. And I am doubtful that most OSS actually gets all that "other eyeballs vetting" that everyone talks about. Most folks that I know, that use a lot of these dependencies, would be unable to understand what goes on i…

> I don't claim to "never write security bugs." Sorry if I gave that impression; I wasn't trying to put those words in your mouth. > And I am doubtful that most OSS actually gets all that "other eyeballs vetting" that everyone talks about. Agreed, but a reasonably popular OSS project has a much higher chance of getting that vetting (or any vetting at all, really) than a bespoke library that the public never sees. The…

> I don't think I could in good conscience recommend your approach as a general practice.

I can live with that, but ... (There's always a "but").

I am not happy at all, with the general industry practice of writing every project to be something that can be understood by inexperienced, undisciplined coders.

Every language and programming methodology has an "advanced" type of thing, requiring people to have experience and/or book-larnin'.

I write Swift at a fairly advanced level. I am not at the level of some heavy-duty advanced Swift people, but I am pretty "idiomatic," in my approach. It is not "rewritten TypeScript," like so much code out there.

My code is very well-documented, and I hold myself to standards of Quality that most folks in the industry consider to be obsessive to the point of insanity. My testing code usually dwarfs my implementation code, and my documentation is, let's say ... complete. You can see what I mean in my latest module[0].

The fact that so many folks have so much scorn for my posture and approach, tells me pretty much all I need to know. My values are not in sync with the prevailing industry values. I guess that I'm an "anomaly." I can live with that, as well.

I won't write junk, so that someone used to junk, can comprehend it. If people aren't willing to learn enough to understand my middle-of-the-road semi-advanced Swift, then I can't help them. Swift is an awesome language. I feel that we are doing ourselves a disservice, if we do not explore it.

I write for myself. I write code and documentation that I want to use (and I use it). I really don't care, whether or not someone else "approves" of it. I am not relying on others to review, maintain, or patch my code.

When I do use other people's code, I vet it fairly carefully. Including a dependency is a really serious matter. I'm handing full control of my execution context to code that someone else wrote. I'd damn well better take that Responsibility seriously.

[0] https://github.com/RiftValleySoftware/RVS_UIKit_Toolbox

Re: Log4j: The pain just keeps going

#276

Earlier quoted context omitted.

We see this all the time in healthcare. MRI imaging machine that cost 500k 20 years ago still works fine but only on Windows XP. Cost to rewrite the software for windows 10 is more than 500k so rather than paying for that instead they firewall the shit out of it so it can only talk to some printing / output device. That’s just one example. Medical machines are expensive as hell and companies don’t want to rebuy or co…

Not familiar with such kind of software, but if it were written for, say, Linux, would it need such an expensive rewrite to work with current versions?

that gameboy cartridge isn't working on a modern system either...

Re: Log4j: The pain just keeps going

#277
post #106

Earlier quoted context omitted.

I run a security team in a large company. What you described is usually the result of some "consulting company" (in our case big ones) that drop stuff with zero actual knowledge. Every year we "review" these with the board and it is annoying as fuck. Recently they got a new manager who understands we are in the same boat. They have to provide a report with some findings and I need to have a secure environment. We tal…

Hi! Question from a fellow practitioner: What tools or processes do you and your team use to identify false positives in such scans and reports? How do you track out-of-band mitigations against real-but-unreachable vulnerabilities that have been identified? e.g. A WAF set to intercept requests that trigger a specific vuln, or network segmentation to limit the scope of a moderate vulnerability. In my experience tuning…

Well, this is a real circus.

I have tested and deployed several scanning tool (and have contributed to nessus some years ago - their scanning capacities were really mediocre compared to the precision of the results. In out 100+k IPs on 10.x networks the time to scan was horrendous. But once an IP was scanned, the results were fine). A small disclaimer: I use several solutions but have no stakes in any of them (neither company nor personal). I have build relationships with them that helps me to provide substantiated feedback that sometimes event get taken into account :)

There are several issues, none of them has an automated way of fixing.

First there is the criticality of the finding. We found that often (especially on Linux) you will get a "raw" critical issue on a component that was not patched; And why it was not patched? Because the vendor set a low level. It is then that the discussion between the security team and the OS one starts - I have experimentally found that the vendor rating is usually better because they take into account the actual context of the deployment, especially for libraries.

OTOH crowdstrike vulnerability rating (I forgot the name of the component, it is on top of the EDR) is actually two ratings: the CVSS-reated one, and their own where they look at actual exploits and deployment. I have decided that critical+critical = actually critical that must be patched no matter what. This is the only way I found to quantitatively auto-decide on important patches. The goal is of course to patch everything but the reality is that there are so many loose ends in products that fighting for anything below the top two levels does not make sense.

The very sad thing here is that this is perceived as an issue of the OS/middleware and not the shitty products vendors deploy. They use some undocumented or weak solutions and then cry when everything breaks down. I develop myself (amateur developer) and have never had any issues for 30 years when upgrading the OS because I actually care about following the docs.

The second problem is that today products have nested dependencies. For our own products we do an awfully time taking and precise review of the actual impact on the product but this is because people actually care (this is not my team so I ma not taking any credit - I just like their approach). When a customer comes with a question we have a comprehensive response. This works or not, some people when they see "vulnerability" they stop at that and want that word to disappear.

When we are on the receiving side of such products (and log4j is a perfect example), we ask the vendor. If this is open source the answer is usually very good, there are some github/gitlab issues that address the point. Commercial vendors are much worse because they either say "no worries" or take weeks to respond. When a "no worry" comment comes from a company that cannot explain why they limit their passwords to 27 characters, it is worrisome - a bit like when my children say "no worries" (this is the moment when I start to worry).

Scanners are quite slow to respond to non obvious vulnerabilities (the ones you cannot easily tag though OVAL). For log4j for instance I coded a scanner/reflector on the night of Thursday (the information popped up about noon on Thursday western europe time). The scanners followed up after a week, but one could fine good ones open source quickly as well. So my advice would be to closely monitor GitHub for such tools when there is pressure (we switched to another tool for log4j when my team and the OS ones were successively finding better ones). This also led us to investigate a fantastic tool/framework to write scanners, which name escapes me right now but I can comment here when I am back from vacation and get hold of the gal who is looking at that.

The TL;DR version is that

- a few tools provide a way to quantify the risk (that's a huge slit but the best I can do) - at least there is a reasonable chance that you will not miss the worst ones

- GitHub activity when shit hits the fan is extraordinarily useful - the best tools emerge there, before the vendors catch up

- vendors are medium, but is is not as bad as you read

- your CISO should make the effort to communicate that there will be availability issues when the IT/security team decides to switch off a key component (say - email) if this presents a risk for the company. I have the chance to have a CEO who not only understands this but actually challenged me once to why I did not cut earlier. Having that arranged upfront talks a huge stress off the security team.

- and finally the sad reality is that the vast, vast majority of issues we get are from users who click on stuff

I am sure I forgot a lot - let me know if I can help with some details.

Re: Log4j: The pain just keeps going

#278
post #269
post #28

Earlier quoted context omitted.

Security engineer here. This is sadly common. The grim fact seems to be that we have a dearth of information security analysts with engineering experience. If they don't have "Engineer" or similar in their title, odds are they haven't had the pleasure of building or maintaining a non-trivial unit of software over more than a quarter or so. That said, I've seen the inverse problem: engineering staff that either don't…

> Trust can be built between engineering and infosec teams Yes it can, and I have worked with 'good' infosec people who were all about improving practices together. That attitude goes a long way - developers (well, the good ones) want to ship secure stuff and want to fold in best practices. But the profession is unfortunately rife with people who don't know as much as they think they do, and who wade in heavy-handed,…

Some organizations do do this and they call them product security engineers or similar. Sadly, this doesn’t remove the need for an informed organization at large—that function covers a lot more than securing code—but it’s absolutely something that has been done.

There is one caveat however: you’ll probably still need dedicated security analysis or security engineers outside the chain of command of your product or engineering org.

There is a degree of necessary opposing pressures and incentives for infosec departments vs product departments: product organizations are rewarded for shipping more. Infosec organizations are rewarded for preventing incidents.

The most common result of this contention are situations people often lament in the comments here—security programs that are either ineffective, heavy handed, or both.

In my mind, a good security program positions itself as an enablement function and should strive to educate and inform over all else. People close to the work will be best positioned to make decisions about effective ways to harden their systems when informed of likely risks, requirements, or other security considerations.

With any luck this minimizes the need to mandate security motivated changes and changes they dynamic when your local security professional comes by with specific concerns. It’ll be less of “the sky is falling” and more of “we saw this specific problem and would like to work with you to fix it”.

Re: Log4j: The pain just keeps going

#279
post #106

Earlier quoted context omitted.

Hi! Question from a fellow practitioner: What tools or processes do you and your team use to identify false positives in such scans and reports? How do you track out-of-band mitigations against real-but-unreachable vulnerabilities that have been identified? e.g. A WAF set to intercept requests that trigger a specific vuln, or network segmentation to limit the scope of a moderate vulnerability. In my experience tuning…

Well, this is a real circus. I have tested and deployed several scanning tool (and have contributed to nessus some years ago - their scanning capacities were really mediocre compared to the precision of the results. In out 100+k IPs on 10.x networks the time to scan was horrendous. But once an IP was scanned, the results were fine). A small disclaimer: I use several solutions but have no stakes in any of them (neithe…

Thank you for the fantastic response, I always enjoy seeing how folks approach these problems. Your points about inconsistency between how vendors vs OS distributions handle things is well taken and mirrors some things I've seen.

Please do feel free to reach out via email or LinkedIn (deets in my profile), always a pleasure talking shop with folks in the industry.

Cheers!

Re: Log4j: The pain just keeps going

#280

Earlier quoted context omitted.

So then what do you do with software that doesn't meet this standard? Software that works, that meets a need you have, but that is a fixed artifact, or at least will be in the foreseeable future. Do you refuse to use said software, and instead choose a less useful alternative that is actively developed?

We see this all the time in healthcare. MRI imaging machine that cost 500k 20 years ago still works fine but only on Windows XP. Cost to rewrite the software for windows 10 is more than 500k so rather than paying for that instead they firewall the shit out of it so it can only talk to some printing / output device. That’s just one example. Medical machines are expensive as hell and companies don’t want to rebuy or co…

Not to split hairs, but I think what you've described is constant investment in something that works and does one thing well. The firewalls and management and maintenance are all ongoing costs to manage the risk.

Software is never just an artifact. It's a whole system, of which the source code and binaries are some components. If you do not manage those artfacts, the next best option is generally managing the environment.

Post reply on HN