This shouldn't have happened: A vulnerability postmortem • Hacker News Board

Re: This shouldn't have happened: A vulnerability postmortem

#351

Post by randomswede » Thu, Dec 02, 2021, 8:10 AM UTC

terinjokes wrote: post #320

elabajaba wrote: post #265

Earlier quoted context omitted.

Discord was consistently seeing pauses in the range of several hundred ms every 2 minutes a couple years ago. https://blog.discord.com/why-discord-is-switching-from-go-to...

Hard to say without more details, but those graphs look very similar to nproc numbers of goroutines interacting with the Linux-of-the-time's CFS CPU scheduler. I've seen significant to entire improvement to latency graphs simply by setting GOMAXPROC to account for the CFS behavior. Unfortunately the blog post doesn't even make a passing mention to this.

Anecdotally, the main slowdown we saw of Go code running in Kubernetes at my previous job was not "GC stalls", but "CFS throttling". By default[1], the runtime will set GOMACSPROCS to the number of cores on the machine, not the CPU allocation for the cgroup that the container runs in. When you hand out 1 core, on a 96-core machine, bad things happen. Well, you end up with a non-smooth progress. Setting GOMACPROCS to ceil(cpu allocation) alleviated a LOT of problems

Similar problems with certain versions of Java and C#[1]. Java was exacerbated by a tendency for Java to make everything wake up in certain situations, so you could get to a point where the runtime was dominated by CFS throttling, with occasional work being done.

I did some experiments with a roughly 100 Hz increment of a prometheus counter metric, and with a GOMAXPROCS of 1, the rate was steady at ~100 Hz down to a CPU allocation of about 520 millicores, then dropping off (~80 Hz down to about 410 millicores, ~60 hz down to about 305 millicores, then I stopped doing test runs).

[1] This MAY have changed, this was a while and multiple versions of the compiler/runtime ago. I know that C# had a runtime release sometime in 2020 that should've improved things and I think Java now also does the right thing when in a cgroup.

Re: This shouldn't have happened: A vulnerability postmortem

#352

Post by throwaway984393 » Thu, Dec 02, 2021, 8:22 AM UTC

saghm wrote: post #266

throwaway984393 wrote: post #176

Earlier quoted context omitted.

A century ago, buildings were quite dangerous, and likely to kill you in all sorts of situations. Wood burns, and concrete and brick don't. Clearly wood is an "unsafe material". But just changing the material didn't result in safer buildings. Buildings made of brick and concrete still killed people. It turns out that there are a lot of factors that go into building safety. The material is one vulnerability, sure. But…

> A century ago, buildings were quite dangerous, and likely to kill you in all sorts of situations. Wood burns, and concrete and brick don't. Clearly wood is an "unsafe material". But just changing the material didn't result in safer buildings. Buildings made of brick and concrete still killed people. This will seem trite, but I think it's just literally easier to figure out how to build wooden buildings that are fir…

Well, it did take us a few thousand years to get to safe wooden buildings...

I actually don't think securing C/C++ code is that hard. It's certainly a skill you need to learn, but so is writing linked lists and qsort. I think people just aren't applying themselves. But the language seems to catch the flack rather than the programmer.

From the article:

  The bug is that there is simply no bounds checking at all; sig and key are
  arbitrary-length, attacker-controlled blobs, and cx->u is a fixed-size buffer.

As we can see, the programmer just made no effort to secure the code. But we still blame the language, like blaming wood for being flammable.

Anyway. I'm definitely not against new languages. But I think before a program is rewritten, it should be for a reason much better than "I didn't want to secure the code".

Re: This shouldn't have happened: A vulnerability postmortem

#353

Post by lelanthran » Thu, Dec 02, 2021, 8:24 AM UTC

lucb1e wrote: post #69

rfoo wrote: post #65

Earlier quoted context omitted.

What's special here is the bug is a memory corruption, and memory corruption bugs in such libraries are usually instantly security bugs. Otherwise, the same story could be told as a generic software testing joke: "unit-tests are short-sighted and coverage lies", i.e. an "extremely well-maintained codebase, with extensive unittest, >98% test coverage and constantly scanned by all-static-analyzers-you-may-come-up" can…

> What's special here is the bug is a memory corruption, and memory corruption bugs in such libraries are usually instantly security bugs. Is that special? Are there buffer overflow bugs that are not security bugs? It could be just my bubble as a security consultant, since (to me) "buffer overflow" assumes remote code execution is a given. It's not my area of expertise, though, so perhaps indeed not all reachable buf…

> Are there buffer overflow bugs that are not security bugs? It could be just my bubble as a security consultant, since (to me) "buffer overflow" assumes remote code execution is a given.

Not necessarily:

1. Most compilers use aligned variables and fields unless forced not to with a flag. Going 3 bytes over an array of 12 bytes can result in an overflow that is never detectable at runtime because the extra memory being used is used exclusively by the bug.

2. Malloced memory is harder (but not impossible) to turn into an unintended execution because the pages may be marked for data only by the OS. The stack pages are ~marked~ [EDIT: ~NOT marked~] as executable.

There's probably millions of buffer overflows that will not only never be exploited, but will also never be detected.

Re: This shouldn't have happened: A vulnerability postmortem

#354

Post by gpderetta » Thu, Dec 02, 2021, 8:26 AM UTC

titzer wrote: post #281

fleventynine wrote: post #172

Earlier quoted context omitted.

If it can really guarantee single-digit microsecond pauses in my realtime thread no matter what happens in other threads of my application, that is indeed a game changer. But I'll believe it when I see it with my own eyes. I've never even used a garbage collector that can guarantee single-digit millisecond pauses.

Have you measured the pause times of free()? Because they are not deterministic, and I have met few people who understand in detail how complex it can be in practice. In the limit, free() can be as bad as GC pause times because of chained deallocation--i.e. not statically bounded.

People don't call free from their realtime threads.

Re: This shouldn't have happened: A vulnerability postmortem

#355

Post by renewiltord » Thu, Dec 02, 2021, 8:40 AM UTC

nineteen999 wrote: post #300

titzer wrote: post #275

Earlier quoted context omitted.

The whole post is a giant blinking red sign that says (or should say) "Fuzzing is a horribly ineffective workaround for a treacherous language." No offense to the many bright and capable people who have worked hard on the C/C++ language, tools, compilers, libraries, kernels, etc over the years, but we will someday look back on it as asbestos and wonder why we kept at it for so damn long .

No issue with the first sentence of your message at all, but... > No offense to the many bright and capable people who have worked hard on the C/C++ language, tools, compilers, libraries, kernels, etc over the years, but we will someday look back on it as asbestos and wonder why we kept at it for so damn long. We won't wonder at all. We will understand that those people are the ONLY ones that stepped up to the task o…

Lmao. Absolute perfection. Software that exists has way more value.

Re: This shouldn't have happened: A vulnerability postmortem

#356

Post by rfoo » Thu, Dec 02, 2021, 8:41 AM UTC

throwaway894345 wrote: post #91

rfoo wrote: post #66

Earlier quoted context omitted.

> far, far easier to build than all of these C programs One of my friends who work on AIX machines without direct Internet access does not share the same view, though.

Why is indirect Internet access less of a problem for C than Rust/Go/etc? Seems like for modern systems, you just run a pre-populated caching proxy on your target and `cargo install` like you normally would. In C, you're manually checking versions and putting files in the right spot on disk for every stage of the build (this can be alleviated a bit if you can find pre-built binaries and so on, but even in the best ca…

> Why is indirect Internet access less of a problem for C than Rust/Go/etc?

Because C codes tend to have less dependencies and shallow/more "clustered" dependency graph.

To be fair, that's more or less due to dependency management being a 100% pain 0 fun experience.

Re: This shouldn't have happened: A vulnerability postmortem

#357

Post by blub » Thu, Dec 02, 2021, 8:47 AM UTC

pjmlp wrote: post #342

nineteen999 wrote: post #300

Earlier quoted context omitted.

No issue with the first sentence of your message at all, but... > No offense to the many bright and capable people who have worked hard on the C/C++ language, tools, compilers, libraries, kernels, etc over the years, but we will someday look back on it as asbestos and wonder why we kept at it for so damn long. We won't wonder at all. We will understand that those people are the ONLY ones that stepped up to the task o…

> We will wonder why with all the millions of lines of C/C++ reference code available to be perused and then rewritten in Rust, Pascal, C#, Zig or Nim, and the vociferousness of their advocates, why that didn't happen in a reasonable timeframe. Easy, every single time .NET team does some advances into that direction, it gets sabotaged by WinDev and their C++ love. XNA vs DirectXTK, .NET vs COM/WinRT,... Windows could…

Before I ditched Windows a couple of years ago, I was able to experience first hand how bloated and slow was the software that Microsoft rewrote in C#, so I kind of understand why such rewrites were being sabotaged.

If by some minor miracle a C# or Java GUI app is not slow, then it will use a ton of memory. A whole OS of such apps would be a nightmare.

Re: This shouldn't have happened: A vulnerability postmortem

#358

Post by throwaway5371 » Thu, Dec 02, 2021, 8:59 AM UTC

i love the fact that people can read and write whatever they want wherever they want

go and rust make things a bit boring, c makes me feel like a kid

of course the price for freedom is security haha

Re: This shouldn't have happened: A vulnerability postmortem

#359

Post by GoblinSlayer » Thu, Dec 02, 2021, 9:05 AM UTC

titzer wrote: post #275

lucb1e wrote: post #8

A title that actually describes the post, mostly paraphrasing the first paragraph: Reasons why this buffer overflow wasn't caught earlier despite doing all the right things And then to give those reasons: - "each component is fuzzed independently" ... "This fuzzer might have produced a SECKEYPublicKey that could have reached the vulnerable code, but as the result was never used to verify a signature, the bug could ne…

The whole post is a giant blinking red sign that says (or should say) "Fuzzing is a horribly ineffective workaround for a treacherous language." No offense to the many bright and capable people who have worked hard on the C/C++ language, tools, compilers, libraries, kernels, etc over the years, but we will someday look back on it as asbestos and wonder why we kept at it for so damn long .

Ironically STL supports bound checking, but is always turned off.

Re: This shouldn't have happened: A vulnerability postmortem

#360

Post by lelanthran » Thu, Dec 02, 2021, 9:06 AM UTC

zionic wrote: post #25

jandrese wrote: post #17

Earlier quoted context omitted.

It's hard to fault a project written in 2003 for not using Go, Rust, Haskell, etc... It is also hard to convince people to do a ground up rewrite of code that is seemingly working fine.

>seemingly worked fine That’s just it though, it never was. That C/C++ code base is like a giant all-brick building on a fault line. It’s going to collapse eventually, and your users/the people inside will pay the price.

>>seemingly worked fine

>That’s just it though, it never was. That C/C++ code base is like a giant all-brick building on a fault line. It’s going to collapse eventually, and your users/the people inside will pay the price.

Sure, but everything is a trade-off[1]. In this particular case (and many others) no user appeared to pay any price, which tells me that the price is a spectrum ranging from 'Nothing' to 'FullyPwned' with graduations in between.

Presumably the project will decide on what trade-off they are willing to make.

[1] If I understand your comment correctly, you are saying that any C/C++ project has a 100% chance of a 'FullyPwned' outcome.