This shouldn't have happened: A vulnerability postmortem • Hacker News Board

Re: This shouldn't have happened: A vulnerability postmortem

#281

Post by titzer » Thu, Dec 02, 2021, 4:06 AM UTC

fleventynine wrote: post #172

throwaway894345 wrote: post #158

Earlier quoted context omitted.

If your "deterministic amount of time" can tolerate single-digit microsecond pauses, then Go's GC is just fine. If you're building hard real time systems then you probably want to steer clear of GCs. Also, "developer velocity" is an important criteria for a lot of shops, and in my opinion that rules out Rust, C, C++, and every dynamically typed language I've ever used (of course, this is all relative, but in my exper…

If it can really guarantee single-digit microsecond pauses in my realtime thread no matter what happens in other threads of my application, that is indeed a game changer. But I'll believe it when I see it with my own eyes. I've never even used a garbage collector that can guarantee single-digit millisecond pauses.

Have you measured the pause times of free()? Because they are not deterministic, and I have met few people who understand in detail how complex it can be in practice. In the limit, free() can be as bad as GC pause times because of chained deallocation--i.e. not statically bounded.

Re: This shouldn't have happened: A vulnerability postmortem

#282

Post by speed_spread » Thu, Dec 02, 2021, 4:08 AM UTC

titzer wrote: post #275

lucb1e wrote: post #8

A title that actually describes the post, mostly paraphrasing the first paragraph: Reasons why this buffer overflow wasn't caught earlier despite doing all the right things And then to give those reasons: - "each component is fuzzed independently" ... "This fuzzer might have produced a SECKEYPublicKey that could have reached the vulnerable code, but as the result was never used to verify a signature, the bug could ne…

The whole post is a giant blinking red sign that says (or should say) "Fuzzing is a horribly ineffective workaround for a treacherous language." No offense to the many bright and capable people who have worked hard on the C/C++ language, tools, compilers, libraries, kernels, etc over the years, but we will someday look back on it as asbestos and wonder why we kept at it for so damn long .

At least Asbestos is inert once in place.

Re: This shouldn't have happened: A vulnerability postmortem

#283

Post by Godel_unicode » Thu, Dec 02, 2021, 4:17 AM UTC

tptacek wrote: post #153

Veserv wrote: post #102

Earlier quoted context omitted.

How do you know that it was actually "extremely well-maintained"? Everybody thought OpenSSL was well-maintained since it was used as critical infrastructure by multi-billion dollar megacorporations, but it was actually maintained by two full-time employees and a few part-time volunteers with maybe a quick once-over before a commit if they were lucky. How about sudo, a blindly trusted extremely sensitive program [1],…

I don't know anybody who thought OpenSSL was well-maintained in and before the Heartbleed era (it's a fork of SSLeay, which was Eric Young's personal project). Post-Heartbleed --- a decade ago, longer than the time lapse between SSLay and OpenSSL --- maintenance of OpenSSL has improved dramatically.

I think you and a sibling comment might be too close to the problem. When heartbleed dropped my Twitter feed had a few crypto engineers saying "I mean, eventually this was going to happen" and a ton of developers whose main language starts with a "p" going "how?? OpenSSL is core plumbing of the internet, it can't be this bad can it???".

Edit: to be clear, not maligning the "p" language developers, I was one myself. Simply saying there are a ton of technical people who are blissfully ignorant of the tiny pillars of clay that hold up the internet.

Re: This shouldn't have happened: A vulnerability postmortem

#284

Post by loeg » Thu, Dec 02, 2021, 4:21 AM UTC

rurban wrote: post #147

Now take a deep look at the POSIX C standard, Annex K. The bounds checked extensions. Using these would have definitely avoided the problem. memcpy_s requires the size of dest to be defined. The applause goes to the glibc maintainers, who still think they are above that.

Annex K is pretty awful[1]. There are plenty of fine solutions here in hindsight, but glibc adopting Annex K isn’t one of them. NSS has a plethora of build targets, including Windows - which does not implement Annex K either (despite inspiring it).

[1]: http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1967.htm

Re: This shouldn't have happened: A vulnerability postmortem

#285

Post by FpUser » Thu, Dec 02, 2021, 4:43 AM UTC

josephg wrote: post #255

FpUser wrote: post #238

Earlier quoted context omitted.

>"1kloc / day is a reasonable ballpark figure, landing us at about one person-year to port boringssl to rust." Porting 1k lines a day, testing and catching and fixing errors to language with incompatible memory model and doing it for a year is insanity. Programmers who propose this kind of productivity most likely have no idea about real world.

There’s an old saying attributed to Abe Lincoln: “Give me six hours to chop down a tree and I will spend the first four sharpening the axe.” I contend that most software engineers (and most engineering houses) don’t spend any time sharpening the axe. Certainly not when it comes to optimizing for development speed. Nobody even talks about deliberate practice in software. Isn’t that weird? 1000 lines of code is way too…

I've ported / developed from scratch way more than 1000 a day on emergency basis. Doing it reliably and with proper testing every day for a year - thanks, but no thanks.

>"I have no doubt this sort of speed is unheard of in your office, with your processes."

I am my own boss for the last 20+ years and I do not have predefined processes. And I am not in a dick size contest.

Re: This shouldn't have happened: A vulnerability postmortem

#286

Post by adrianN » Thu, Dec 02, 2021, 4:46 AM UTC

titzer wrote: post #275

lucb1e wrote: post #8

A title that actually describes the post, mostly paraphrasing the first paragraph: Reasons why this buffer overflow wasn't caught earlier despite doing all the right things And then to give those reasons: - "each component is fuzzed independently" ... "This fuzzer might have produced a SECKEYPublicKey that could have reached the vulnerable code, but as the result was never used to verify a signature, the bug could ne…

The whole post is a giant blinking red sign that says (or should say) "Fuzzing is a horribly ineffective workaround for a treacherous language." No offense to the many bright and capable people who have worked hard on the C/C++ language, tools, compilers, libraries, kernels, etc over the years, but we will someday look back on it as asbestos and wonder why we kept at it for so damn long .

Unfortunately it's too much work to throw away all code written in unsafe languages. So it's valuable to try and improve tools and techniques that make these languages less unsafe.

Re: This shouldn't have happened: A vulnerability postmortem

#287

Post by steveklabnik » Thu, Dec 02, 2021, 5:01 AM UTC

lmm wrote: post #273

shp0ngle wrote: post #267

Earlier quoted context omitted.

I don't think you need to explain to Mozilla about Just Rewriting It to Rust

Didn't they recently fire the Rust team?

While they did let go the folks who were working on Rust (see the other comments in this thread) they are still members of the Rust foundation, and folks still write code in Rust at Mozilla. For example you have this from last week https://groups.google.com/a/mozilla.org/g/dev-platform/c/dVU...

Re: This shouldn't have happened: A vulnerability postmortem

#288

Post by saurik » Thu, Dec 02, 2021, 5:06 AM UTC

titzer wrote: post #275

lucb1e wrote: post #8

A title that actually describes the post, mostly paraphrasing the first paragraph: Reasons why this buffer overflow wasn't caught earlier despite doing all the right things And then to give those reasons: - "each component is fuzzed independently" ... "This fuzzer might have produced a SECKEYPublicKey that could have reached the vulnerable code, but as the result was never used to verify a signature, the bug could ne…

The whole post is a giant blinking red sign that says (or should say) "Fuzzing is a horribly ineffective workaround for a treacherous language." No offense to the many bright and capable people who have worked hard on the C/C++ language, tools, compilers, libraries, kernels, etc over the years, but we will someday look back on it as asbestos and wonder why we kept at it for so damn long .

More strongly: the idea that "fuzzing" is "doing all the right things" is insane and disappointing of a narrative. The code snippets I am seeing here are ridiculous with manual, flag-based error handling? There is a use of something like memcpy that isn't hidden behind some range abstraction that manages all of the bounds?! This code probably is riddled with bugs, because they are doing all the wrong things, and fuzzing the result counts for nothing. C++ sucks, but it is way better than this.

Re: This shouldn't have happened: A vulnerability postmortem

#289

Post by duped » Thu, Dec 02, 2021, 5:07 AM UTC

smolder wrote: post #198

duped wrote: post #137

Earlier quoted context omitted.

> Or are C/C++ benefits just too high to ever give up? FFI is inherently memory-unsafe. You get to rewrite security critical things from scratch, or accept some potentially memory-unsafe surface area for your security critical things for the benefit that the implementation behind it is sound. This is true even for memory-safe languages like Rust. The way around this is through process isolation and serializing/deseri…

> FFI is inherently memory-unsafe Maybe this specific problem needs attention. I wonder, is there a way we can make FFI safer while minimizing overhead? It'd be nice if an OS or userspace program could somehow verify or guarantee the soundness of function calls without doing it every time. If we moved to a model where everything was compiled AOT or JIT locally, couldn't that local system determine soundness from the…

This is a really hard problem because you have to discard the magic wand of a compiler and look at what is really happening under the hood.

At its most rudimentary level, a "memory safe" program is one that does not access memory that is forbidden to it at any point during execution. Memory safety can be achieved using managed languages or subsets[1] of languages like Rust - but that only works if the language implementations have total knowledge of memory accesses that the program may perform.

The trouble with FFIs is that by definition, the language implementations cannot know anything about memory accesses on the other side of the interface - it is foreign, a black box. The interface/ABI does not provide details about who is responsible for managing this memory, whether it is mutable or not, if it is safe to be reused in different threads, indeed even what the memory "points to."

On top of that, most of the time it's on the programmer to express to the implementation what the FFI even is. It's possible to get it wrong without actually breaking the program. You can do things like ignore signedness and discard mutability restrictions on pointers with ease in an FFI binding. There's nothing a language can do to prevent that, since the foreign code is a black box.

Now there are some exceptions to this, the most common probably being COM interfaces defined by IDL files, which are language agnostic and slightly safer than raw FFI over C functions. In this model, languages can provide some guarantees over what is happening across FFI and which operations are allowed (namely, reference counting semantics).

The way around all of this is simple : don't share memory between programs in different languages. Serialize to a data structure, call the foreign code in an isolated process, and only allow primitive data (like file descriptors) across FFI bounds.

This places enormous burden on language implementers, fwiw, which is probably why no one does it. FFI is too useful, and it's simple to tell a programmer not to screw up than reimplement POSIX in your language's core subroutines.

[1] "safe" rust, vs "unsafe" where memory unsafety is a promise upheld by the programmer, and violations may occur

Re: This shouldn't have happened: A vulnerability postmortem

#290

Post by sneak » Thu, Dec 02, 2021, 5:17 AM UTC

You'd think that in the wake of "goto fail;" the organization that pioneered rust would have rewritten their core TLS certificate checking library in rust by now, hacker news tropes notwithstanding.