Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why did the OpenSSL punycode vulnerability happen?

words.filippo.io

Post reply on HN
Print view View original

11–20 of 104 posts

Re: Why did the OpenSSL punycode vulnerability happen?

#11

Post by ahartmetz »

tester756 wrote: post #9
nightfly wrote: post #8

Earlier quoted context omitted.

Probably low level + safe + performant == hard

But why? where does the complexity come from

Parsing code without any kind of framework or high level helpers is, for lack of a better word, fiddly. C strings are an awful tool to do it because memory safety depends on getting a lot of buffer size / string length calculations right - there are plenty of opportunities to make mistakes. The fiddliness also induces developers to make changes in existing code in the form of local clever tricks instead of adapting the code to cleanly implement new requirements. It's only a small change and the tests (hopefully there are any) pass, job done! But maybe it violates a non-obvious assumption for another clever trick somewhere else, etc...

Re: Why did the OpenSSL punycode vulnerability happen?

#12

Post by 0xbadcafebee »

tester756 wrote: post #9
nightfly wrote: post #8

Earlier quoted context omitted.

Probably low level + safe + performant == hard

But why? where does the complexity come from

The first level of complexity comes from the format. A bit array is super easy to parse (in C, and assuming you take care of endianness). JSON is more complicated; YAML is more complicated than JSON; XML is more complicated than YAML; X.509 is more complicated than XML (I think, anyway). The more complex the data format, the more complex the parsing; the more complex; the more opportunity for bugs.

The second level of complexity comes from variability. A bit array doesn't vary, every bit is in the same place. A string varies. Anything that can vary causes complexity; more varying, more complexity. This applies to the data format and the data.

The third level of complexity comes from features. Every feature is a new thing that has to be parsed and then affects some code somewhere, the result of which affects more parsing and code. The more features and options there are, the more complexity.

"Why does it seem easier in high-level languages?" High-level languages have slowly had their bugs stripped out, and give you features that are rarer in low-level languages. You literally aren't writing the same routines in high-level languages because you don't need to. If you had to do all the same things, you'd have the same bugs. And a lot of newbies simply are lucky and don't personally run into the bugs that are already there.

Re: Why did the OpenSSL punycode vulnerability happen?

#13

Post by insanitybit »

tester756 wrote: post #7

It feels like issues like those are more common in parsers, this specific kind of software. But why? Why is parsing so hard? or is it just in low lvl languages? or maybe languages with poor string primitives? I've written parsers in high level languages and it didnt felt dangerous or insanely hard

I think it's easy to hand write a parser that can get into a weird state. Some formats are also easier to write a parser for safely vs others.

For example, if you have a Tag Length Value data format that says "5ABCDE" maybe that means "the next 5 bytes are a string, then there's something else afterwards or it's the end", you don't want to allocate a buffer of 5 values and just keep writing into it without checking that you're reading valid values.

Doing this sort of thing efficiently may also mean that you're tempted to remove something like a bounds check. After all, if you already know that the value is 5 bytes, why check on every access? People really want parsers to be fast. Ideally in the above example I wouldn't even need to copy those bytes out, I could just reference those values, which now leads to potential lifetime issues.

Further, in C, buffers don't have lengths attached to them. In every other language you typically don't work with null terminated strings. So now you have to manage sizes of things throughout your parser.

One way to view this is that the programmer's mental model of the parsing machine can easily drift from the implementation of the parsing machine, leading to vulnerabilities.

Basically it ends up being very easy to accidentally end up with out of bounds reads/writes + there's pressure to be fast + formats can be very complex.

That's my view on it at least.

Re: Why did the OpenSSL punycode vulnerability happen?

#14

Post by roblabla »

tester756 wrote: post #9
nightfly wrote: post #8

Earlier quoted context omitted.

Probably low level + safe + performant == hard

But why? where does the complexity come from

parsers and serializers have one thing they often do: read (and write) to a (usually manually allocated) byte array. And the content of that byte array is often under attacker control.

C has terrible support for things dealing with byte arrays. They must be manually allocated, and accesses must be checked to be in-bound manually.

Lots of critical software have parsers written in C. This combination leads to CVEs like this one.

FWIW, a bug such as this one (which ends up with an invalid array access) could happen in any language, and would end up with a panic in Rust, or a NullPointerException in Java, etc... The thing that makes this especially dangerous is that, because C is low-level and unchecked, this can also lead to Remote Code Execution instead of a simple Denial Of Service/crash.

Re: Why did the OpenSSL punycode vulnerability happen?

#15

Post by ahartmetz »

tux3 wrote: post #10
tester756 wrote: post #9

Earlier quoted context omitted.

But why? where does the complexity come from

There are naturally lots of edge cases when you parse a format, because you have to constrain the combination of all the different fields. Some formats are simple and the fields don't interact with each other at all, some are complex and the format changes depending on other values. Parsing is hard because you have to handle all the possible inputs someone could throw at you, and depending on the format that can leav…

You are of course correct. This also ties into my sibling comment - the cleverness in clever low-level parsing code is based on assumptions that may be wrong for some part of the huge input space.

Re: Why did the OpenSSL punycode vulnerability happen?

#16

Post by ninkendo »

tester756 wrote: post #7

It feels like issues like those are more common in parsers, this specific kind of software. But why? Why is parsing so hard? or is it just in low lvl languages? or maybe languages with poor string primitives? I've written parsers in high level languages and it didnt felt dangerous or insanely hard

Poor string primitives is the answer, I think. Or more generally, C’s tendency to require a buffer of sufficient size to be passed for output, and the complexity involved with:

    - What size to pick for the buffer
    - Making sure the buffer is of the size you think it is
    - What to do when the buffer isn’t big enough
    - How to know for sure that the buffer is big enough
A parser runs right into these problems quickly: An input string may have some formatting specifiers (%d, etc), may involve control characters that cause the output buffer to expand, etc etc… Any function that deals with this kind of thing has to do be able to correctly tell the caller that their buffer wasn’t big enough, and by how much, and has to make sure it didn’t accidentally write past the end of the buffer before doing this.

Callers also have to make sure the buffer size they’re passing to the parser is actually accurate… you don’t want to malloc 100 bytes and tell the parser the buffer is 200. They also have to make sure that if they say the buffer is 100 bytes, that the string actually terminates with a \0 by the end of it, or it gets even more confusing.

It’s overall a complicated problem, and arises specifically because of the tendency of C library functions to avoid allocations… because the ownership model of idiomatic C is that callers should be the ones that allocate (because they probably know best how to allocate and when to free, etc.)

Re: Why did the OpenSSL punycode vulnerability happen?

#17

Post by tester756 »

insanitybit wrote: post #13
tester756 wrote: post #7

It feels like issues like those are more common in parsers, this specific kind of software. But why? Why is parsing so hard? or is it just in low lvl languages? or maybe languages with poor string primitives? I've written parsers in high level languages and it didnt felt dangerous or insanely hard

I think it's easy to hand write a parser that can get into a weird state. Some formats are also easier to write a parser for safely vs others. For example, if you have a Tag Length Value data format that says "5ABCDE" maybe that means "the next 5 bytes are a string, then there's something else afterwards or it's the end", you don't want to allocate a buffer of 5 values and just keep writing into it without checking t…

>Doing this sort of thing efficiently may also mean that you're tempted to remove something like a bounds check. After all, if you already know that the value is 5 bytes, why check on every access? People really want parsers to be fast.

In what kind of software bounds check have this significant perf. penalty?

The only people I've heard talking about such a stuff were firmware devs.

>Further, in C, buffers don't have lengths attached to them. In every other language you typically don't work with null terminated strings. So now you have to manage sizes of things throughout your parser.

Cannot C have some wrapper over those poor strings

that leads to better safety? dev's experience, etc, etc?

Re: Why did the OpenSSL punycode vulnerability happen?

#18

Post by alserio »

tester756 wrote: post #7

It feels like issues like those are more common in parsers, this specific kind of software. But why? Why is parsing so hard? or is it just in low lvl languages? or maybe languages with poor string primitives? I've written parsers in high level languages and it didnt felt dangerous or insanely hard

I'll give you a tautological and useless answer. I hope you don't mind. A parser is a kind of interpreter, where code is executed based on external input. When user input controls how code is executed you have opened the doors of hell: it's hard to guarantee that all of the possible executions are safe.

Re: Why did the OpenSSL punycode vulnerability happen?

#19

Post by tester756 »

alserio wrote: post #18
tester756 wrote: post #7

It feels like issues like those are more common in parsers, this specific kind of software. But why? Why is parsing so hard? or is it just in low lvl languages? or maybe languages with poor string primitives? I've written parsers in high level languages and it didnt felt dangerous or insanely hard

I'll give you a tautological and useless answer. I hope you don't mind. A parser is a kind of interpreter, where code is executed based on external input. When user input controls how code is executed you have opened the doors of hell: it's hard to guarantee that all of the possible executions are safe.

But why?

Parsers executes safe code fragments in order based on the input

How safe operations result in unsafe results?

I'm not talking about side channels here.

Re: Why did the OpenSSL punycode vulnerability happen?

#20

Post by bhk »

tester756 wrote: post #7

It feels like issues like those are more common in parsers, this specific kind of software. But why? Why is parsing so hard? or is it just in low lvl languages? or maybe languages with poor string primitives? I've written parsers in high level languages and it didnt felt dangerous or insanely hard

It's not hard to use a "vector" library or even just a small set of macros that ensure memory safety when you are manipulating slices of strings or buffers.

Unfortunately it's very tempting to play fast and loose with raw pointers, with ad hoc validation logic all over the place.

OTOH, maybe SSL maintainers consider punycode parsing performance more important than security.

Post reply on HN