Did Ken, Ritchie and Brian choose wrong with NUL-terminated text strings? (2011) • Hacker News Board

Re: Did Ken, Ritchie and Brian choose wrong with NUL-terminated text strings? (2011)

#41

Post by mikestew » Tue, Jan 16, 2018, 7:35 PM UTC

korethr wrote: post #24

tptacek wrote: post #9

Worth keeping in mind that the security cost here is illusory: the track record of length-delimited data structures isn't much better than that of ASCIIZ, especially since integer handling in C is so treacherous.

> ...especially since integer handling in C is so treacherous. I'm still learning C, having not had much reason to do so until recently, so I'm not quite sure I understand this statement. How is integer handling in C treacherous (any more so than any other language, especially other languages that operate as close to the metal as C)? Is it something to do with signed vs unsigned, and beware of over/under-flow when do…

Here’s a lovely real-world example, discussed just recently here on HN: https://kotaku.com/why-gandhi-is-such-an-asshole-in-civiliza...

Unfortunately, most integer issues don’t have outcomes that are nearly that amusing.

Re: Did Ken, Ritchie and Brian choose wrong with NUL-terminated text strings? (2011)

#42

Post by roywiggins » Tue, Jan 16, 2018, 7:38 PM UTC

bobsc123 wrote: post #17

Null terminated strings are just application level logic. "strings" are just bytes in memory. There are no strings.

> Should the C language represent strings as an address + length tuple or just as the address with a magic character (NUL) marking the end?

This explicitly is a different kind of representation in memory.

Re: Did Ken, Ritchie and Brian choose wrong with NUL-terminated text strings? (2011)

#43

Post by callesgg » Tue, Jan 16, 2018, 7:40 PM UTC

bobsc123 wrote: post #17

Null terminated strings are just application level logic. "strings" are just bytes in memory. There are no strings.

No, If you quote a string in c "string data here" you get a piece of data with null termination.

Null termination is part of the C language.

Example: char str[]= "1234";

    printf ("%s: %lu", str, sizeof(str));

Prints: 1234: 5

Re: Did Ken, Ritchie and Brian choose wrong with NUL-terminated text strings? (2011)

#44

Post by pishpash » Tue, Jan 16, 2018, 7:41 PM UTC

JoachimSchipper wrote: post #37

jandrese wrote: post #26

Earlier quoted context omitted.

Interesting some Unix command line utilities will send null separated records if you pass a flag (often -0) because it's the least likely character to show up as part of the string. find and xargs are examples of programs with this feature. It depends a bit on what you call a "string". If you're thinking "something a human will want to read", then yeah, there's no much need to encode null. If however you take a loose…

Elaborating on the NUL bytes on the command-line, e.g. find -print0 | xargs -0: Using find -print0 etc. is a good idea not so much because NUL is an uncommon character (the various record separators / vertical tab / ... are no more common), but because UNIX - being a C system through and through - allows any character to appear in a file name except '/' (path separator) and NUL. Thus, NUL makes a perfect separator be…

Then it sounds like 'find' f'ed up, if, when these things are passed around, they are not escaped properly (not saying this is the case). Just like today with various charsets, whenever there is a charset boundary, say between bytes and C library strings, which is what this is, there has to be a charset conversion.

Re: Did Ken, Ritchie and Brian choose wrong with NUL-terminated text strings? (2011)

#45

Post by vorg » Tue, Jan 16, 2018, 7:42 PM UTC

jerf wrote: post #18

function_seven wrote: post #16

Earlier quoted context omitted.

A string contains characters. NUL is not a character; it's nothing. "Fundamental" in this case means "matches reality". Having a number at the beginning doesn't match reality as closely as having the string of characters in sequential memory addresses with something to terminate them. The quick fox made the jump\N or 27The quick fox made the jump The second one requires more work to store (a character-counting routin…

"A string contains characters. NUL is not a character; it's nothing." You already couldn't make this argument stick in the ASCII era, where a string can't contain NUL but can contain SOH (Start of Heading), STX (Start of Text), ETX (End of Text), EOT (End of Transmission), ENQ (Enquiry), ACK (Acknowledge), BEL, BS, HT (horizontal tab), LF, VT (vertical tab), FF (form feed), CR, SO (shift out), SI (shift in), DLE (dat…

> but Unicode makes that argument even sillier

Unicode Standard (version 10.0, section 23.1 Control Codes) makes it clear that it "specifies semantics for the use" of only 9 of those ASCII control codes you mentioned, i.e. U+0009 to U+000D (HT, LF, VT, FF, CR) and U+001C to U+001F (RS, GS, RS, US). The rest of the 65 ASCII and Latin-1 control codes, except U+0085 (NEL), "constitute a higher-level protocol that is outside the scope of the Unicode Standard".

Particularly about NUL, it says: "U+0000 null may be used as a Unicode string terminator, as in the C language. Such usage is outside the scope of the Unicode Standard, which does not require any particular formal language representation of a string or any particular usage of null."

So Unicode makes that argument less silly.

Re: Did Ken, Ritchie and Brian choose wrong with NUL-terminated text strings? (2011)

#46

Post by Sir_Cmpwn » Tue, Jan 16, 2018, 7:59 PM UTC

>The CPUs that offered string manipulation instructions—for example, Z-80 and DEC VAX—did so in terms of the far more widespread adr+len model.

Hold up, the z80 offered string maniuplation instructions? Using adr+len, no less? You miiiight call LDIR a string manipulation function using adr+len but in practice almost all z80 machines I've seen use NUL terminated strings and something like CPIR to find it.

Re: Did Ken, Ritchie and Brian choose wrong with NUL-terminated text strings? (2011)

#47

Post by gpvos » Tue, Jan 16, 2018, 8:00 PM UTC

saint_fiasco wrote: post #13

IshKebab wrote: post #10

Earlier quoted context omitted.

It's not fundamental at all. You can't even represent null bytes in a null-terminated string. A length prefix is pretty clearly superior.

Why would you want to represent a null byte in a string? Is there a character encoding where the null value has a meaning?

Because you just read it from a file and don't want to corrupt it?

Re: Did Ken, Ritchie and Brian choose wrong with NUL-terminated text strings? (2011)

#48

Post by kbenson » Tue, Jan 16, 2018, 8:21 PM UTC

PopsiclePete wrote: post #2

>Using an address + length format would cost one more byte of overhead than an address + magic_marker format, and their PDP computer had limited core memory. I find this interesting - why only one more byte of overhead? That would've limited string lengths to 256. So 2 bytes would seem the minimum, and even then, how do you go to 4 bytes once memory becomes cheap without breaking everything? Using NUL-termination, th…

There are various schemes to address this. A simple one is to make the legtth prefix variable, nad use the high order bit use the highorder bit of each byte to indicate there is another length byte following the current one. This gives you 128 possible values for each length byte. The max string size for a single byte prefix would be 127 (2^(8-1), the max size for two bytes would be 16384 (2^16-2), etc.

It's not necessarily a good solution, there are likely ones that are much more performant, but many solutions exist, and a few have even been implemented in widely used languages.

Re: Did Ken, Ritchie and Brian choose wrong with NUL-terminated text strings? (2011)

#49

Post by aplorbust » Tue, Jan 16, 2018, 8:23 PM UTC

There is one author who does not use memcpy, strcpy, etc. He wrote his own "standard" C library routines. Others have used these routines; they are public domain.

The security track record on the authors internet-facing programs is better than most. In fact, I cannot think of any author writing similar software with a better track record.

Sometimes the most popular solutions are not necessarily the best ones for every purpose. Whenever I write programs in C from scratch, I use byte.h, buffer.h, etc., from the above mentioned author. I do not use memcpy.

In doing this, I am not a professional programmer and I am not writing internet-facing programs for other users. I am a student of C learning how to use C, the language. If I know how the language works, then it stands to reason I should be able to use a variety of libraries, including alternatives to the "standard libraries".

Otherwise it is arguable I would be just learning how to use a standard library, not a language.

The C language has utility on its own, as form of a notation, and it is that utility which I seek to learn about. Historical records indicate there was C language in productive use for some time before there was a "standard library".

Re: Did Ken, Ritchie and Brian choose wrong with NUL-terminated text strings? (2011)

#50

Post by JoachimSchipper » Tue, Jan 16, 2018, 8:28 PM UTC

pishpash wrote: post #44

JoachimSchipper wrote: post #37

Earlier quoted context omitted.

Elaborating on the NUL bytes on the command-line, e.g. find -print0 | xargs -0: Using find -print0 etc. is a good idea not so much because NUL is an uncommon character (the various record separators / vertical tab / ... are no more common), but because UNIX - being a C system through and through - allows any character to appear in a file name except '/' (path separator) and NUL. Thus, NUL makes a perfect separator be…

Then it sounds like 'find' f'ed up, if, when these things are passed around, they are not escaped properly (not saying this is the case). Just like today with various charsets, whenever there is a charset boundary, say between bytes and C library strings, which is what this is, there has to be a charset conversion.

By default, find separates by newline; this is human-friendly, but breaks if an attacker/script/... puts a newline in the filename.

The UNIX filesystem, qua filesystem, doesn't have a character set, just NUL-terminated strings. On the plus side, it's simple to handle, and means that retrofitting UTF-8 or another encoding is pretty easy. On the downside, two bytestrings that Unicode-canonicalize to the same value may name different files, which is surprising for humans.

It's notable that many of early UNIX' competitors were much more full-fledged systems, featuring full-fledged record-oriented files and typed data instead of UNIX' bytestrings-everywhere approach.