String tokenization in C • Hacker News Board

Re: String tokenization in C

#101

Post by loeg » Sat, Dec 15, 2018, 11:00 PM UTC

avar wrote: post #96

jstimpfle wrote: post #63

Earlier quoted context omitted.

I look at it the other way: I've hard coded the reading and writing routines inside the tokenization logic. Being able to do that is exactly the point why it's so much simpler to avoid a silly API such as strcspn (or, god forbid, strtok). > non-ASCII yeah i know... Do you prefer strcspn("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVXYZ")? Do you think it's faster? If you're pedantic, you could lex (0x41 > I suppos…

> If you're pedantic, you could lex (0x41 'A' v.s. 0x41 makes no difference for portability. The thing that's unportable about that is that it assumes that the characters A..Z are continuous in your character encoding, which isn't portable C. Although admittedly having to deal with EBCDIC these days is rare in anything except highly portable programs like C compilers or popular script interpreters. This is why ctype.…

> 'A' v.s. 0x41 makes no difference for portability. The thing that's unportable about that is that it assumes that the characters A..Z are continuous in your character encoding, which isn't portable C.

Wait, what? If C does not require A..Z to be contiguous, the distinction between 'A' and 0x41 is extremely significant to portable programs intending to parse ASCII when the native compiler encoding is whatever franken-coding doesn't have contiguous latin characters.

Re: String tokenization in C

#102

Post by zozbot123 » Sat, Dec 15, 2018, 11:03 PM UTC

loeg wrote: post #101

avar wrote: post #96

Earlier quoted context omitted.

> If you're pedantic, you could lex (0x41 'A' v.s. 0x41 makes no difference for portability. The thing that's unportable about that is that it assumes that the characters A..Z are continuous in your character encoding, which isn't portable C. Although admittedly having to deal with EBCDIC these days is rare in anything except highly portable programs like C compilers or popular script interpreters. This is why ctype.…

> 'A' v.s. 0x41 makes no difference for portability. The thing that's unportable about that is that it assumes that the characters A..Z are continuous in your character encoding, which isn't portable C. Wait, what? If C does not require A..Z to be contiguous, the distinction between 'A' and 0x41 is extremely significant to portable programs intending to parse ASCII when the native compiler encoding is whatever franke…

EBCDIC famously does not have A..Z as contiguous characters, and I wouldn't describe it as a 'franken-coding' just yet - it still finds plenty of use in some places.

Re: String tokenization in C

#103

Post by fork1 » Sat, Dec 15, 2018, 11:35 PM UTC

zozbot123 wrote: post #102

loeg wrote: post #101

Earlier quoted context omitted.

> 'A' v.s. 0x41 makes no difference for portability. The thing that's unportable about that is that it assumes that the characters A..Z are continuous in your character encoding, which isn't portable C. Wait, what? If C does not require A..Z to be contiguous, the distinction between 'A' and 0x41 is extremely significant to portable programs intending to parse ASCII when the native compiler encoding is whatever franke…

EBCDIC famously does not have A..Z as contiguous characters, and I wouldn't describe it as a 'franken-coding' just yet - it still finds plenty of use in some places.

Unless you're dealing with mainframes, it's not like you see it everyday.

Re: String tokenization in C

#104

Post by loeg » Sat, Dec 15, 2018, 11:40 PM UTC

zozbot123 wrote: post #102

loeg wrote: post #101

Earlier quoted context omitted.

> 'A' v.s. 0x41 makes no difference for portability. The thing that's unportable about that is that it assumes that the characters A..Z are continuous in your character encoding, which isn't portable C. Wait, what? If C does not require A..Z to be contiguous, the distinction between 'A' and 0x41 is extremely significant to portable programs intending to parse ASCII when the native compiler encoding is whatever franke…

EBCDIC famously does not have A..Z as contiguous characters, and I wouldn't describe it as a 'franken-coding' just yet - it still finds plenty of use in some places.

EBCDIC is a classic example of a franken-coding.

If your compiler's source character set is EBCDIC and you want to parse ASCII files, you must use 0x41, etc, instead of 'A'.

Re: String tokenization in C

#105

Post by avar » Sat, Dec 15, 2018, 11:52 PM UTC

loeg wrote: post #101

avar wrote: post #96

Earlier quoted context omitted.

> If you're pedantic, you could lex (0x41 'A' v.s. 0x41 makes no difference for portability. The thing that's unportable about that is that it assumes that the characters A..Z are continuous in your character encoding, which isn't portable C. Although admittedly having to deal with EBCDIC these days is rare in anything except highly portable programs like C compilers or popular script interpreters. This is why ctype.…

> 'A' v.s. 0x41 makes no difference for portability. The thing that's unportable about that is that it assumes that the characters A..Z are continuous in your character encoding, which isn't portable C. Wait, what? If C does not require A..Z to be contiguous, the distinction between 'A' and 0x41 is extremely significant to portable programs intending to parse ASCII when the native compiler encoding is whatever franke…

Yes. If the problem was trying to parse ASCII consistently that would be the right solution.

My response was to OPs moving the goal post to "portably parsing ASCII" in response to his suggested replacement for a C library function not being portable on non-ASCII systems, which make no sense.

Re: String tokenization in C

#106

Post by satyenr » Sun, Dec 16, 2018, 5:45 AM UTC

> Next, strtok is not thread-safe. That's because it uses a static buffer internally. So, you should take care that only one thread in your program calls strtok at a time.

I wonder why strtok() does not use an output parameter similar to scanf() — and return the number of tokens. Something like:

  int strtok(char *str, char *delim, char **tokens);

Granted, it would involve dynamic memory allocation and the implementation that immediately comes to mind would be less efficient than the current implementation, but surely it’s worth eliminating the kind of bugs the current strtok() can introduce?

Does anyone here have the historical prospective?

Re: String tokenization in C

#107

Post by Hypergraphe » Sun, Dec 16, 2018, 9:57 AM UTC

jstimpfle wrote: post #12

strtok is one of the silliest parts of the standard library. (And there are many bad ones). It's broken. It's not thread safe (yes there is strtok_r). It's needlessly hard to use. And it writes zeros to the input array. The latter means it's unfit for most use cases, including non-trivial tokenization where you want e.g. to split "a+1" into three tokens. If you program in C please just write those four obvious lines…

Agreed, I basically avoid using strtok because of that. Why would you write zeros in my input...

Re: String tokenization in C

#108

Post by yason » Sun, Dec 16, 2018, 11:33 AM UTC

Someone wrote: post #51

yason wrote: post #25

Earlier quoted context omitted.

If you program in C please just write those four obvious lines yourself. Those are not necessarily obvious lines, there are several pitfalls to avoid, and for that reason strtok() is much longer than four lines. When it comes to the standard library functions strtok() has well-defined behaviour that is easy to reason with and near-magically approaches the string-splitting convenience close to scripting languages. In…

> A few strtok() calls can easily parse lines like: keyword=value1, value2, value3 The challenge with parsing isn’t parsing correct inputs; it’s generating useful error messages and recovering on incorrect inputs such as keyword=,,value1, value2, value3,,,, or even =keyword=,,value1, value2, value3,,,, strtok isn’t the best tool for doing that. (Yes, those could be valid inputs, but if they are, chances are they shou…

That's what I meant: strtok() is well-defined in what it does. The man page is really short. I don't understand people who complain it doesn't do something it's not supposed to do. Yet it's very useful in what it does.

Your mileage may vary but it's also a common issue having to filter out empty items out of something like ",foo,bar ,,,,baz,,xyzzy," where you only really want those four words. You will especially encounter this in parsing user input which might have extra whitespace, or badly formed lists of items.

Use strtok() to split C strings into tokens delimited by the given set of delimiters. If you want to catch each comma but skip over any whitespace and split at the first '=' only, then use something else.

Let's say if I'm going to need a configuration file for my program I'd most certainly start with something I can parse with strtok(). I would really need very specific needs to warrant a more complex format that would require a more sophisticated parser in which case using one would be a no-brainer.

Re: String tokenization in C

#109

Post by yason » Sun, Dec 16, 2018, 11:45 AM UTC

jstimpfle wrote: post #28

yason wrote: post #25

Earlier quoted context omitted.

If you program in C please just write those four obvious lines yourself. Those are not necessarily obvious lines, there are several pitfalls to avoid, and for that reason strtok() is much longer than four lines. When it comes to the standard library functions strtok() has well-defined behaviour that is easy to reason with and near-magically approaches the string-splitting convenience close to scripting languages. In…

Here is the musl implementation. > https://github.com/esmil/musl/blob/master/src/string/strtok.... It's a bit longer than 4 lines because strtok does things you should not want. If you insist on parsing that configuration line with strtok, go ahead and write that brittle code. It breaks as soon as you want empty strings (try "keyword=value1, , value3" with strtok) or escape sequences or other transformations, or as s…

So, here's that implementation:

    static char *p;
    if (!s && !(s = p)) return NULL;
    s += strspn(s, sep);
    if (!*s) return p = 0;
    p = s + strcspn(s, sep);
    if (*p) *p++ = 0;
    else p = 0;
    return s;

Instead of carrying that code, or something similar, with my source code or my own utility library I'd much rather have the already debugged version from the standard library.

Overwriting the input in C is more efficient than maintaining more internal state and returning a pointer and the length of each token which you would need to strncpy() to get the token into a C string. strtok() does not want to do the initial strdup() for you because only you will know whether your input can already be mutated or whether you need to use a copy.

As I pointed in the other reply, strtok() does not break on strings like "keyword=value1,, , value3" unless you skipped RTFM and expect it to do something completely different. And more often than not that's exactly what you want when parsing non-computer readable input which you can expect to take a specific form.

If you want to handle escape sequences, parse from a stream (without having the option to fgets() the next line into memory), or parse CSV tables without collapsin colunms then you will want to use something more specific to that. Luckily, strtok() was not advertised as a Swiss army knife so it's off the hook for specific parsing purposes like those.

Re: String tokenization in C

#110

Post by yason » Sun, Dec 16, 2018, 11:49 AM UTC

nly wrote: post #30

yason wrote: post #25

Earlier quoted context omitted.

If you program in C please just write those four obvious lines yourself. Those are not necessarily obvious lines, there are several pitfalls to avoid, and for that reason strtok() is much longer than four lines. When it comes to the standard library functions strtok() has well-defined behaviour that is easy to reason with and near-magically approaches the string-splitting convenience close to scripting languages. In…

Unless you're using a pre-specified configuration file format (e.g. TOML), then parsing configuration files requires a general parsing library. This is a non-trivial task requiring a real parser operating over a well-specified grammar. A tokenization pipeline just won't cut it. I worked on a project a few years ago that read its custom-format config file in line by line, chopped everything off each line following the…

Unless you're using a pre-specified configuration file format (e.g. TOML), then parsing configuration files requires a general parsing library.

If you have needs that require a general parsing library then why are you criticizing strtok()? It doesn't parse XML either, not C source code, nor any unspecified configuration file formats.