Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

String tokenization in C

onebyezero.blogspot.com

Post reply on HN
Print view View original

111–117 of 117 posts

Re: String tokenization in C

#111

Post by jstimpfle »

avar wrote: post #94
jstimpfle wrote: post #12

strtok is one of the silliest parts of the standard library. (And there are many bad ones). It's broken. It's not thread safe (yes there is strtok_r). It's needlessly hard to use. And it writes zeros to the input array. The latter means it's unfit for most use cases, including non-trivial tokenization where you want e.g. to split "a+1" into three tokens. If you program in C please just write those four obvious lines…

> it writes zeros to the input array [which] means it's unfit for most use cases[...] Other issues with strtok() aside, this seems like a silly reason to discount a standard library function. If you don't want your input munged you can strdup() it. It's rare to find a C program that's so specialized that the performance hit of a strdup() would be unacceptable in a case where strtok() could otherwise have been used.

It would be wasteful to duplicate the input string which includes so much garbage. I would rather just go through the input string and append token by token to the output string, terminated with a single NUL.

Re: String tokenization in C

#112

Post by jstimpfle »

avar wrote: post #105
loeg wrote: post #101

Earlier quoted context omitted.

> 'A' v.s. 0x41 makes no difference for portability. The thing that's unportable about that is that it assumes that the characters A..Z are continuous in your character encoding, which isn't portable C. Wait, what? If C does not require A..Z to be contiguous, the distinction between 'A' and 0x41 is extremely significant to portable programs intending to parse ASCII when the native compiler encoding is whatever franke…

Yes. If the problem was trying to parse ASCII consistently that would be the right solution. My response was to OPs moving the goal post to "portably parsing ASCII" in response to his suggested replacement for a C library function not being portable on non-ASCII systems, which make no sense.

I explicitly wrote "or whatever test" as a comment to the code snippet, and obviously the test was not the point of my comment.

Anyway I think most programming languages nowadays have their source encoding specified as UTF-8 or at least something ASCII-like, so ('A' <= c && c <= 'Z') is in fact what I would likely write, and using isalpha() would technically be a bug just as well.

Re: String tokenization in C

#113

Post by jstimpfle »

kahirsch wrote: post #86
jstimpfle wrote: post #74

Earlier quoted context omitted.

I was saying "canned", not "boxed" or "containerized". It's a lot easier to write the little things yourself. (And there are good benefits to be had from writing specialized code yourself, instead of relying on big fat generalized tankers.)

I don't know what distinction you're trying to make about "canned" vs. other things. There are advantages and disadvantages of using libraries in any language. There are some aspects of C —e.g. the lack of garbage collection— that make it harder to reuse code compared to some languages. But it is definitely not against "C's spirit".

The distinction is one of size and number. An npm for C would make no sense.

Re: String tokenization in C

#114

Post by jstimpfle »

yason wrote: post #109
jstimpfle wrote: post #28

Earlier quoted context omitted.

Here is the musl implementation. > https://github.com/esmil/musl/blob/master/src/string/strtok.... It's a bit longer than 4 lines because strtok does things you should not want. If you insist on parsing that configuration line with strtok, go ahead and write that brittle code. It breaks as soon as you want empty strings (try "keyword=value1, , value3" with strtok) or escape sequences or other transformations, or as s…

So, here's that implementation: static char *p; if (!s && !(s = p)) return NULL; s += strspn(s, sep); if (!*s) return p = 0; p = s + strcspn(s, sep); if (*p) *p++ = 0; else p = 0; return s; Instead of carrying that code, or something similar, with my source code or my own utility library I'd much rather have the already debugged version from the standard library. Overwriting the input in C is more efficient than main…

As someone else pointed out what you really want is the implementation of strspn/strcspn which is where the loop is. You don't "carry" that code along. You just write

    while (i
or something along those lines. Whatever you need. It's not rocket science. Putting highly fluctuating and project-specific code like this in a library would only have disadvantages. Not everything should be in a library. In fact, most things should not be.

Re: String tokenization in C

#115

Post by bsenftner »

syrrim wrote: post #91
bsenftner wrote: post #43

...And then the application is required to implement variable length characters, a la Unicode, and you start your strings logic all over...

As long as you're fine with ascii delimiters, strtok et al. work fine for utf-8 strings.

Would you happen to be aware of good Unicode normalization function/lib in C/C++?

Re: String tokenization in C

#116

Post by rurban »

tails4e wrote: post #41
jstimpfle wrote: post #12

strtok is one of the silliest parts of the standard library. (And there are many bad ones). It's broken. It's not thread safe (yes there is strtok_r). It's needlessly hard to use. And it writes zeros to the input array. The latter means it's unfit for most use cases, including non-trivial tokenization where you want e.g. to split "a+1" into three tokens. If you program in C please just write those four obvious lines…

strncat is another good example. It's a buffer overflow waiting to happen, and worse it's the 'safe' version of strcat. The trick is the parameter you pass is not the length of the destination buffer, but the remaining length of the destination buffer. Most people really want the behaviour of strlcat.

Nope, the safe versions of strcat are strcat_s and strncat_s, with bounds check and ensured NULL termination. strlcat does no bounds check.

Re: String tokenization in C

#117

Post by nly »

yason wrote: post #110
nly wrote: post #30

Earlier quoted context omitted.

Unless you're using a pre-specified configuration file format (e.g. TOML), then parsing configuration files requires a general parsing library. This is a non-trivial task requiring a real parser operating over a well-specified grammar. A tokenization pipeline just won't cut it. I worked on a project a few years ago that read its custom-format config file in line by line, chopped everything off each line following the…

Unless you're using a pre-specified configuration file format (e.g. TOML), then parsing configuration files requires a general parsing library. If you have needs that require a general parsing library then why are you criticizing strtok()? It doesn't parse XML either, not C source code, nor any unspecified configuration file formats.

My point is that strtok has almost no reasonable uses in real programs.
Post reply on HN