Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

String tokenization in C

onebyezero.blogspot.com

Post reply on HN
Print view View original

51–60 of 117 posts

Re: String tokenization in C

#51

Post by Someone »

yason wrote: post #25
jstimpfle wrote: post #12

strtok is one of the silliest parts of the standard library. (And there are many bad ones). It's broken. It's not thread safe (yes there is strtok_r). It's needlessly hard to use. And it writes zeros to the input array. The latter means it's unfit for most use cases, including non-trivial tokenization where you want e.g. to split "a+1" into three tokens. If you program in C please just write those four obvious lines…

If you program in C please just write those four obvious lines yourself. Those are not necessarily obvious lines, there are several pitfalls to avoid, and for that reason strtok() is much longer than four lines. When it comes to the standard library functions strtok() has well-defined behaviour that is easy to reason with and near-magically approaches the string-splitting convenience close to scripting languages. In…

> A few strtok() calls can easily parse lines like:

     keyword=value1, value2, value3
The challenge with parsing isn’t parsing correct inputs; it’s generating useful error messages and recovering on incorrect inputs such as

     keyword=,,value1, value2, value3,,,,
or even

     =keyword=,,value1, value2, value3,,,,
strtok isn’t the best tool for doing that.

(Yes, those could be valid inputs, but if they are, chances are they should be parsed differently)

Re: String tokenization in C

#52

Post by tptacek »

beefhash wrote: post #39
tptacek wrote: post #16

Earlier quoted context omitted.

It's a tiny function, written in ANSI C, so if you're really concerned about this, just include it with your program. It's an extension to the standard C library , not to C itself.

Except then you have the issue about compilers complaining about double-declarations of the function, meaning you'll either have a lot of warning spam on every #include or now hard require some kind of header defines for HAVE_STRSEP. Once you go that way, there's no going back and it's only gonna become more and more.

People have been including strsep in packages since the 1990s (people used to include their own snprintfs, ffs). If you're really this freaked out about it, call your local copy "mystrsep" or something like that.

You know what else isn't in POSIX? All the rest of your C code.

Re: String tokenization in C

#53

Post by jstimpfle »

tails4e wrote: post #41
jstimpfle wrote: post #12

strtok is one of the silliest parts of the standard library. (And there are many bad ones). It's broken. It's not thread safe (yes there is strtok_r). It's needlessly hard to use. And it writes zeros to the input array. The latter means it's unfit for most use cases, including non-trivial tokenization where you want e.g. to split "a+1" into three tokens. If you program in C please just write those four obvious lines…

strncat is another good example. It's a buffer overflow waiting to happen, and worse it's the 'safe' version of strcat. The trick is the parameter you pass is not the length of the destination buffer, but the remaining length of the destination buffer. Most people really want the behaviour of strlcat.

I'd say just use memcpy.

Re: String tokenization in C

#54

Post by dahart »

jstimpfle wrote: post #48
dahart wrote: post #46

Earlier quoted context omitted.

> Yout don’t really use regex in C. Speak for yourself. There’s a POSIX standard for regex that is more than 30 years old & a GNU implementation that comes with gcc. C++ has regex in the standard library.

"You" used like "one", or "in common practice it isn't really used". It's just not in C's spirit to use canned libraries. Such use cases have long been transferred to Python and other languages. Sure, there is a regex API in the C standard library. I'd bet not even grep uses it. The one use-case I'm envisioning is quickly exposing POSIX conformant regexen to the command-line.

> I’d bet not even grep uses it.

I’ll take you up on that bet. I see #include in every source repo of grep I can find right now.

http://git.savannah.gnu.org/cgit/grep.git/tree/src/search.h

https://opensource.apple.com/source/text_cmds/text_cmds-99/g...

https://android.googlesource.com/platform/system/core.git/+/...

https://github.com/c9/node-gnu-tools/blob/master/grep-src/sr...

You owe me a beer. :)

BTW, I do super agree with your comment to just not use strtok, and also the idea that most people are better off parsing text in perl or python...

Re: String tokenization in C

#56

Post by tptacek »

jstimpfle wrote: post #53
tails4e wrote: post #41

Earlier quoted context omitted.

strncat is another good example. It's a buffer overflow waiting to happen, and worse it's the 'safe' version of strcat. The trick is the parameter you pass is not the length of the destination buffer, but the remaining length of the destination buffer. Most people really want the behaviour of strlcat.

I'd say just use memcpy.

How does that help? You have the same bookkeeping problem with memcpy as you do with strncat.

Re: String tokenization in C

#57

Post by nwmcsween »

nly wrote: post #14
jstimpfle wrote: post #12

strtok is one of the silliest parts of the standard library. (And there are many bad ones). It's broken. It's not thread safe (yes there is strtok_r). It's needlessly hard to use. And it writes zeros to the input array. The latter means it's unfit for most use cases, including non-trivial tokenization where you want e.g. to split "a+1" into three tokens. If you program in C please just write those four obvious lines…

Much of libc is terrible from an API design perspective, even given the limitations of C as a language. libc has somehow managed to hit the sweet spot and have APIs that are both inconvenient to use properly, and perform poorly.

libc aka ISO, POSIX, GNU wasn't designed it was more or less evolved from what was in use.

Re: String tokenization in C

#58

Post by dkrikun »

jstimpfle wrote: post #24
agumonkey wrote: post #19

Earlier quoted context omitted.

any attempt has been made to build a nicer foundational C/OS library since ?

There are of course tons of code on the net, but there is no need to standardize another grab-bag of bad API calls. If you need a batteries-included library you use python or similar. If you write in C, you care so much that you largely avoid the standard library and design your code from the ground up.

There are numerous reasons to write in C, apart from caring so much. Using strtok() is not that bad after all because its a simple function after all.

Re: String tokenization in C

#59

Post by jstimpfle »

dahart wrote: post #54
jstimpfle wrote: post #48

Earlier quoted context omitted.

"You" used like "one", or "in common practice it isn't really used". It's just not in C's spirit to use canned libraries. Such use cases have long been transferred to Python and other languages. Sure, there is a regex API in the C standard library. I'd bet not even grep uses it. The one use-case I'm envisioning is quickly exposing POSIX conformant regexen to the command-line.

> I’d bet not even grep uses it. I’ll take you up on that bet. I see #include in every source repo of grep I can find right now. http://git.savannah.gnu.org/cgit/grep.git/tree/src/search.h https://opensource.apple.com/source/text_cmds/text_cmds-99/g... https://android.googlesource.com/platform/system/core.git/+/... https://github.com/c9/node-gnu-tools/blob/master/grep-src/sr... You owe me a beer. :) BTW, I do super a…

Not sure why that header is included there, though. I can't find any uses of the regex library. There are multiple custom matchers implemented. So maybe it's that the GNU regex library uses the grep sources. In any case, there don't seem to be any uses of regexec() or regcomp(), for example. Which would have surprised me anyway since that API is rather limiting (you cannot search incrementally).

Let's get that beer sometime, anyway.

Post reply on HN