...And then the application is required to implement variable length characters, a la Unicode, and you start your strings logic all over...
As long as you're fine with ascii delimiters, strtok et al. work fine for utf-8 strings.
String tokenization in C
91–100 of 117 posts
Re: String tokenization in C
#92strtok is one of the silliest parts of the standard library. (And there are many bad ones). It's broken. It's not thread safe (yes there is strtok_r). It's needlessly hard to use. And it writes zeros to the input array. The latter means it's unfit for most use cases, including non-trivial tokenization where you want e.g. to split "a+1" into three tokens. If you program in C please just write those four obvious lines…
There are valuable use cases where this matters. For example, parsing FIX messages in finance, this allows you to parse the tag/value pairs with no memory allocations, which matters in low latency HFT applications.
Re: String tokenization in C
#93strtok is one of the silliest parts of the standard library. (And there are many bad ones). It's broken. It's not thread safe (yes there is strtok_r). It's needlessly hard to use. And it writes zeros to the input array. The latter means it's unfit for most use cases, including non-trivial tokenization where you want e.g. to split "a+1" into three tokens. If you program in C please just write those four obvious lines…
There are valuable use cases where this matters. For example, parsing FIX messages in finance, this allows you to parse the tag/value pairs with no memory allocations, which matters in low latency HFT applications.
If you want speed, just build a lexer with ragel [1]. It's hard to go faster than a DFA.
Re: String tokenization in C
#94strtok is one of the silliest parts of the standard library. (And there are many bad ones). It's broken. It's not thread safe (yes there is strtok_r). It's needlessly hard to use. And it writes zeros to the input array. The latter means it's unfit for most use cases, including non-trivial tokenization where you want e.g. to split "a+1" into three tokens. If you program in C please just write those four obvious lines…
> it writes zeros to the input array [which] means it's unfit for most use cases[...]
Other issues with strtok() aside, this seems like a silly reason to discount a standard library function. If you don't want your input munged you can strdup() it. It's rare to find a C program that's so specialized that the performance hit of a strdup() would be unacceptable in a case where strtok() could otherwise have been used.
Re: String tokenization in C
#95strtok is one of the silliest parts of the standard library. (And there are many bad ones). It's broken. It's not thread safe (yes there is strtok_r). It's needlessly hard to use. And it writes zeros to the input array. The latter means it's unfit for most use cases, including non-trivial tokenization where you want e.g. to split "a+1" into three tokens. If you program in C please just write those four obvious lines…
> And it writes zeros to the input array.
If you pass strtok_r a const string it can and will bus fault in some systems. This happens when it tries to write a /0 to the input string. Being an old crusty firmware guy I'm not sailing on good cargo cult ship HMS Immutability, but generating side effects in your input data stream is terrible.
There is no way to back up/undo when using strtok_r. When your parsing involves a decision tree that kinda sucks.
Re: String tokenization in C
#96Earlier quoted context omitted.
You've not only hard-coded your tokenization rules inside your logic, but you've managed to make it break on anything non-ASCII. I suppose you consider isalpha() to have a weird API?
I look at it the other way: I've hard coded the reading and writing routines inside the tokenization logic. Being able to do that is exactly the point why it's so much simpler to avoid a silly API such as strcspn (or, god forbid, strtok). > non-ASCII yeah i know... Do you prefer strcspn("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVXYZ")? Do you think it's faster? If you're pedantic, you could lex (0x41 > I suppos…
> If you're pedantic, you could lex (0x41 'A' v.s. 0x41 makes no difference for portability. The thing that's unportable about that is that it assumes that the characters A..Z are continuous in your character encoding, which isn't portable C.
Although admittedly having to deal with EBCDIC these days is rare in anything except highly portable programs like C compilers or popular script interpreters.
This is why ctype.h functions exist. Just use them.
Re: String tokenization in C
#97Well, yes, using strtok works if the data happens to be structured in a certain simple way. Very often you want to do something more advanced though, and using regex for matching tokens is then necessary.
You don't really use regex in C. You just write a few simple loops. Look up the lexer of the programming language of your choice.
Lexer generators are pretty popular in C land. I mean, there's Yacc, obviously. And then there's more low-level stuff like re2c.
Re: String tokenization in C
#98Earlier quoted context omitted.
You don't really use regex in C. You just write a few simple loops. Look up the lexer of the programming language of your choice.
Lexer generators are pretty popular in C land. I mean, there's Yacc, obviously. And then there's more low-level stuff like re2c.
Yacc is a parser generator, not a scanner generator. You meant lex/flex.
Re: String tokenization in C
#99strtok is one of the silliest parts of the standard library. (And there are many bad ones). It's broken. It's not thread safe (yes there is strtok_r). It's needlessly hard to use. And it writes zeros to the input array. The latter means it's unfit for most use cases, including non-trivial tokenization where you want e.g. to split "a+1" into three tokens. If you program in C please just write those four obvious lines…
I use strtok_r from time to time, it does the job if you have a mutable input. Of course having to write zeroes is a bit cumbersome but it's one of the drawbacks of C-style strings. The plain truth is that string handling in C is a huge pain in the ass no matter how you look at it. Splitting, concatenating, regex-ing... All of that is a huge pain in C. If you need to write a high-performance parser then it might be w…
One of the big performance wins of the D programming language over C is that arrays are length terminated instead of 0 terminated, so you can "slice" strings to get substrings, rather than allocate/copy/zero (and then get the free in the right place!).
Re: String tokenization in C
#100strtok is one of the silliest parts of the standard library. (And there are many bad ones). It's broken. It's not thread safe (yes there is strtok_r). It's needlessly hard to use. And it writes zeros to the input array. The latter means it's unfit for most use cases, including non-trivial tokenization where you want e.g. to split "a+1" into three tokens. If you program in C please just write those four obvious lines…
strncat is another good example. It's a buffer overflow waiting to happen, and worse it's the 'safe' version of strcat. The trick is the parameter you pass is not the length of the destination buffer, but the remaining length of the destination buffer. Most people really want the behaviour of strlcat.
glibc's continued resistance to adding strlcat and strlcpy is a travesty. :-(