Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

String tokenization in C

onebyezero.blogspot.com

Post reply on HN
Print view View original

41–50 of 117 posts

Re: String tokenization in C

#41

Post by tails4e »

jstimpfle wrote: post #12

strtok is one of the silliest parts of the standard library. (And there are many bad ones). It's broken. It's not thread safe (yes there is strtok_r). It's needlessly hard to use. And it writes zeros to the input array. The latter means it's unfit for most use cases, including non-trivial tokenization where you want e.g. to split "a+1" into three tokens. If you program in C please just write those four obvious lines…

strncat is another good example. It's a buffer overflow waiting to happen, and worse it's the 'safe' version of strcat. The trick is the parameter you pass is not the length of the destination buffer, but the remaining length of the destination buffer. Most people really want the behaviour of strlcat.

Re: String tokenization in C

#42

Post by fredmorcos »

jstimpfle wrote: post #12

strtok is one of the silliest parts of the standard library. (And there are many bad ones). It's broken. It's not thread safe (yes there is strtok_r). It's needlessly hard to use. And it writes zeros to the input array. The latter means it's unfit for most use cases, including non-trivial tokenization where you want e.g. to split "a+1" into three tokens. If you program in C please just write those four obvious lines…

strtok is there for things like and similar to /etc/hosts and /etc/fstab

Re: String tokenization in C

#44

Post by kazinator »

The actions of strtok can easily be coded using strspn and strcspn.

https://groups.google.com/forum/message/raw?msg=comp.lang.c/... [2001]

https://groups.google.com/forum/message/raw?msg=comp.lang.c/... [2011 repost]

strspn(s, bag) calculates the length of the prefix of string s which consists only of the characters in string bag. strcspn(s, bag) calculates the length of the prefix of s consisting of characters not in bag.

The bag is like a one-character regex class; so that is to say strspn(s, "abcd") is like calculating the length of the token at the front of input s matching the regex [abcd]* , and in the case of strcspn, that becomes [^abcd]* .

Re: String tokenization in C

#45

Post by ygra »

fredmorcos wrote: post #42
jstimpfle wrote: post #12

strtok is one of the silliest parts of the standard library. (And there are many bad ones). It's broken. It's not thread safe (yes there is strtok_r). It's needlessly hard to use. And it writes zeros to the input array. The latter means it's unfit for most use cases, including non-trivial tokenization where you want e.g. to split "a+1" into three tokens. If you program in C please just write those four obvious lines…

strtok is there for things like and similar to /etc/hosts and /etc/fstab

Or have those file formats been designed around having to parse them in a C program?

Re: String tokenization in C

#46

Post by dahart »

jstimpfle wrote: post #18
jfries wrote: post #2

Well, yes, using strtok works if the data happens to be structured in a certain simple way. Very often you want to do something more advanced though, and using regex for matching tokens is then necessary.

You don't really use regex in C. You just write a few simple loops. Look up the lexer of the programming language of your choice.

> Yout don’t really use regex in C.

Speak for yourself. There’s a POSIX standard for regex that is more than 30 years old & a GNU implementation that comes with gcc. C++ has regex in the standard library.

Re: String tokenization in C

#47

Post by barrkel »

colejohnson66 wrote: post #36
jfries wrote: post #2

Well, yes, using strtok works if the data happens to be structured in a certain simple way. Very often you want to do something more advanced though, and using regex for matching tokens is then necessary.

I wrote a tokenizer for a language I’m creating, and all I needed was read character and peek character from an iterator

Guess what? You wrote a state machine (DFA most likely) in code, where the program counter represents the state. There is in all probability (if your language is sane), a 1:1 correspondence between the code you wrote and the regular grammar of your tokens. You implemented a regular language matcher, i.e. a regex matcher, but in code rather than via a regular expression language interpreter or compiler.

Typical pattern:

    start = p;
    while (isspace(*p) && p
Corresponds to:

    IDENT ::= [a-z][a-z0-9]* ;
    NUMBER ::= [0-9][0-9]* ;
    SPACE ::= [ ]* ;

    TOKEN ::= SPACE (IDENT | NUMBER) ;
Inline those nonterminals, and guess what - regular expression!

Re: String tokenization in C

#48

Post by jstimpfle »

dahart wrote: post #46
jstimpfle wrote: post #18

Earlier quoted context omitted.

You don't really use regex in C. You just write a few simple loops. Look up the lexer of the programming language of your choice.

> Yout don’t really use regex in C. Speak for yourself. There’s a POSIX standard for regex that is more than 30 years old & a GNU implementation that comes with gcc. C++ has regex in the standard library.

"You" used like "one", or "in common practice it isn't really used". It's just not in C's spirit to use canned libraries. Such use cases have long been transferred to Python and other languages. Sure, there is a regex API in the C standard library. I'd bet not even grep uses it.

The one use-case I'm envisioning is quickly exposing POSIX conformant regexen to the command-line.

Re: String tokenization in C

#49

Post by barrkel »

jstimpfle wrote: post #18
jfries wrote: post #2

Well, yes, using strtok works if the data happens to be structured in a certain simple way. Very often you want to do something more advanced though, and using regex for matching tokens is then necessary.

You don't really use regex in C. You just write a few simple loops. Look up the lexer of the programming language of your choice.

Most lexers are state machines, either explicit with tables (like you get from lex) or implicit with program counter (with loops and switches). Those state machines implement matchers for regular languages; they're effectively hand-coded implementations of regular expression matching.

Regular expressions don't show up outside the spec, sure; but if you're writing the code (for implicit state machine), you need to know exactly where you are in the regular language that defines the tokens to write good code. Writing a regex matcher in code like this is like writing code in assembly - mentally, you're mapping to a different set of concepts all the time.

Re: String tokenization in C

#50

Post by jstimpfle »

barrkel wrote: post #49
jstimpfle wrote: post #18

Earlier quoted context omitted.

You don't really use regex in C. You just write a few simple loops. Look up the lexer of the programming language of your choice.

Most lexers are state machines, either explicit with tables (like you get from lex) or implicit with program counter (with loops and switches). Those state machines implement matchers for regular languages; they're effectively hand-coded implementations of regular expression matching. Regular expressions don't show up outside the spec, sure; but if you're writing the code (for implicit state machine), you need to kno…

Yes. I don't think anyone is disagreeing here.

If you're implying that we should then use a regex implementation instead: Coding up a lexer (for a mainstream programming language) using simple counter increments and such is not a lot of work. It has the advantage that it results in faster code (unless you're going for real heavy machinery) and that you can easily code up additional transformations. For example, how would you parse string literals (including escape sequences) with a regex?

Post reply on HN