Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

String tokenization in C

onebyezero.blogspot.com

Post reply on HN
Print view View original

81–90 of 117 posts

Re: String tokenization in C

#81

Post by jcranberry »

barrkel wrote: post #47
colejohnson66 wrote: post #36

Earlier quoted context omitted.

I wrote a tokenizer for a language I’m creating, and all I needed was read character and peek character from an iterator

Guess what? You wrote a state machine (DFA most likely) in code, where the program counter represents the state. There is in all probability (if your language is sane), a 1:1 correspondence between the code you wrote and the regular grammar of your tokens. You implemented a regular language matcher, i.e. a regex matcher, but in code rather than via a regular expression language interpreter or compiler. Typical patter…

You could say the same about strtok--it would be trivial to create a regular grammar expressing whatever your code is lexing. However, by looking at it from this perspective it removes the distinction not just between a hand rolled lexer and a regex matching library, but also between those and using strtok. There would be no point in having this conversation.

Re: String tokenization in C

#82

Post by bqe »

enriquto wrote: post #34
lixtra wrote: post #17

I have an obsession with unsafe example code: strcpy(str,"abc,def,ghi"); token = strtok(str,","); printf("%s \n",token); Even if the author knows how many tokens are returned I would prefer a check for NULL here since a good fraction might not read further than this bad example.

> I have an obsession with unsafe example code: It is perfectly OK for example code to be unsafe. You do not wear a parachute when you learn to fly using a simulator. You realize that things will become more serious and complicated in the future, but you have to start with something simple and unsafe, no big deal. Otherwise you will never see the consequences of unsafe code in simple cases.

I think you underestimate how many people blindly copy examples without understanding them. Safe example code results in more correct programs.

Re: String tokenization in C

#83

Post by tails4e »

jstimpfle wrote: post #53
tails4e wrote: post #41

Earlier quoted context omitted.

strncat is another good example. It's a buffer overflow waiting to happen, and worse it's the 'safe' version of strcat. The trick is the parameter you pass is not the length of the destination buffer, but the remaining length of the destination buffer. Most people really want the behaviour of strlcat.

I'd say just use memcpy.

Sure, however the problem I'm highlighting is most programmers are not aware this std library function is not like the other str[n] functions. There are plenty of solutions, but it was an objectively poor decision to define strncat the way it was.

Re: String tokenization in C

#84

Post by saagarjha »

kazinator wrote: post #44

The actions of strtok can easily be coded using strspn and strcspn . https://groups.google.com/forum/message/raw?msg=comp.lang.c/... [2001] https://groups.google.com/forum/message/raw?msg=comp.lang.c/... [2011 repost] strspn(s, bag) calculates the length of the prefix of string s which consists only of the characters in string bag . strcspn(s, bag) calculates the length of the prefix of s consisting of characters not…

And it’s nicer, since you can pass in a const char * and use it in concurrent code.

Re: String tokenization in C

#85

Post by saagarjha »

paavoova wrote: post #69
simias wrote: post #64

Earlier quoted context omitted.

I use strtok_r from time to time, it does the job if you have a mutable input. Of course having to write zeroes is a bit cumbersome but it's one of the drawbacks of C-style strings. The plain truth is that string handling in C is a huge pain in the ass no matter how you look at it. Splitting, concatenating, regex-ing... All of that is a huge pain in C. If you need to write a high-performance parser then it might be w…

> but you'll be incompatible with any code dealing with native C-strings Not entirely, see https://github.com/antirez/sds Basically, you have a header storing length, etc, but still null terminate, so library functions like strlen are none the wiser.

You lose the ability to pass create zero-cost slices, though. C++11 actually implements strings like this, but iterators allow you to pass in parts of the string as necessary.

Re: String tokenization in C

#86

Post by kahirsch »

jstimpfle wrote: post #74
kahirsch wrote: post #68

Earlier quoted context omitted.

GNU grep uses the regex library in Gnulib. It can also use the Perl-compatible pcre library. I am finding it hard to believe that you think it's "just not in C's spirit to use canned libraries."

I was saying "canned", not "boxed" or "containerized". It's a lot easier to write the little things yourself. (And there are good benefits to be had from writing specialized code yourself, instead of relying on big fat generalized tankers.)

I don't know what distinction you're trying to make about "canned" vs. other things.

There are advantages and disadvantages of using libraries in any language. There are some aspects of C —e.g. the lack of garbage collection— that make it harder to reuse code compared to some languages. But it is definitely not against "C's spirit".

Re: String tokenization in C

#87

Post by enriquto »

bqe wrote: post #82
enriquto wrote: post #34

Earlier quoted context omitted.

> I have an obsession with unsafe example code: It is perfectly OK for example code to be unsafe. You do not wear a parachute when you learn to fly using a simulator. You realize that things will become more serious and complicated in the future, but you have to start with something simple and unsafe, no big deal. Otherwise you will never see the consequences of unsafe code in simple cases.

I think you underestimate how many people blindly copy examples without understanding them. Safe example code results in more correct programs.

> I think you underestimate how many people blindly copy examples without understanding them. Safe example code results in more correct programs.

Even if this is true, the reasoning here is disturbingly short-sighted. Copying code that you do not understand is unacceptable behavior, and I'd say the sooner it blows up in your face, the better. The goal of code examples is to illustrate how things work in a simplified way, and code without error checks is often easier to understand at first. Imagine a hello world with all the possible error checks. That would be incomprehensible.

Re: String tokenization in C

#88

Post by megous »

Other approach from library calls and flex is re2c. It preprocesses the source code and inlines regular expression parsing where you needed. It's very powerful in combination with goto.

Re: String tokenization in C

#89

Post by GordonS »

johnisgood wrote: post #26
jfries wrote: post #2

Well, yes, using strtok works if the data happens to be structured in a certain simple way. Very often you want to do something more advanced though, and using regex for matching tokens is then necessary.

I do not believe that using regex is necessary. I have parsed a lot of code in my life and regex was not a necessity.

Agreed. Regex can make parsing code much more succinct and easiet to grok (although usually at a small performance cost). So not "necessary", but it can be really useful.
Post reply on HN