Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

String tokenization in C

onebyezero.blogspot.com

Post reply on HN
Print view View original

21–30 of 117 posts

Re: String tokenization in C

#21

Post by nly »

agumonkey wrote: post #19
nly wrote: post #14

Earlier quoted context omitted.

Much of libc is terrible from an API design perspective, even given the limitations of C as a language. libc has somehow managed to hit the sweet spot and have APIs that are both inconvenient to use properly, and perform poorly.

any attempt has been made to build a nicer foundational C/OS library since ?

BSD has some minor improvements to libc. There's also glib.

Generally though, C is just a terrible language to do anything other than write extremely low level routines in. You should probably just never use strtok() and go straight to something like Ragel or re2c for building high performance tokenizers... then call those from a higher level language.

Re: String tokenization in C

#22

Post by graycat »

A lot of experience shows that the string tokenization in Open Object Rexx is darned useful. E.g., for many years, IBM's internal computing was from about 3600 mainframe computers around the world running VM/CMS with a lot of service machines written in Rexx. Rexx is no toy but a powerful, polished, scripting language and really good at handling strings.

A little example of some Rexx code with some string parsing is in

https://news.ycombinator.com/item?id=18648999

Re: String tokenization in C

#23

Post by zakk »

tinus_hn wrote: post #4
pasokan wrote: post #3

It used to be that gcc will warn against strtok and recommend strsep instead. Do not know what the status is today

Strtok is not thread safe and can’t be made thread safe without changing the API. You should not use it.

That’s not a good reason not to use it.

A function can be not thread-safe and still safe to use in single-threaded programs.

The point is that strtok is not a good choice even for single-threaded code.

Re: String tokenization in C

#24

Post by jstimpfle »

agumonkey wrote: post #19
nly wrote: post #14

Earlier quoted context omitted.

Much of libc is terrible from an API design perspective, even given the limitations of C as a language. libc has somehow managed to hit the sweet spot and have APIs that are both inconvenient to use properly, and perform poorly.

any attempt has been made to build a nicer foundational C/OS library since ?

There are of course tons of code on the net, but there is no need to standardize another grab-bag of bad API calls. If you need a batteries-included library you use python or similar. If you write in C, you care so much that you largely avoid the standard library and design your code from the ground up.

Re: String tokenization in C

#25

Post by yason »

jstimpfle wrote: post #12

strtok is one of the silliest parts of the standard library. (And there are many bad ones). It's broken. It's not thread safe (yes there is strtok_r). It's needlessly hard to use. And it writes zeros to the input array. The latter means it's unfit for most use cases, including non-trivial tokenization where you want e.g. to split "a+1" into three tokens. If you program in C please just write those four obvious lines…

If you program in C please just write those four obvious lines yourself.

Those are not necessarily obvious lines, there are several pitfalls to avoid, and for that reason strtok() is much longer than four lines. When it comes to the standard library functions strtok() has well-defined behaviour that is easy to reason with and near-magically approaches the string-splitting convenience close to scripting languages.

In contrast, an example of truly sickening part of stdlib is converting strings to number. The atoi()/atol() family doesn't check for errors at all so you want to use strtol(). But the way error checking works in strtol() is so complex that the man page has a specific example of how to do it correctly. All sane programmers quickly write a clean wrapper around strtol() to encode the complexity once. Now, strtok() is nothing like that.

In its simplicity, strtok() is quite versatile. A few strtok() calls can easily parse lines like:

    keyword=value1, value2, value3
that you might find in configuration files. And I mean truly in just a few lines which you might expect in Python but with C string handling? No.

Re: String tokenization in C

#26

Post by johnisgood »

jfries wrote: post #2

Well, yes, using strtok works if the data happens to be structured in a certain simple way. Very often you want to do something more advanced though, and using regex for matching tokens is then necessary.

I do not believe that using regex is necessary. I have parsed a lot of code in my life and regex was not a necessity.

Re: String tokenization in C

#27

Post by johnisgood »

zakk wrote: post #23
tinus_hn wrote: post #4

Earlier quoted context omitted.

Strtok is not thread safe and can’t be made thread safe without changing the API. You should not use it.

That’s not a good reason not to use it. A function can be not thread-safe and still safe to use in single-threaded programs. The point is that strtok is not a good choice even for single-threaded code.

> The point is that strtok is not a good choice even for single-threaded code.

Why isn't it a good choice exactly? Could you sum it up?

Re: String tokenization in C

#28

Post by jstimpfle »

yason wrote: post #25
jstimpfle wrote: post #12

strtok is one of the silliest parts of the standard library. (And there are many bad ones). It's broken. It's not thread safe (yes there is strtok_r). It's needlessly hard to use. And it writes zeros to the input array. The latter means it's unfit for most use cases, including non-trivial tokenization where you want e.g. to split "a+1" into three tokens. If you program in C please just write those four obvious lines…

If you program in C please just write those four obvious lines yourself. Those are not necessarily obvious lines, there are several pitfalls to avoid, and for that reason strtok() is much longer than four lines. When it comes to the standard library functions strtok() has well-defined behaviour that is easy to reason with and near-magically approaches the string-splitting convenience close to scripting languages. In…

Here is the musl implementation.

> https://github.com/esmil/musl/blob/master/src/string/strtok....

It's a bit longer than 4 lines because strtok does things you should not want. If you insist on parsing that configuration line with strtok, go ahead and write that brittle code. It breaks as soon as you want empty strings (try "keyword=value1, , value3" with strtok) or escape sequences or other transformations, or as soon as you want to do something as basic as parsing from a stream instead of a string that is completely in memory.

So to clarify, of course you are never done with parsing in 4 lines. But even if it wasn't as braindead to overwrite the input string, the functionality strtok provides would not be worth more than 4 lines.

Re: String tokenization in C

#29

Post by [deleted] »

Sharlin wrote: post #15
heinrichhartman wrote: post #13

Earlier quoted context omitted.

Well, strtok could use thread local variables to store intermediate state, to make it threadsafe while maintaining the same API. Not saying this is a good idea, but technically it would work, no?

Yes, as long as you can guarantee that there’s only one tokenization going on per thread at a time.

[deleted]

Re: String tokenization in C

#30

Post by nly »

yason wrote: post #25
jstimpfle wrote: post #12

strtok is one of the silliest parts of the standard library. (And there are many bad ones). It's broken. It's not thread safe (yes there is strtok_r). It's needlessly hard to use. And it writes zeros to the input array. The latter means it's unfit for most use cases, including non-trivial tokenization where you want e.g. to split "a+1" into three tokens. If you program in C please just write those four obvious lines…

If you program in C please just write those four obvious lines yourself. Those are not necessarily obvious lines, there are several pitfalls to avoid, and for that reason strtok() is much longer than four lines. When it comes to the standard library functions strtok() has well-defined behaviour that is easy to reason with and near-magically approaches the string-splitting convenience close to scripting languages. In…

Unless you're using a pre-specified configuration file format (e.g. TOML), then parsing configuration files requires a general parsing library. This is a non-trivial task requiring a real parser operating over a well-specified grammar. A tokenization pipeline just won't cut it.

I worked on a project a few years ago that read its custom-format config file in line by line, chopped everything off each line following the first '#' character (to support comments), and then trimmed the whitespace. This sounds like a reasonable and elegant approach until you consider that now none of your user controlled fields (via a GUI in our case) can contain the '#' character. This effected customers, but nobody ever fixed it.

With the tools and languages out there now, there's just no excuse for this crap.

Post reply on HN