Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

String tokenization in C

onebyezero.blogspot.com

Post reply on HN
Print view View original

31–40 of 117 posts

Re: String tokenization in C

#31

Post by nly »

heinrichhartman wrote: post #13
tinus_hn wrote: post #4

Earlier quoted context omitted.

Strtok is not thread safe and can’t be made thread safe without changing the API. You should not use it.

Well, strtok could use thread local variables to store intermediate state, to make it threadsafe while maintaining the same API. Not saying this is a good idea, but technically it would work, no?

You could, but that would change the behaviour of existing programs. It might well be that there are well-defined programs out there that use strtok across separate, but properly synchronized, threads.

This is why it's crucial to get APIs right first time.

Re: String tokenization in C

#32

Post by enriquto »

seba_dos1 wrote: post #11
morbusfonticuli wrote: post #5

Earlier quoted context omitted.

> Strtok is not thread safe and can’t be made thread safe without changing the API. You should not use it. Well, there is already a thread-safe variant [0]: > The strtok() function uses a static buffer while parsing, so it's not thread safe. Use strtok_r() if this matters to you. [0] https://linux.die.net/man/3/strtok_r

...with different API :P

of course! the original API cannot be made re-entrant

Re: String tokenization in C

#33

Post by enriquto »

johnisgood wrote: post #27
zakk wrote: post #23

Earlier quoted context omitted.

That’s not a good reason not to use it. A function can be not thread-safe and still safe to use in single-threaded programs. The point is that strtok is not a good choice even for single-threaded code.

> The point is that strtok is not a good choice even for single-threaded code. Why isn't it a good choice exactly? Could you sum it up?

Maybe he means that the function is not re-entrant. You cannot run a loop that tokenizes a string, and somewhere inside, you call a function that uses strtok itself. This can happen inadvertently.

Re: String tokenization in C

#34

Post by enriquto »

lixtra wrote: post #17

I have an obsession with unsafe example code: strcpy(str,"abc,def,ghi"); token = strtok(str,","); printf("%s \n",token); Even if the author knows how many tokens are returned I would prefer a check for NULL here since a good fraction might not read further than this bad example.

> I have an obsession with unsafe example code:

It is perfectly OK for example code to be unsafe. You do not wear a parachute when you learn to fly using a simulator. You realize that things will become more serious and complicated in the future, but you have to start with something simple and unsafe, no big deal. Otherwise you will never see the consequences of unsafe code in simple cases.

Re: String tokenization in C

#35

Post by fanf2 »

jstimpfle wrote: post #12

strtok is one of the silliest parts of the standard library. (And there are many bad ones). It's broken. It's not thread safe (yes there is strtok_r). It's needlessly hard to use. And it writes zeros to the input array. The latter means it's unfit for most use cases, including non-trivial tokenization where you want e.g. to split "a+1" into three tokens. If you program in C please just write those four obvious lines…

Use strcspn() instead

Re: String tokenization in C

#36

Post by colejohnson66 »

jfries wrote: post #2

Well, yes, using strtok works if the data happens to be structured in a certain simple way. Very often you want to do something more advanced though, and using regex for matching tokens is then necessary.

I wrote a tokenizer for a language I’m creating, and all I needed was read character and peek character from an iterator

Re: String tokenization in C

#37

Post by jstimpfle »

fanf2 wrote: post #35
jstimpfle wrote: post #12

strtok is one of the silliest parts of the standard library. (And there are many bad ones). It's broken. It's not thread safe (yes there is strtok_r). It's needlessly hard to use. And it writes zeros to the input array. The latter means it's unfit for most use cases, including non-trivial tokenization where you want e.g. to split "a+1" into three tokens. If you program in C please just write those four obvious lines…

Use strcspn() instead

    Token tok;
    start_token(&tok);
    for (;;) {
        int c = look_next_char();
        if (('A'
Done. There's no point in going through a weird API.

Re: String tokenization in C

#38

Post by stochastic_monk »

I recommend ksplit/ksplit_core from Heng Li’s excellent klib kstring.{h,c}[0]. It modifies the string in-place, adding null terminators, and provides a list of offsets into the string. This gives you the flexibility of accessing tokens by index without paying costs of copying or memory allocation.

[0] https://github.com/attractivechaos/klib

Re: String tokenization in C

#39

Post by beefhash »

tptacek wrote: post #16
caf wrote: post #6

Note though that strsep() is not as portable, because it is an extension to standard C.

It's a tiny function, written in ANSI C, so if you're really concerned about this, just include it with your program. It's an extension to the standard C library , not to C itself.

Except then you have the issue about compilers complaining about double-declarations of the function, meaning you'll either have a lot of warning spam on every #include or now hard require some kind of header defines for HAVE_STRSEP. Once you go that way, there's no going back and it's only gonna become more and more.
Post reply on HN