String tokenization in C • Hacker News Board

Re: String tokenization in C

#11

Post by seba_dos1 » Sat, Dec 15, 2018, 12:30 PM UTC

morbusfonticuli wrote: post #5

tinus_hn wrote: post #4

Earlier quoted context omitted.

Strtok is not thread safe and can’t be made thread safe without changing the API. You should not use it.

> Strtok is not thread safe and can’t be made thread safe without changing the API. You should not use it. Well, there is already a thread-safe variant [0]: > The strtok() function uses a static buffer while parsing, so it's not thread safe. Use strtok_r() if this matters to you. [0] https://linux.die.net/man/3/strtok_r

...with different API :P

Re: String tokenization in C

#12

Post by jstimpfle » Sat, Dec 15, 2018, 12:31 PM UTC

strtok is one of the silliest parts of the standard library. (And there are many bad ones). It's broken. It's not thread safe (yes there is strtok_r). It's needlessly hard to use. And it writes zeros to the input array. The latter means it's unfit for most use cases, including non-trivial tokenization where you want e.g. to split "a+1" into three tokens.

If you program in C please just write those four obvious lines yourself.

Re: String tokenization in C

#13

Post by heinrichhartman » Sat, Dec 15, 2018, 12:42 PM UTC

tinus_hn wrote: post #4

pasokan wrote: post #3

It used to be that gcc will warn against strtok and recommend strsep instead. Do not know what the status is today

Strtok is not thread safe and can’t be made thread safe without changing the API. You should not use it.

Well, strtok could use thread local variables to store intermediate state, to make it threadsafe while maintaining the same API. Not saying this is a good idea, but technically it would work, no?

Re: String tokenization in C

#14

Post by nly » Sat, Dec 15, 2018, 12:44 PM UTC

jstimpfle wrote: post #12

strtok is one of the silliest parts of the standard library. (And there are many bad ones). It's broken. It's not thread safe (yes there is strtok_r). It's needlessly hard to use. And it writes zeros to the input array. The latter means it's unfit for most use cases, including non-trivial tokenization where you want e.g. to split "a+1" into three tokens. If you program in C please just write those four obvious lines…

Much of libc is terrible from an API design perspective, even given the limitations of C as a language.

libc has somehow managed to hit the sweet spot and have APIs that are both inconvenient to use properly, and perform poorly.

Re: String tokenization in C

#15

Post by Sharlin » Sat, Dec 15, 2018, 12:53 PM UTC

heinrichhartman wrote: post #13

tinus_hn wrote: post #4

Earlier quoted context omitted.

Strtok is not thread safe and can’t be made thread safe without changing the API. You should not use it.

Well, strtok could use thread local variables to store intermediate state, to make it threadsafe while maintaining the same API. Not saying this is a good idea, but technically it would work, no?

Yes, as long as you can guarantee that there’s only one tokenization going on per thread at a time.

Re: String tokenization in C

#16

Post by tptacek » Sat, Dec 15, 2018, 12:56 PM UTC

caf wrote: post #6

Note though that strsep() is not as portable, because it is an extension to standard C.

It's a tiny function, written in ANSI C, so if you're really concerned about this, just include it with your program. It's an extension to the standard C library, not to C itself.

Re: String tokenization in C

#17

Post by lixtra » Sat, Dec 15, 2018, 12:56 PM UTC

I have an obsession with unsafe example code:

  strcpy(str,"abc,def,ghi");
  token = strtok(str,",");
  printf("%s \n",token);

Even if the author knows how many tokens are returned I would prefer a check for NULL here since a good fraction might not read further than this bad example.

Re: String tokenization in C

#18

Post by jstimpfle » Sat, Dec 15, 2018, 12:56 PM UTC

jfries wrote: post #2

Well, yes, using strtok works if the data happens to be structured in a certain simple way. Very often you want to do something more advanced though, and using regex for matching tokens is then necessary.

You don't really use regex in C. You just write a few simple loops. Look up the lexer of the programming language of your choice.

Re: String tokenization in C

#19

Post by agumonkey » Sat, Dec 15, 2018, 1:08 PM UTC

nly wrote: post #14

jstimpfle wrote: post #12

strtok is one of the silliest parts of the standard library. (And there are many bad ones). It's broken. It's not thread safe (yes there is strtok_r). It's needlessly hard to use. And it writes zeros to the input array. The latter means it's unfit for most use cases, including non-trivial tokenization where you want e.g. to split "a+1" into three tokens. If you program in C please just write those four obvious lines…

Much of libc is terrible from an API design perspective, even given the limitations of C as a language. libc has somehow managed to hit the sweet spot and have APIs that are both inconvenient to use properly, and perform poorly.

any attempt has been made to build a nicer foundational C/OS library since ?

Re: String tokenization in C

#20

Post by johannes1234321 » Sat, Dec 15, 2018, 1:08 PM UTC

teddyh wrote: post #10

beefhash wrote: post #9

Earlier quoted context omitted.

In fact, it's not even in POSIX .

To quote the GNU C library manual: “ This function was introduced in 4.3BSD and therefore is widely available. ”¹ 1. https://www.gnu.org/software/libc/manual/html_node/Finding-T...

Widely except Windows and different embedded platforms etc.