How about just using a properly suited language por string manipulation?
String tokenization in C
71–80 of 117 posts
Re: String tokenization in C
#72Earlier quoted context omitted.
I use strtok_r from time to time, it does the job if you have a mutable input. Of course having to write zeroes is a bit cumbersome but it's one of the drawbacks of C-style strings. The plain truth is that string handling in C is a huge pain in the ass no matter how you look at it. Splitting, concatenating, regex-ing... All of that is a huge pain in C. If you need to write a high-performance parser then it might be w…
> but you'll be incompatible with any code dealing with native C-strings Not entirely, see https://github.com/antirez/sds Basically, you have a header storing length, etc, but still null terminate, so library functions like strlen are none the wiser.
Nim (the programming language) implements strings this way, AFAIK, with the same reason: to be easily compatible with the C ecosystem.
Re: String tokenization in C
#73Earlier quoted context omitted.
any attempt has been made to build a nicer foundational C/OS library since ?
There are of course tons of code on the net, but there is no need to standardize another grab-bag of bad API calls. If you need a batteries-included library you use python or similar. If you write in C, you care so much that you largely avoid the standard library and design your code from the ground up.
You should. However there are lots of organizations around that use C because that’s what’s used in their domain (E.g. embedded) AND use the standard library, because they don’t know how to do better.
The results are often worse than if they would have used a higher level language right from the start.
Re: String tokenization in C
#74Earlier quoted context omitted.
"You" used like "one", or "in common practice it isn't really used". It's just not in C's spirit to use canned libraries. Such use cases have long been transferred to Python and other languages. Sure, there is a regex API in the C standard library. I'd bet not even grep uses it. The one use-case I'm envisioning is quickly exposing POSIX conformant regexen to the command-line.
GNU grep uses the regex library in Gnulib. It can also use the Perl-compatible pcre library. I am finding it hard to believe that you think it's "just not in C's spirit to use canned libraries."
I was saying "canned", not "boxed" or "containerized". It's a lot easier to write the little things yourself. (And there are good benefits to be had from writing specialized code yourself, instead of relying on big fat generalized tankers.)
Re: String tokenization in C
#75Earlier quoted context omitted.
I wrote a tokenizer for a language I’m creating, and all I needed was read character and peek character from an iterator
Guess what? You wrote a state machine (DFA most likely) in code, where the program counter represents the state. There is in all probability (if your language is sane), a 1:1 correspondence between the code you wrote and the regular grammar of your tokens. You implemented a regular language matcher, i.e. a regex matcher, but in code rather than via a regular expression language interpreter or compiler. Typical patter…
That works until you need your own implementation of ++p;. Now wiring up that implementation to a generic library is already more work than just doing it all yourself. Not even considering the integration costs of the library into the sources and the build.
And you cannot really do transformation instead of only matching with RE. Those are needed already in simple cases like string and number literals. Now your code will look more like
%{
#include "y.tab.h"
int num_lines = 1;
int comment_mode=0;
int stack =0;
%}
digit ([0-9])
integer ({digit}+)
float_num ({digit}+\.{digit}+)
%%
{integer} { //deal with integer
printf("#%d: NUM:",num_lines); ECHO;printf("\n");
yylval.Integer = atoi(yytext);
return INT;
}
{float_num} {// deal with float
printf("#%d: NUM:",num_lines);ECHO;printf("\n");
yylval.Float = atof(yytext);
return FLOAT;
}
\n { ++num_lines; }
. if(strcmp(yytext," "))ECHO;
%%
int yywrap() {
return 1;
}Re: String tokenization in C
#76Problem is that your token string is going to be quite large. Is there a built-in solution for when tokens are just single chars?
Re: String tokenization in C
#77Earlier quoted context omitted.
any attempt has been made to build a nicer foundational C/OS library since ?
BSD has some minor improvements to libc. There's also glib. Generally though, C is just a terrible language to do anything other than write extremely low level routines in. You should probably just never use strtok() and go straight to something like Ragel or re2c for building high performance tokenizers... then call those from a higher level language.
Hey! I take exception to that. I've been writing shellcode all week. C is uncomfortably high level for writing low level routines!
Re: String tokenization in C
#78Earlier quoted context omitted.
I use strtok_r from time to time, it does the job if you have a mutable input. Of course having to write zeroes is a bit cumbersome but it's one of the drawbacks of C-style strings. The plain truth is that string handling in C is a huge pain in the ass no matter how you look at it. Splitting, concatenating, regex-ing... All of that is a huge pain in C. If you need to write a high-performance parser then it might be w…
> The plain truth is that string handling in C is a huge pain in the ass no matter how you look at it. It is. And it’s not even only Cs fault. 80% of it is bad API Design. Strings could be accepted as a struct consisting of a pointer and a length, aka string_view. And there could be some manipulation functions around it. That would make those APIs a lot more flexible (one no longer needs to care whether things are nu…
What happens if you truncate your string? You lose information about buffer size. So now you need to store two sizes for such strings to be useful, string size and buffer size, which is 16 bytes for size_t on 64-bit systems. On top of that, strings are no longer arrays. So either the language would have to incorporate first class support for these strings, or you'll have people extracting the string pointer from the struct to perform indexing operations on themselves.
Re: String tokenization in C
#79Earlier quoted context omitted.
Most lexers are state machines, either explicit with tables (like you get from lex) or implicit with program counter (with loops and switches). Those state machines implement matchers for regular languages; they're effectively hand-coded implementations of regular expression matching. Regular expressions don't show up outside the spec, sure; but if you're writing the code (for implicit state machine), you need to kno…
Yes. I don't think anyone is disagreeing here. If you're implying that we should then use a regex implementation instead: Coding up a lexer (for a mainstream programming language) using simple counter increments and such is not a lot of work. It has the advantage that it results in faster code (unless you're going for real heavy machinery) and that you can easily code up additional transformations. For example, how w…
String literals are easy without escaped quotes. With escaped quotes its annoying and non regex is much cleaner.
Re: String tokenization in C
#80Earlier quoted context omitted.
> The plain truth is that string handling in C is a huge pain in the ass no matter how you look at it. It is. And it’s not even only Cs fault. 80% of it is bad API Design. Strings could be accepted as a struct consisting of a pointer and a length, aka string_view. And there could be some manipulation functions around it. That would make those APIs a lot more flexible (one no longer needs to care whether things are nu…
What happens if you truncate your string? You lose information about buffer size. So now you need to store two sizes for such strings to be useful, string size and buffer size, which is 16 bytes for size_t on 64-bit systems. On top of that, strings are no longer arrays. So either the language would have to incorporate first class support for these strings, or you'll have people extracting the string pointer from the…
The approach obviously only works for read access and not for mutation. However those are the most often required operations. For owned strings and mutations different APIs are required. C++ string_view vs string, and Rusts str slices vs owned Strings work like this.