Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Simple Dynamic Strings library for C, compatible with null-terminated strings

github.com

Post reply on HN
Print view View original

61–70 of 85 posts

Re: Simple Dynamic Strings library for C, compatible with null-terminated strings

#61

Post by m463 »

poiuyt098 wrote: post #54
antirez wrote: post #24

Hi, author here. May make sense to make SDS in perspective given a few comments I'm reading here. 1. Yep, more than "strings" SDS may be consider a library for dynamic buffers, especially from people coming from C++ or higher level languages. However I think that for C, it makes sense to provide a very low level thing like that. 2. In practice, if you see how SDS is used (extensively) inside Redis, it normally models…

> show how much you can easily improve C, with little work, and how many unsafe things in C are about lack of abstractions C has always badly needed built-in strings (and arrays with size info, generally). To save a byte, C designers committed The Most Expensive One-byte Mistake https://queue.acm.org/detail.cfm?id=2010365

A shortcoming (or outright failure) of C is the lack of an extensive "batteries included" library.

Maybe not C + Knuth, but some way of portably advancing the language over time.

Re: Simple Dynamic Strings library for C, compatible with null-terminated strings

#62

Post by jhallenworld »

electrograv wrote: post #38
antirez wrote: post #35

Earlier quoted context omitted.

One of the main points of the library is the ability to pass SDSs where char* is expected without doing anything. So this is a "feature" in the author's spirit.

As another reply has suggested, you can design the wrapper struct so you can simply write: legacy_fn(my_text.s); Versus the current: legacy_fn(my_text); I don’t think saving two characters per legacy function call is even remotely worth the loss of static type safety (which risks serious memory corruption and/or security holes, which are entirely preventable at compile-time in this way). In fact, I even find the expl…

A main problem with C++ is that it uses .c_str() instead of something nice like .s for this.

Re: Simple Dynamic Strings library for C, compatible with null-terminated strings

#63

Post by ndesaulniers »

m463 wrote: post #61
poiuyt098 wrote: post #54

Earlier quoted context omitted.

> show how much you can easily improve C, with little work, and how many unsafe things in C are about lack of abstractions C has always badly needed built-in strings (and arrays with size info, generally). To save a byte, C designers committed The Most Expensive One-byte Mistake https://queue.acm.org/detail.cfm?id=2010365

A shortcoming (or outright failure) of C is the lack of an extensive "batteries included" library. Maybe not C + Knuth, but some way of portably advancing the language over time.

Why? So we could be stuck with horrible interfaces forever (most of string.h comes to mind)? No thanks.

Re: Simple Dynamic Strings library for C, compatible with null-terminated strings

#64

Post by microcolonel »

antirez wrote: post #24

Hi, author here. May make sense to make SDS in perspective given a few comments I'm reading here. 1. Yep, more than "strings" SDS may be consider a library for dynamic buffers, especially from people coming from C++ or higher level languages. However I think that for C, it makes sense to provide a very low level thing like that. 2. In practice, if you see how SDS is used (extensively) inside Redis, it normally models…

> Probably an UTF-32 layer could be implemented on top of that

UTF-32 is, in all but extreme niche cases, a supremely wasteful and inefficient way to store and process strings, even in light of the cost of variable-length codepoints. The longest UTF-8 sequence is now four bytes anyway, so UTF-8 is never ever less compact than UTF-32. The only conceivable practical use of UTF-32 is a case where the upper 11 bits are used to tag characters with metadata, but in that case you'll have to mask those to display them anyway.

In practice, if your application must be aware of codepoints, it most likely should be aware of extended grapheme clusters as well, and those have an arbitrary number of codepoints in them.

UTF-16 is closer to being useful in some locales, but the marginal benefit (one byte less per BMP CJK codepoint) is, in my view, outweighed by the incompatibility with ASCII. Even a small number of ASCII strings in a given transmission will wipe out any advantage UTF-16 could have in CJK locales, and most transmissions these days have enough ASCII to do so. For example, today's Japanese Wikipedia main page is 102,620 bytes in UTF-8, and 182,480 bytes in UTF-16, despite most of the visible graphemes being CJK characters (which require three bytes in UTF-8). The Korean Wikipedia main page has larger text, and less of it, so the difference is even more stark (81,072 bytes in UTF-8, and 146,630 bytes in UTF-16). Even after aggressive gzip compression (with zopfli), text-heavy UTF-16 Japanese and Korean HTML documents are about 10% larger than their UTF-8 equivalent; and this is before you've transmitted any CSS or JavaScript.

Re: Simple Dynamic Strings library for C, compatible with null-terminated strings

#65

Post by cmrdporcupine »

tsegratis wrote: post #6
azhenley wrote: post #2

I rarely work with C but when I do I am always reminded at how painful it is to use strings. Is this a complete drop-in replacement?? If so it could occasionally make my life much easier!

This alternative (by me) covers char , int , etc with a compiletime inlined generic api much the same way SDS does strings https://tse.gratis/aArray/ aStr("string"); then aMap aFold aConcat, etc, but aAppend(&array,'\0') needs to be done manually if passing to standard str functions. This is so long* or whatever arrays can have the same interface

Nice. I wrote myself up a quick C++ wrapper just now. Will play with this. Intend to use on embedded/bare-metal systems with no C stdlib, so will have to rip out the printfs and the like.

Re: Simple Dynamic Strings library for C, compatible with null-terminated strings

#66

Post by hsaliak »

m463 wrote: post #61
poiuyt098 wrote: post #54

Earlier quoted context omitted.

> show how much you can easily improve C, with little work, and how many unsafe things in C are about lack of abstractions C has always badly needed built-in strings (and arrays with size info, generally). To save a byte, C designers committed The Most Expensive One-byte Mistake https://queue.acm.org/detail.cfm?id=2010365

A shortcoming (or outright failure) of C is the lack of an extensive "batteries included" library. Maybe not C + Knuth, but some way of portably advancing the language over time.

glib

Re: Simple Dynamic Strings library for C, compatible with null-terminated strings

#67

Post by nimish »

microcolonel wrote: post #64
antirez wrote: post #24

Hi, author here. May make sense to make SDS in perspective given a few comments I'm reading here. 1. Yep, more than "strings" SDS may be consider a library for dynamic buffers, especially from people coming from C++ or higher level languages. However I think that for C, it makes sense to provide a very low level thing like that. 2. In practice, if you see how SDS is used (extensively) inside Redis, it normally models…

> Probably an UTF-32 layer could be implemented on top of that UTF-32 is, in all but extreme niche cases, a supremely wasteful and inefficient way to store and process strings, even in light of the cost of variable-length codepoints. The longest UTF-8 sequence is now four bytes anyway, so UTF-8 is never ever less compact than UTF-32. The only conceivable practical use of UTF-32 is a case where the upper 11 bits are u…

Utf-32 allows for constant time indexing of a codepoint which may be important. Utf-16 is the worst of all worlds with its bloat and surrogate pair nonsense

Re: Simple Dynamic Strings library for C, compatible with null-terminated strings

#68

Post by pstch »

cardiffspaceman wrote: post #47
jstimpfle wrote: post #32

Earlier quoted context omitted.

UTF-8 is the preferred internal storage format for most applications. The reason is space efficiency.

I certainly have leaned on UTF-8 but I wonder how efficient it is for the numerically-higher code points if the language is not heavily cp1252, like Korean.

An interesting - but not surprising - thing about this is that compression algorithms can be more efficient on wider representations of numerically-high code points (e.g, for some Korean corpus, using UTF-32 instead of UTF-8 improves LZMA compression by ~10%).

Re: Simple Dynamic Strings library for C, compatible with null-terminated strings

#69

Post by zzo38computer »

asveikau wrote: post #34
Keyframe wrote: post #22

Earlier quoted context omitted.

Unfortunately, yes. Limited usefulness at best without unicode support, at least to a degree. Even UTF-16 or 32 internally would suffice, treating UTF-8 only as ser/de format is good enough these days.

I don't get entirely why people want or expect to store UTF-32 in memory for any substantial period. It makes more sense to me as: if you need to process codepoint at a time, parse from UTF-8 or UTF-16 in a codepoint-at-a-time fashion, leaving only ~1 codepoint decoded at a time. People seem to think that 1 codepoint = 1 integer frees you from thinking Unicode is hard. But 1 codepoint is not 1 glyph. You have combini…

The Unicode functions of Glk use UTF-32 (and numbers outside of Unicode range are used for function keys), and Glulx supports strings of 32-bit characters (usually Unicode, although this is not required) (strings of 8-bit characters are also supported, but this doesn't support UTF-8 or any other multibyte encoding). (However, strings in Glulx are usually huffed anyways, so other than the temporary buffers you would ordinarily store a variable number of bits per character (or sometimes a string of several characters has its own code), and can be shorter than using UTF-8 or UTF-32 or other uncompressed encodings.)

The stuff you mention does make Unicode very messy though.

Re: Simple Dynamic Strings library for C, compatible with null-terminated strings

#70

Post by zzo38computer »

pstch wrote: post #68
cardiffspaceman wrote: post #47

Earlier quoted context omitted.

I certainly have leaned on UTF-8 but I wonder how efficient it is for the numerically-higher code points if the language is not heavily cp1252, like Korean.

An interesting - but not surprising - thing about this is that compression algorithms can be more efficient on wider representations of numerically-high code points (e.g, for some Korean corpus, using UTF-32 instead of UTF-8 improves LZMA compression by ~10%).

How well does that corpus compress with LZMA if using a Korean specific character code (such as EUC-KR)? And what about other combinations, with other character codings and other compression algorithms?
Post reply on HN