Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Use string views instead of passing std:wstring by const&

giodicanio.com

Post reply on HN
Print view View original

41–50 of 55 posts

Re: Use string views instead of passing std:wstring by const&

#41

Post by ack_complete »

chirsz wrote: post #39

It's best to avoid using std::wstring and other wchar_t-related facilities, as they are highly non-portable across different platforms. If you need to interact with the Win32 API, use char16_t and std::u16string, so that anyone knows it contains a UTF-16 encoded string and knows how to use and process it.

The Windows API uses WCHAR = wchar_t, so if you use char16_t, you have to convert back and forth to avoid running afoul of strict aliasing rules. This imposes conversion costs without benefits; both using wchar_t directly or converting to/from UTF-8 are better.

Re: Use string views instead of passing std:wstring by const&

#42

Post by tialaramex »

masklinn wrote: post #21
theamk wrote: post #5

Earlier quoted context omitted.

x86 had 6 general-purpose working registers total. Using length + pointers would have caused a lot of extra spills.

“Sure your software crashes and your machines get owned, but at least they’re not-working very fast!”

Right. This is so often the excuse for terrible designs in C and C++. It's wrong, "But it's faster". No, it's just wrong, only for correct answers does it matter whether you were faster. If just any answer was fine there's no need to write any of this software.

Re: Use string views instead of passing std:wstring by const&

#43

Post by tialaramex »

Starting by defining the non-owning slice references, rather than the owning container types is such a massive advantage that it's very telling that Stroustrup didn't get this right in his language.

Today Bjarne will insist that the correct understanding of ownership and lifetimes was always inherent in his language and if you point out that it basically only warrants a brief mention in his early books about the language he'll say that the newer books have much more about this, as if that's not a confession...

Because &str (the non-owning reference to a slice of UTF-8 encoded text) in Rust is so ubiquitous, it's completely reasonable in Rust to use anything from the simple standard library owning String type, which is literally a growable array, Vec plus the promise about UTF-8 encoding, through to text which lives only on the stack as a re-interpreted array, or at the other extreme a raw pointer-sized short-string optimisation https://crates.io/crates/cold-string -- where unless it fits inline the string is length-prefixed on the heap like Pascal. All these approaches fit different niches and since a mere reference to the string is the same for all of them they're compatible. In C++ you will run into too many problems and regret trying.

Edited: Correct that String lives in the stdlib -- it isn't baked in to the core language because your tiny embedded system may not have an allocator to go around creating fancy growable arrays, but it still might want the non-owning string slice, &str which truly is in the core language.

Re: Use string views instead of passing std:wstring by const&

#44

Post by nopurpose »

masklinn wrote: post #18
delta_p_delta_x wrote: post #13

Earlier quoted context omitted.

> C Strings at least let you take a zero copy substring of the tail This is a special-case optimisation that I'm happy to lose in favour of the massive performance and security benefits otherwise. Isn't length + pointer... Basically a Pascal string? Unless I am mistaken. I think what was unsaid in your second point is that we really need to type-differentiate constant strings, dynamic strings, and string 'views', whi…

> Isn't length + pointer... Basically a Pascal string? Unless I am mistaken. Length + pointer is a record string, a pascal string has the length at the head of the buffer, behind the pointer.

Many years ago when reading Redis code I saw the same pattern: they pass around simple pointer to data, but there is a fixed length metadata just before that.

Re: Use string views instead of passing std:wstring by const&

#45

Post by tialaramex »

dev-ns8 wrote: post #29
theamk wrote: post #4

Earlier quoted context omitted.

First common 32 bit system was Win 95, which required 4MB of RAM (not GB!). The 4-byte prefix would be considered extremely wasteful in those times - maybe not for a single string, but anytime when there is a list of strings involved, such as constants list. (As a point of reference, Turbo Pascal's default strings still had 1-byte length field). Plus, C-style strings allow a lot of optimizations - if you have a mutab…

Besides my DA/Algo classes in College, I've never used C seriously. And you know, it's semantics like this that really make me go WTF lol.... From strtok man page... "The first time that strtok() is called, str should be specified; subsequent calls, wishing to obtain further tokens from the same string, should pass a null pointer instead." Really?? a null pointer.. This is valid code: char str[] = "C is fucking weird…

It's like this because the 1970s C programmer, typically a single individual, is expected to maintain absolute knowledge of the full context of everything at all times. So these functions (the old non-re-entrant C functions) just assume you - that solo programmer - will definitely know which string you're currently tokenising and would never have say, a sub-routine which also needs to tokenize strings.

All of this is designed before C11, which means that hilariously it's actually always Undefined Behaviour to write multi-threaded code in C. There's no memory ordering rules yet in the language, and if you write a data race (how could you not in multi-threaded code) then the Sequentially Consistent if Data Race Free proof, SC/DRF does not apply and in C all bets are off if you lose Sequential Consistency† So in this world that's enough, absolute mastery and a single individual keeping track of everything. Does it work? Not very well but hey, it was cheap.

† This is common and you should assume you're fucked in any concurrent language which doesn't say otherwise. In safe Rust you can't write a data race so you're always SC, in Java losing SC is actually guaranteed safe (you probably no longer understand your program, but it does have a meaning and you could reason about it) but in many languages which say nothing it's game over because it was game over in C and they can't do better.

Re: Use string views instead of passing std:wstring by const&

#46

Post by priceishere »

theamk wrote: post #4
delta_p_delta_x wrote: post #2

The zero-terminated string is by far C's worst design decision. It is single-handedly the cause for most performance, correctness, and security bugs, including many high-profile CVEs. I really do wish Pascal strings had caught on earlier and platform/kernel APIs used it, instead of an unqualified pointer-to-char that then hides an O(n) string traversal (by the platform) to find the null byte. There are then questions…

First common 32 bit system was Win 95, which required 4MB of RAM (not GB!). The 4-byte prefix would be considered extremely wasteful in those times - maybe not for a single string, but anytime when there is a list of strings involved, such as constants list. (As a point of reference, Turbo Pascal's default strings still had 1-byte length field). Plus, C-style strings allow a lot of optimizations - if you have a mutab…

How do you drop nulls in the middle of a string without requiring O(N) extra space to restore the original characters?

Re: Use string views instead of passing std:wstring by const&

#47

Post by priceishere »

dh2022 wrote: post #19
delta_p_delta_x wrote: post #15

Earlier quoted context omitted.

> zero copy and zero allocations This is a red herring, because when you actually read the strings out, you still need to iterate through the length for each string—zero copy, zero allocation, but linear complexity. > query file size, allocate buffer once, read it into the buffer, drop some NULL's into strategic positions, maybe shuffle some bytes around for that rare escape case, and you have a whole bunch of C stri…

I think the concern was conserving memory ( which was scarce back then) and not iterating through each substring.

But does it even conserve memory? Copying a string when you have the length is 2 bytes of machine code on x86 (rep movsb).

Remember, code takes up memory too.

Re: Use string views instead of passing std:wstring by const&

#48

Post by masklinn »

nopurpose wrote: post #44
masklinn wrote: post #18

Earlier quoted context omitted.

> Isn't length + pointer... Basically a Pascal string? Unless I am mistaken. Length + pointer is a record string, a pascal string has the length at the head of the buffer, behind the pointer.

Many years ago when reading Redis code I saw the same pattern: they pass around simple pointer to data, but there is a fixed length metadata just before that.

I assume it’s either Antirez’s sds or a variant / ancestor thereof, yes. It stores a control block at the head of the string, but the pointer points past that block, so it has metadata but “is” a C string.

Re: Use string views instead of passing std:wstring by const&

#49

Post by Someone »

Joker_vD wrote: post #14
david2ndaccount wrote: post #3

Earlier quoted context omitted.

Pascal strings might be the only string design worse than C strings. C Strings at least let you take a zero copy substring of the tail. Pascal strings require a copy for any substring! Strings should be two machine words - length + pointer (aka what is commonly called a string view). This is no different than any other array view. Strings are not a special case.

Yeah, I too feel that storing the array's length glued to the array's data is not that good of an idea, it should be stored next to the pointer to the array aka in the array view. But the thrall of having to pass around only a single pointer is quite a strong one.

> I too feel that storing the array's length glued to the array's data is not that good of an idea, it should be stored next to the pointer to the array aka in the array view.

That’s not cache-friendly, though. I think the short string optimization (keeping short strings alongside the string length, but allocating a separate buffer for longer strings. See https://devblogs.microsoft.com/oldnewthing/20240510-00/?p=10... for how various C++ compilers implement that) may be the best option.

Re: Use string views instead of passing std:wstring by const&

#50

Post by Joker_vD »

Someone wrote: post #49
Joker_vD wrote: post #14

Earlier quoted context omitted.

Yeah, I too feel that storing the array's length glued to the array's data is not that good of an idea, it should be stored next to the pointer to the array aka in the array view. But the thrall of having to pass around only a single pointer is quite a strong one.

> I too feel that storing the array's length glued to the array's data is not that good of an idea, it should be stored next to the pointer to the array aka in the array view. That’s not cache-friendly, though. I think the short string optimization (keeping short strings alongside the string length, but allocating a separate buffer for longer strings. See https://devblogs.microsoft.com/oldnewthing/20240510-00/?p=10..…

> That’s not cache-friendly, though.

How so? The string implementations in that post are pretty much that:

    struct string
    {
        char* ptr;
        size_t size;
        union {
            size_t capacity;
            char buf[16];
        };
The pointer and the size are stored together, and they may optionally be located right next to the string's actual data, but only for very small, locally-allocated, short-lived strings; but in normal usage, that pointer points somewhere into the heap.
Post reply on HN