Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why German Strings Are Everywhere?

cedardb.com

Post reply on HN
Print view View original

21–30 of 58 posts

Re: Why German Strings Are Everywhere?

#21

Post by masklinn »

maweki wrote: post #18
f1shy wrote: post #5

Earlier quoted context omitted.

I knew the C++ strings were optimized so. I do not like calling them "German". First time I see them called so (I know it, you can search yourself, as SSO -short string opt.), and looks as some kind of nationalist pride thing to me. Is certainly not a unique or new idea, many scheme/lisps implementations do that for strings AND numbers. Downvotes coming from other connationals :) love you! I know… never say anything…

What do you like to call Hungarian notation?

"System's Horrendous Pile Of Shit".

If anyone ever referred to Apps Hungarian that would be "Simonyi's Wish For A Proper Type System", but nobody ever does.

Re: Why German Strings Are Everywhere?

#22

Post by msichert »

StopDisinfo910 wrote: post #11
jmclnx wrote: post #3

If I understood what I was reading about German Strings, I think UTF-8 could add complications to these things.

Not really, no. The main difference is that you don't know how many code points you have in the prefix as they use variable encoding so it can be up to four but as little as one. I imagine the choice of four bytes for the prefix was actually done specifically for this reason. That's the maximum length of a UTF-8 code point. The length is not the number of characters anymore but just the size of the string. Apart from…

We chose 4B because that was the maximum number of bytes that would be unused otherwise (4B for the length, 8B for the pointer leaves 4B), the UTF8 encoding doesn't really matter.

Also, for UTF8 specifically, cutting code points in half is fine as long as all strings are valid UTF8. The UTF8 encoding is prefix free, i.e., no valid code point is a prefix of another valid code point, so for prefix matching we can usually just compare bytes.

It only gets more complicated if you add collations or want to match case-insensitively. But at that point you need to take into account all edge cases of the Unicode spec anyway.

Re: Why German Strings Are Everywhere?

#23

Post by f1shy »

maweki wrote: post #18
f1shy wrote: post #5

Earlier quoted context omitted.

I knew the C++ strings were optimized so. I do not like calling them "German". First time I see them called so (I know it, you can search yourself, as SSO -short string opt.), and looks as some kind of nationalist pride thing to me. Is certainly not a unique or new idea, many scheme/lisps implementations do that for strings AND numbers. Downvotes coming from other connationals :) love you! I know… never say anything…

What do you like to call Hungarian notation?

I don’t know any other name for it. While this strings are basically SSO (or a twist of it).

Re: Why German Strings Are Everywhere?

#24

Post by masklinn »

jmclnx wrote: post #3

If I understood what I was reading about German Strings, I think UTF-8 could add complications to these things.

They just store bytes. A leading astral codepoint means your prefix store contains just one codepoint, but that doesn't really change anything per se.

Re: Why German Strings Are Everywhere?

#25

Post by bjourne »

"Optimized" string types are everywhere and I bet that multiple people have already created string types almost identical to German strings. But the memory savings are small and they are not more efficient than ordinary strings. For string comparison you compare the pointers, which is cheaper than comparing two pairs of registers. If the pointers mismatch you compare the (cached) hashes and only if they match do you need to compare characters. For the prefix query, starts_with(content, 'http'), just store a string of the four-character prefix. With immutable strings the overhead is just one pointer.

Re: Why German Strings Are Everywhere?

#26

Post by f1shy »

masklinn wrote: post #19
f1shy wrote: post #5

Earlier quoted context omitted.

I knew the C++ strings were optimized so. I do not like calling them "German". First time I see them called so (I know it, you can search yourself, as SSO -short string opt.), and looks as some kind of nationalist pride thing to me. Is certainly not a unique or new idea, many scheme/lisps implementations do that for strings AND numbers. Downvotes coming from other connationals :) love you! I know… never say anything…

C++ strings are not "optimized so". C++ strings (generally) do SSO (up to 23 bytes depending on implementation), these also do SSO but only 8 bytes (to a total of 12), the first 4 bytes are always stored inline for fast lookup even when the rest of the string is on the heap (in which case they're duplicated), and the strings are limited to 4GB (32 bits length). IIRC they also have a bunch of other limitations (e.g. t…

Yes. Sorry. That was not 100% correct. Still, your words (my emphasis)

>C++ strings are not "optimized so". C++ strings (generally) do SSO (up to 23 bytes depending on implementation), these also do SSO but only 8 bytes (to a total of 12)

That is what I meant. I would (like you did) call it SSO still.

Re: Why German Strings Are Everywhere?

#27

Post by cubefox »

nkrisc wrote: post #13
cubefox wrote: post #4

The added question mark in the HN submission makes little sense.

It also makes it grammatically incorrect. If it were actually a question it should be, “Why are German strings everywhere?”

The other form seems to be an Indian English colloquialism.

Re: Why German Strings Are Everywhere?

#28

Post by masklinn »

aDyslecticCrow wrote: post #20

Interesting to see a deepdive about string formats. I hadn't thought very deeply about it before. I do agree with the string imutable argument. Mutable and imutable strings have different usecases and design tradeoffs. They perhaps shouldn't be the same type at all. The transient string is particularly brilliant. Ive worked with some low level networking code in c, and being able to create a string containing the "pa…

> The transient string is particularly brilliant. Ive worked with some low level networking code in c, and being able to create a string containing the "payload" by pointing directly to an offset in the raw circular packet buffer is very clean. (the alternative is juggling offsets, or doing excessive memcpy)

It's not anything special? That's just `string_view` (C++17). Java also used to do that as an optimisation (but because it was implicit and not trivial to notice it caused difficult do diagnose memory leaks, IIRC it was introduced in Java 1.4 and removed in 1.7).

Re: Why German Strings Are Everywhere?

#29

Post by [deleted] »

bjourne wrote: post #25

"Optimized" string types are everywhere and I bet that multiple people have already created string types almost identical to German strings. But the memory savings are small and they are not more efficient than ordinary strings. For string comparison you compare the pointers, which is cheaper than comparing two pairs of registers. If the pointers mismatch you compare the (cached) hashes and only if they match do you…

[deleted]

Re: Why German Strings Are Everywhere?

#30

Post by f1shy »

bjourne wrote: post #25

"Optimized" string types are everywhere and I bet that multiple people have already created string types almost identical to German strings. But the memory savings are small and they are not more efficient than ordinary strings. For string comparison you compare the pointers, which is cheaper than comparing two pairs of registers. If the pointers mismatch you compare the (cached) hashes and only if they match do you…

Do you have a pointer to real world data about the effectiveness of these optimizations? I learned about it (SSO, in std lib which is basically the same) in an article which really made it look as that would make anything in C++ blazing fast. In the codebases I worked, a couple of times, I did measure (what you shoud do before optimizing) and the results where between absolutely negligible to worst when active. But that were 3 data points. Mind you one in a real time database.
Post reply on HN