Live data from Hacker News

Plain text has been around for decades and it’s here to stay

unsung.aresluna.org

81–90 of 172 posts

Re: Plain text has been around for decades and it’s here to stay

#81
post #33

Earlier quoted context omitted.

UTF-8 encodes European glyphs in two bytes and oriental glyphs in three bytes. This is due to the assumption that you're not going to be using oriental glyphs. If you are going to use them, UTF-8 is a very poor choice.

UTF-8 does not encode "European glyphs" in two bytes, no. Most European languages use variations of the latin alphabet, meaning most glyphs in European languages use the 1-byte ASCII subset of UTF-8. The occasional non-ASCII glyph becomes two bytes, that's correct, but that's a much smaller bloat than what you imply. Anyway, what are you comparing it to, what is your preferred alternative? Do you prefer using code pa…

Unicode could have just been encoded statefuly with a "current code page" mark byte.

With UTF and emojis we can't have random access to characters anyways, so why not go the whole way?

Re: Plain text has been around for decades and it’s here to stay

#82
post #58

Earlier quoted context omitted.

> The terminal is capable of flexbox now. You mean like https://silvery.dev/examples/layout.html ? This is definitely not a UI development paradigm I would have expected to see.

That’s the tip of a conceivable iceberg but exactly. Also look at kitty graphics protocol. Look at the amount of engineering resources we pour into OS GUI toolkits and then browsers. Those layers of complexity aren’t there because we stood back and said, “given what we know in 2026 how should we design a GUI compositor?”. The majority of the stack is written how it is by archeological happenstance. One generation add…

I 100% agree, and this isna big reason why I find the current state of education so suboptimal. Everyone just goes on to do webdev, completely ignoring the lower levels and taking it all for granted. The thing is, there's no real innovation to be done that high up the stack. When you're that high you mostly just write glue code to stick parts someone else wrote together. Real innovation comes from quite a few levels down the stack, starting at the native code level downwards.

Like you pointed out, the current stack is heavily unoptimized and has a terrible architecture; it's only the way it is because of happenstance and tides of the market (companies always reaching for faster over better). An actual "nirvana" in computing like the other guy said would require bulldozing a good chunk of our current stack, keeping only kernels and core utilities, if even.

I really wish we had a bigger focus on getting good foundation instead of making yet another JS framework and SaaS, but then again, who's paying developers to actually do something of quality nowadays?

Re: Plain text has been around for decades and it’s here to stay

#83
post #73

Earlier quoted context omitted.

> If you only care about the UX of TUIs, that I can stand behind This is a confusing concession. Of course we love TUIs because of the UX, what other reason is there? Constraint breeds consistency and consistency breeds coherence. Take 1,000 random TUI designers and 1,000 random GUI designers and plot the variations between them (use any method you like)—the TUI designers will be more tightly clustered together becau…

> the TUI designers will be more tightly clustered together because the TUI interface constrains what's reasonable. It constrains what’s possible, not what’s reasonable. For example, one could typically fit more text on a screen by compressing it, but most of the time, that’s not the reasonable thing to do. I’m saying most of the time because of the existence of English Braille ( https://en.wikipedia.org/wiki/English…

> It constrains what’s possible, not what’s reasonable.

Why do you say "constrains what’s possible, not what’s reasonable", as though it's one and not the other? Does possibility conflict with reasonability? I would think it's not an either/or, it's a both/and.

The set of reasonable things is bounded by the set of possible things. So if the constraints of TUI design make certain things impossible, surely they make those same things unreasonable at the same time.

Re: Plain text has been around for decades and it’s here to stay

#84
Text is Lindy. It has withstood the test of time and it's as ubiquitous as SQL or TCP/IP.

Reminds me of this decade old post (and discussion) by Graydon Hoare, "Always bet on text".

[1]: https://news.ycombinator.com/item?id=8451271

[2]: https://graydon2.dreamwidth.org/193447.html

Re: Plain text has been around for decades and it’s here to stay

#85
post #33

Earlier quoted context omitted.

UTF-8 does not encode "European glyphs" in two bytes, no. Most European languages use variations of the latin alphabet, meaning most glyphs in European languages use the 1-byte ASCII subset of UTF-8. The occasional non-ASCII glyph becomes two bytes, that's correct, but that's a much smaller bloat than what you imply. Anyway, what are you comparing it to, what is your preferred alternative? Do you prefer using code pa…

Unicode could have just been encoded statefuly with a "current code page" mark byte. With UTF and emojis we can't have random access to characters anyways, so why not go the whole way?

A huge, central, part of UTF-8 design is that you can start decoding it from any arbitrary offset, it is self-aligning.

Re: Plain text has been around for decades and it’s here to stay

#86
post #33

Earlier quoted context omitted.

UTF-8 does not encode "European glyphs" in two bytes, no. Most European languages use variations of the latin alphabet, meaning most glyphs in European languages use the 1-byte ASCII subset of UTF-8. The occasional non-ASCII glyph becomes two bytes, that's correct, but that's a much smaller bloat than what you imply. Anyway, what are you comparing it to, what is your preferred alternative? Do you prefer using code pa…

Unicode could have just been encoded statefuly with a "current code page" mark byte. With UTF and emojis we can't have random access to characters anyways, so why not go the whole way?

Yikes. That would lose the ability to know the meaning of the current bytes, or misinterpret them badly, if you happen to get one critical byte dropped or mangled in transmission. At least UTF-8 is self-syncing: if you end up starting to read in the middle of a non-rewindable stream whose beginning has already passed, you can identify the start of the next valid codepoint sequence unambiguously, and then end up being able to sync up with the stream, and you're guaranteed not to have to read more than 4 bytes (6 bytes when UTF-8 was originally designed) in order to find a sync point.

But if you have to rely on a byte that may have already gone past? No way to pick up in the middle of a stream and know what went before.

Re: Plain text has been around for decades and it’s here to stay

#87
post #49

Earlier quoted context omitted.

But why? You easily have 4k pixels, why use a tiny subset of those in a very inefficient way? We have proper hardware to make a bunch of these computations actually fast, and yet we should stuck with drawing relatively expensive text everywhere? If you only care about the UX of TUIs, that I can stand behind (though mostly as a guideline, it doesn't fit every workflow), but you can do that with a proper GUI just as we…

One of TUI advantages over GUIs (including modern web sites) - all text can be selected/copied (you may need to use modifies in some TUI). It's a bit frustrating when GUI shows text but I cannot select and copy it.

That's a very good point. I hadn't thought about that aspect before.

Re: Plain text has been around for decades and it’s here to stay

#88
post #31

Earlier quoted context omitted.

Hm? UTF-8 encodes all of ASCII with one byte per character, and is pretty efficient for everything else. I think the only advantage UTF-16 has over UTF-8 is that some ranges (such as Han characters I believe?) are often 3 bytes of UTF-8 while they're 2 bytes of UTF-16. Is that your use case? Seems weird to describe that as "all text files" though?

UTF-8 encodes European glyphs in two bytes and oriental glyphs in three bytes. This is due to the assumption that you're not going to be using oriental glyphs. If you are going to use them, UTF-8 is a very poor choice.

UTF-8 may still be a good choice for Japanese text, though.

For one thing, pure text is often not the only thing in the file. Markup is often present, and most markup syntaxes (such as HTML or XML) use characters from the ASCII range for the markup, so those characters are one byte (but would be two bytes in UTF-16). Back when the UTF-8 Everywhere manifesto (https://utf8everywhere.org/) was being written, they took the Japanese-language Wikipedia article on Japan, and compared the size of its HTML source between UTF-8 and UTF-16. (Scroll down to section 6 to see the results I'm about to cite). UTF-8 was 767 KB, UTF-16 was 1186 KB, a bit more than 50% larger than UTF-8. The space savings from the HTML markup outweighed the extra bytes from having a less-efficient encoding of Japanese text. Then they did a copy-and-paste of just the Japanese text into a text file, to give UTF-16 the biggest win. There, the UTF-8 text was 222 KB while the UTF-16 encoding got it down to 176 KB, a 21% win for UTF-16 — but not the 50% win you would have expected from a naive comparison, because Japanese text still uses many characters from the ASCII set (space, punctuation...) and so there are still some single-byte UTF-8 characters in there. And once the files were compressed, both UTF-8 and UTF-16 were nearly the same size (83 KB vs 76 KB) which means there's little efficiency gain anyway if your content is being served over a gzip'ed connection.

So in theory, UTF-8 could be up to 50% larger than UTF-16 for Japanese, Chinese, or Korean text (or any of the other languages that fit into the higher part of the basic multilingual place). But in practice, even giving the UTF-16 text every possible advantage, they only saw a 20% improvement over UTF-8.

Which is not nearly enough to justify all the extra cost of suddenly not knowing what encoding your text file is in any more, not when we've finally reached the point of being able to open a text file and just know the encoding.

P.S. I didn't even mention the Shift JIS encoding, and there's a reason I didn't. I've never had to use it "for real", but I've read about it. No. No thank you. No. Shudder. I'm not knocking the cleverness of it, it was entirely necessary back when all you had was 8 bits to work with. But let me put it this way: it's not a coincidence that Japan invented a word (mojibake) to represent what happens when you see text interpreted in the wrong encoding. There were multiple variations of Shift JIS (and there was also EUC-JP just to throw extra confusion into the works), so Japanese people saw garbled text all the time as it moved from one computer running Windows, to an email server likely running Unix, to another computer running Windows... it was a big mess. It's also not a coincidence that (according to Wikipedia), 99.1% of Japanese websites (defined as "in the .jp domain") are encoded in UTF-8, while Shift JIS is used by only 1% (probably about 0.95% rounded up) of .jp websites.

So in practice, nearly everyone in Japan would rather have slightly less efficient encoding of text, but know for a fact that their text will be read correctly on the other end.

Re: Plain text has been around for decades and it’s here to stay

#89

> Fun to see a contemporary take on something that peaked between 1970s–1980s Maybe that was the peak, but you had some very good TUIs in the early 1990's for DOS apps, where Windows hadn't quite completely taken over yet, but you very likely had a VGA-compatible graphics card and monitor, meaning you had a good, high-resolution, crisp and configurable-font text mode available, and also likely had a mouse . This is t…

The peak of TUIs is now. Take a look at Omarchy, an entire operating system built around terminals and config files, it's nirvana. I can only imagine how much farther down this road things may go as we enter a world where the primary interface is conversation with the machine in text. I'm sure I'll get downvoted for that last part because Reddit -- (cough) I mean Hacker News - hates AI, but I'm genuinely excited for…

We had "opinionated" TUIs with emacs, and Omarchy will never surpass emacs' ease, shallow learning curve, and configurability. Emacs is the operating system of the future, and you can already integrate AI with it. It provides everything you need or want or don't know you want except a decent text editor.
Post reply on HN