Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Everything you ever wanted to know about terminals

xn--rpa.cc

Post reply on HN
Print view View original

171–180 of 191 posts

Re: Everything you ever wanted to know about terminals

#171

Post by Beltalowda »

duskwuff wrote: post #78
Beltalowda wrote: post #35

Earlier quoted context omitted.

Or just allow only using a single script. A domain in all Cyrillic: great! Mixing Latin & Cyrillic: nono. In practice, browsers already check for this and display the "raw" punycode if they detect mixed script usage, but I wish such domains would not be registrable at all. These checks are somewhat complex and difficult, and easy to get wrong.

> Or just allow only using a single script. This would still let some homographs through. In particular, Cyrillic has a lot of characters which are confusingly similar to, or even indistinguishable from, Latin characters (e.g. "авсекморѕтѵху").

Right; you can construct "арр.com" or "аррꙆе.com" from that limited subset.

Those should be valid domains though IMHO; maybe show the used script in the address bar? I think users might be confused by that though and/or just ignore it, so idk. Then again, displaying "xn--80a6aa.com" and "xn--80ak6aa9058r.com" is pretty confusing too.

Re: Everything you ever wanted to know about terminals

#172

Post by csdvrx »

dllthomas wrote: post #136
csdvrx wrote: post #5

> but i swear to god developers have so completely forgotten how terminals work that i might be one of a handful of people left on earth who actually has the knowledge to, so they all just layer their bullshit on top of ncurses (which should never have survived the '90s) instead and it's ｍａｄｄｅｎｉｎｇ． Actually, yes, understanding the tty in detail seems to become a dark art. However it's the best way to do complex thing…

I do wish there was a save/restore cursor variant that would scroll with the text.

It should be a super simple feature to add to your terminal emulator: SCP works with a X,Y position. RCP just "jumps" there.

If you keep an accounting of how many lines you have displayed since then, you could alter the response to RCP by also doing the appropriate amount of scrolling: it should only take one variable, the deltaY to scroll.

If you want to test the idea, I think you could even use tmux and send commands to control the scrollback cf ahttps://superuser.com/questions/209437/how-do-i-scroll-in-tm...

I've used similar tricks with RCP/SCP but for simpler things: the only slight difficulty is the deltaY accounting, like when you are executing commands near the bottom of the screen because you must take into account that scolling will happen - but it's essentially similar to your idea.

So check https://github.com/csdvrx/bash-timestamping-sqlite/blob/main... and make sure you understand both how the __notbottom function works, and why PS0 needs an extra Esc[2a

Actually, now that I think more about your idea, it would be sweet to keep a SCP/RCP stack with multiple values, where you can push values with each SCP then pop them with RCP, say in sequence, or maybe just access the nth value with a different command that wouldn't pop them? That could be done nicely by augmenting RCP.

Also you could augment SCP with an optional flag to specify whether the terminal should scroll back upon RCP of this nth entry, and you'd have a great function that would be quite useful (ex: SCP with a jump bool when the return is non 0: you could make a shortcut to jump to the commands that have returned errors)

There's no reason to stop adding cool features to terminals: we're in a terminal renaissance!

Re: Everything you ever wanted to know about terminals

#173

Post by kazinator »

jart wrote: post #117
kazinator wrote: post #65

Here is something I learned only several weeks ago. While working on Pipe Watch, I strayed into reading the standard. * The ESC [ command start sequence is actually a compromise for 7 bit systems. The [ character is not chosen by accident. It has an obvious positional relationship to ESC in the ASCII code which is why, informally, Ctrl-[ is the same as ESC. * If you have an 8-bit-clean channel to the terminal, only a…

Yes but if you're using a modern UTF-8 terminal the C1 control code for CSI takes two bytes to encode. So I doubt it's very well supported.

Does it? That depends on whether the control sequences are being encoded as UTF-8, or transmitted literally in between UTF-8 characters.

If they are being encoded, this is still useful. Though space isn't saved, any ambiguity between the control bytes and UTF-8 bytes is eliminated. The advantage is still present that CSI is different from ESC, and so in the terminal->host direction, you don't have the ambiguity between ESC as a UI command character versus control sequence signal byte.

Re: Everything you ever wanted to know about terminals

#174

Post by duskwuff »

kazinator wrote: post #169
duskwuff wrote: post #71

Earlier quoted context omitted.

> If you have an 8-bit-clean channel to the terminal, only a single character is required: the "upper escape" from the C1 control character set (0x80 to 0x9F). I'd avoid using this. It conflicts badly with UTF-8's use of 0x80 through 0xBF as continuation characters.

I don't entirely buy the argument because in a regular ASCII control sequence like ESC[5A, you have the same problem. All those characters have a role outside of the signaling, so if either side is in an unexpected state, they get misinterpreted. This is just the risk of in-band signaling. Of course, if the terminal is ignorant of UTF-8 then this is a nonstarter, because whenever CIS occurs as a continuation byte, it…

The fundamental problem with mixing C1 controls with UTF-8 is that it forces the terminal emulator to break layering. It can't run a UTF-8 decoder first, because that'll turn the C1 controls into replacement characters, and it can't run a terminal sequence decoder first either, because that'll treat a lot of the UTF-8 continuation characters as control sequences. And what you're likely to find if you start using C1 controls is that support for them in terminal emulators is often incomplete and/or buggy. Handling them correctly in conjunction with UTF-8 text is difficult, and many terminals just don't bother.

The ambiguous nature of ESC in the terminal->host direction (as you put it) is unfortunate, but is difficult to fix. Some terminals (like iTerm) can be configured to use C1 controls for function keys, but my experience has been that a lot of software fails to recognize these sequences, making it impractical to use.

Re: Everything you ever wanted to know about terminals

#175

Post by entelechy0 »

sophacles wrote: post #46

The author seems not to be aware of the recent TUI rennaisance[1]. There are libraries like termbox and blessings (python) that are a middle-ground between full ncurses and adding your own ansi codes. There are a lot of modern TUI frameworks like tui-go or tui-rs that bring common GUI conventions back to the TUI (heck there are TUI programming libraries that are designed to be similar to react) - these too tend to be…

No post body was provided.

Re: Everything you ever wanted to know about terminals

#176

Post by kazinator »

duskwuff wrote: post #174
kazinator wrote: post #169

Earlier quoted context omitted.

I don't entirely buy the argument because in a regular ASCII control sequence like ESC[5A, you have the same problem. All those characters have a role outside of the signaling, so if either side is in an unexpected state, they get misinterpreted. This is just the risk of in-band signaling. Of course, if the terminal is ignorant of UTF-8 then this is a nonstarter, because whenever CIS occurs as a continuation byte, it…

The fundamental problem with mixing C1 controls with UTF-8 is that it forces the terminal emulator to break layering. It can't run a UTF-8 decoder first, because that'll turn the C1 controls into replacement characters, and it can't run a terminal sequence decoder first either, because that'll treat a lot of the UTF-8 continuation characters as control sequences. And what you're likely to find if you start using C1 c…

The fundamental problem with TCP/IP is that it forces the stack to break layering. In the same frame of bytes, you have a confusing mix of ethernet addressing, IP header, and a payload of application data, all from totally different pieces of software in the system. Even the data itself is fragmented, with some session wrapping around content that are done by different application stacks.

> It can't run a UTF-8 decoder first, because that'll turn the C1 controls into replacement characters, and it can't run a terminal sequence decoder first either because that'll treat a lot of the UTF-8 continuation characters as control sequences.

It has to have a state machine which recognizes the combined language of UTF-8 sequences and control sequences. Which is the approach you would take anyway, even with C0 controls.

That combined language is an unambiguous, regular set, so you could code it with your eyes closed.

Starting in an initial state, the legal inputs are: ASCII character, Unicode character, or escape sequence headed by CSI. This is decidable from reading exactly one byte value with no further lookahead.

That's just one way. You can in fact follow a layered approach whereby the terminal decodes everything with UTF-8 before analyzing it for control or data.

For instance, say we decode UTF-8 into integer code points. A valid character decodes into its implied code point. An invalid byte like CSI can decode into some reserved range like U+DCxx. The higher layer of the terminal's firmware then looks for values in that U+DCXX range: that's where it finds the CSI.

I have years of experience with this exact encoding scheme, which I baked into the text I/O streams of a programming language.

For instance, oh, /proc/self/environ is NUL-separated, right? No problem:

  1> (file-get-string "/proc/self/environ")
  "CLUTTER_IM_MODULE=xim\xDC00XDG_MENU_PREFIX=gnome-\xDC00LANG=en_CA.UTF-8\xDC00;D
  [...]
  -go:@/tmp/.ICE-unix/2065,unix/sun-go:/tmp/.ICE-unix/2065\xDC00GTK_IM_MODULE=ibus
  \xDC00_=/usr/local  /bin/txr\xDC00"
The NULs are rendered into \xDC00 codes. This is called the "pseudo-null" character in the terminology of this language, and has a symbolic name: #\pnul:

  2> #\xDC00
  #\pnul
We can split the data on it to recover the list of environment entries:

  3> (take 4 (spl #\pnul *1))
  ("CLUTTER_IM_MODULE=xim" "XDG_MENU_PREFIX=gnome-" "LANG=en_CA.UTF-8"
   "DISPLAY=:1")
In a similar way, any other invalid byte can be used for framing, if the data in between is all valid.

Re: Everything you ever wanted to know about terminals

#177

Post by knorker »

jart wrote: post #120
knorker wrote: post #51

Yeah this is very very far from "everything". Among many things it doesn't speak of ptys. And terminfo, pty, ctty, process groups, and the rendering width of a utf-8 string, and, and and… > i might be one of a handful of people left on earth who actually has the knowledge to Uh, no. Anyone on BBSes in the 90s is very aware of ANSI, thank you. And we've not died off yet. And honestly it's really not that hard at all.…

> Uh, no. Anyone on BBSes in the 90s is very aware of ANSI, thank you. And we've not died off yet. You guys should be more vocal. When you have great knowledge like that, you can't just keep it to yourself as a fond memory. People wouldn't be saying what the OP said if more of the oldskool crowd was out there blogging and mentoring the younger generation.

It's not just a fond memory. I use it in my open source. Other people use it.

It's not obscure.

One thing I actually agree with the author about is that these escape codes are the only relevant thing. Outside of retro computing nobody should care about supporting anything else.

Other programs doing this are not exactly in short supply. Anyone can do "ls --color /bin/ls | hexdump -C" and see the secret sauce.

I blog about various things, partly to help out people who are less experienced. But I don't pretend what I write about is some sort of lost art, that "only a handful of people know".

Like "how do I make bold text in linux terminal" gives as second result this:

https://askubuntu.com/questions/528928/how-to-do-underline-b..., which also links to https://misc.flogisoft.com/bash/tip_colors_and_formatting.

Searching on youtube I immediately got this, for those who prefer in video form: https://www.youtube.com/watch?v=OL21-EnsNjQ

And it manages to do this without the pure arrogance of "i might be one of a handful of people left on earth who actually has the knowledge".

Seriously, this whole article could just have been just:

"Q: How do I output more than just print black&white text to the terminal? A: ANSI escape codes. See https://en.wikipedia.org/wiki/ANSI_escape_code"

Not "everything you wanted to know about terminals".

Like, "how does Ctrl-C work? What's flow control?". No, this post is entirely about ANSI codes.

What's extra frustrating is that you too are calling this "oldskool crowd". I'm just not that old, and this just isn't forgotten. It's simply another tool that people use when they need to solve the problem of colors, etc.

Just because many people don't know how a malloc()/new becomes a mmap or sbrk doesn't make it "oldskool". It's simply a thing that many people have not learned yet, because they haven't needed to. If and when they need to it's quite documented and many others know it, if they want more hand holding.

Like say I didn't know how garbage collectors worked. I don't go "Oh you older generation of lisp programmers, you need to blog more and teach us younglings, so that we can understand the languages that we use". Sure, blogging etc about GCs is good, but who would be arrogant enough to just write an article about "old gen and new gen" and call it "everything you ever wanted to know about GC" and claim that they are one of the handful of people who understand GCs.

Re: Everything you ever wanted to know about terminals

#178

Post by chris37879 »

duskwuff wrote: post #143
jart wrote: post #116

Earlier quoted context omitted.

Hi I'm one of the people cited by the article. Fraktur is awesome. I implemented it in my terminal. https://github.com/jart/cosmopolitan/blob/c6bbca55e9f977e386... Now that unicode makes it easy, there's no excuse not to have fraktur!

Wow, okay, guess I can cross that one off my list of "ECMA-48 features nobody implements". There's still plenty left on that list, though -- including at least one escape sequence I'm fairly certain was included as a joke (SPQR).

I can think of a use case or 2. Thermal printers (for those wanting old timey unix line printing interfaces) or e-ink displays in console mode.

Re: Everything you ever wanted to know about terminals

#179

Post by randomswede »

voidhorse wrote: post #102

I can tell you why people still use ncurses: if you’re actually interested in accounting for all the nuanced eldritch madness that is terminal emulation implementation and history it’s a monumental effort, at least imo. Sure, if all you care about is coloring text on recent terminals, I do think just using ansi codes is fine. However, I completely disagree with the assertion that hardcoding ansi codes is somehow “mor…

And hard-coding terminal escape codes intrinsically tie you to a VT102-derivative (because no one would hard-code VT-52 terminal codes) and that'd stop (say) my HP terminal from working.

Sure, going with VT102-family codes probably covers somewhere between four and seven 9s of cases, but...

Re: Everything you ever wanted to know about terminals

#180

Post by account42 »

duskwuff wrote: post #69
jjoonathan wrote: post #12

I was aware of the ANSI escape sequences and even used them directly in scripts on occasion, but I still used ncurses "where it mattered" because I didn't know about compatibility. I didn't want to risk Windows or a random flavor of linux I'd never heard of or a group of anti-VT100 enthusiasts getting upset because I didn't use an agreed upon compatibility layer. From the tone of this piece I gather that the ANSI esc…

> From the tone of this piece I gather that the ANSI escape codes are actually standard enough to target. Correct. Hardware terminals are extinct in the wild, and essentially all software terminals (including the Windows terminal!) now support a reasonable subset of "extended VT100" terminal control sequences. Some of the weirder features of the VT100 (like double-high/double-wide text or VT52 compatibility mode) are…

> including the Windows terminal!

Only since Windows 10 version 1511 (released 2015) which may or may not be too recent for you to assume support.

Post reply on HN