Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Introducing the Windows Pseudo Console (ConPty)

blogs.msdn.microsoft.com

Post reply on HN
Print view View original

101–110 of 196 posts

Re: Introducing the Windows Pseudo Console (ConPty)

#101

Post by a3f »

speeder wrote: post #95
mehrdadn wrote: post #83

Earlier quoted context omitted.

Confused, where does 2^21 code points come from and how is that related to the UTF-16 vs. UTF-8 distinction? Can't both of them encode all Unicode code points? Or are you thinking of code units perhaps, and UCS-2? Although even there I'm confused where the 2^21 came from.

Utf-16 is always 16 bit, utf-8 is variable, can go from 8 to 32 as needed... This does make coding for utf-8 harder, but when it works is really wonderful stuff.

UCS-2 is always 2 octets, UTF-16 allows for so-called surrogate pairs which expands the bits available making it variable size.

Re: Introducing the Windows Pseudo Console (ConPty)

#102

Post by mehrdadn »

speeder wrote: post #95
mehrdadn wrote: post #83

Earlier quoted context omitted.

Confused, where does 2^21 code points come from and how is that related to the UTF-16 vs. UTF-8 distinction? Can't both of them encode all Unicode code points? Or are you thinking of code units perhaps, and UCS-2? Although even there I'm confused where the 2^21 came from.

Utf-16 is always 16 bit, utf-8 is variable, can go from 8 to 32 as needed... This does make coding for utf-8 harder, but when it works is really wonderful stuff.

Look up UCS-2 that I mentioned in my comment.

Re: Introducing the Windows Pseudo Console (ConPty)

#103

Post by yuubi »

mehrdadn wrote: post #83
cryptonector wrote: post #58

Earlier quoted context omitted.

Hear hear! wchar_t is a disaster. UTF-16 is terrible. I'm not at all convinced that 2^21 codepoints will be enough, so someday it'd be nice to be able to get past UTF-16 and move to UTF-8, and Windows and ECMAScript are the biggest impediments to that. Your choice of UTF-8 will tend to place UTF-8 on a level playing field in Win32. I guess, too, that this is the end of codepages -- I doubt they'd go away, but there s…

Confused, where does 2^21 code points come from and how is that related to the UTF-16 vs. UTF-8 distinction? Can't both of them encode all Unicode code points? Or are you thinking of code units perhaps, and UCS-2? Although even there I'm confused where the 2^21 came from.

UTF-16 has a limit on the size of a code point because a code point is either a single normal code unit or a pair of surrogate code units, each encoding 10 bits of the code point (I think I used the right terminology). UTF-8 has a natural extension path to up to 7-byte encodings with all the usual UTF-8 properties (first code unit indicates how many remain, other code units are recognizable as not the first).

Re: Introducing the Windows Pseudo Console (ConPty)

#104

Post by pjmlp »

cryptonector wrote: post #55
ChuckMcM wrote: post #40

This is pretty huge. For as long as I can remember the response to command line applications talking to command line applications was "Why would you want to do that? Use (RPC | shared memory | some other IPC mechanism)." And nobody at Microsoft seemed to understand how much simpler it was to use ptys. They seem to have completely capitulated to the notion ptys and are dropping them into the next release of W10. I wis…

It's the reality of the market, which is why Windows is adding Linux compatibility (as is every *BSD, Illumos, ...). But also it's the fact that three decades of not even life support has left the Windows console in pretty sad shape -- the folks tasked with getting it into better shape were bound to see the value of ptys. Lastly, don't forget that Windows NT was meant to be a console OS, like VMS. There must still be…

As power user that gets things done on Windows, it never bothered me that it hasn't an UNIX like console.

If fact it bothered me more that I couldn't get a Borland like devenv on Linux and had to keep myself happy with XEmacs.

Re: Introducing the Windows Pseudo Console (ConPty)

#105

Post by mehrdadn »

yuubi wrote: post #103
mehrdadn wrote: post #83

Earlier quoted context omitted.

Confused, where does 2^21 code points come from and how is that related to the UTF-16 vs. UTF-8 distinction? Can't both of them encode all Unicode code points? Or are you thinking of code units perhaps, and UCS-2? Although even there I'm confused where the 2^21 came from.

UTF-16 has a limit on the size of a code point because a code point is either a single normal code unit or a pair of surrogate code units, each encoding 10 bits of the code point (I think I used the right terminology). UTF-8 has a natural extension path to up to 7-byte encodings with all the usual UTF-8 properties (first code unit indicates how many remain, other code units are recognizable as not the first).

Where are you getting this information though? I haven't worked out the bits myself yet but Wikipedia's first sentence itself says UTF-16 can encode all 1,112,064 valid code points of Unicode, which is already more than 2^(10+10) = 1,048,576.

Re: Introducing the Windows Pseudo Console (ConPty)

#106

Post by cryptonector »

TeMPOraL wrote: post #100
swozey wrote: post #60

Earlier quoted context omitted.

Could you elaborate? I've been under the guise for most of my career that doubling a digit leads to huge benefits that I'm too comp-sci ignorant to understand.

> I've been under the guise for most of my career that doubling a digit leads to huge benefits that I'm too comp-sci ignorant to understand. I was confused about this for years, too. But it turns out it's just a problem of bad naming . Happens more in this industry than we'd like to admit. As other explained, it boils down to UTF-16 being 16-bit, and UTF-8 being anything from 8- to 32-bit. It should have been named U…

UTF-16 is a variable-length encoding using up to two code units which each are 16-bits wide.

UTF-8 is a variable-length encoding using up to 4 code units (though it used to be up to 6, and could again be up to 6) each of which are 8-bits wide.

Both, UTF-16 and UTF-8 are variable-length encodings!

UTF-32 is not variable-length, but even so, the way Unicode works a character like ´ (á) can be written in two different ways, one of which requires one codepoint and one of which requires two (regardless of encoding), while ṻ (LATIN SMALL LETTER U WITH MACRON AND DIAERESIS) can be written in up to five different ways requiring from one to three different codepoints (regardless of encoding).

Not every character has a one-codepoint representation in Unicode, or at least not every character has a canonically-pre-composed one-codepoint representation in Unicode.

Therefore, many characters in Unicode can be expected to be written in multiple codepoints regardless of encoding. Therefore all programmers dealing with text need to be prepared for being unable to do an O(1) array index operation to get at the nth character of a string.

(In UTF-32 you can do an O(1) array index operation to get to the nth codepoint, not character, but one is usually only ever interested in getting the nth character.)

Re: Introducing the Windows Pseudo Console (ConPty)

#107

Post by cryptonector »

caf wrote: post #98
quotemstr wrote: post #93

Earlier quoted context omitted.

I want a pipe2(2) flag or an fcntl or something that lets me signal "the other end of this pipe understands ANSI escapes"

I've considered before that you could allow each end of a pipe to set name/value attributes, which can be read by the other end.

And now you're reinventing the Windows Console API... :)

I mean, both approaches have their pluses, but the API approach is only ever going to work well for remoting if it is standardized and interoperable. And the installed base of Unix termcap/terminfo programs is huge, so plain old text-with-in-band-controls is not going away anytime soon.

Re: Introducing the Windows Pseudo Console (ConPty)

#108

Post by zvrba »

ChuckMcM wrote: post #40

This is pretty huge. For as long as I can remember the response to command line applications talking to command line applications was "Why would you want to do that? Use (RPC | shared memory | some other IPC mechanism)." And nobody at Microsoft seemed to understand how much simpler it was to use ptys. They seem to have completely capitulated to the notion ptys and are dropping them into the next release of W10. I wis…

> Why would you want to do that? Use (RPC | shared memory | some other IPC mechanism).

Yes, structured data exchange is the correct answer. When I have the opportunity to code something from scratch, this is the route I take.

Re: Introducing the Windows Pseudo Console (ConPty)

#109

Post by zvrba »

cryptonector wrote: post #52
tom_ wrote: post #26

Earlier quoted context omitted.

The lack of signals in Windows is the very opposite of a flaw! - Windows has just never pretended you can get away without a message loop.

Yes, of course signals are easily the worst thing in Unix, but job control is nice .

It's already there: https://docs.microsoft.com/en-us/windows/desktop/ProcThread/...

You can set various limits, though I haven't seen functions to stop/resume a job.

Re: Introducing the Windows Pseudo Console (ConPty)

#110

Post by paulie_a »

AnIdiotOnTheNet wrote: post #67
paulie_a wrote: post #42

Earlier quoted context omitted.

Is there a chance this will be connected to a functional shell interface. I get your point that cmd cannot be upgraded because of legacy issues and that is understandable and unfortunate, but windows needs a proper shell. This is obviously a great start for one side of the equation. But until there is a decent terminal app, windows will continue to be a nonstarter.

Windows has a proper shell, it's called PowerShell and it is by far the most discoverable and consistent shell that exists.

I've attempted to use powershell numerous times, it's a piece of shit compared to zsh.
Post reply on HN