Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Introducing the Windows Pseudo Console (ConPty)

blogs.msdn.microsoft.com

Post reply on HN
Print view View original

141–150 of 196 posts

Re: Introducing the Windows Pseudo Console (ConPty)

#141

Post by int_19h »

zadjii wrote: post #32
rogerbinns wrote: post #12

Earlier quoted context omitted.

Have you given thought on how to solve the "unwanted console" problem? For example if you run a .py file (Python) under Windows then you get a console. That is fine for command line stuff, but beyond annoying if the file displays as a gui. So there are now two Python binaries - python.exe and pythonw.exe. The only difference is the latter ensures no console appears. Also good luck if the script printed a help message…

Technically, any executable that's compiled as a commandline application is going to get a console allocated for it, no matter what on Windows. I don't believe that's something we can fix retroactively unfortunately, that's just a part of how things have to be. Now, I believe that python could have python.exe compiled as a win32 application, then call AllocateConsole as soon as the script called print() or something.…

If I remember correctly, this approach breaks if stdin/out is redirected.

There's also some funky stuff about explicit AllocConsole-allocated consoles; for example, when you attach a native debugger, all output from such console is automatically redirected to that debugger (i.e. the VS Output window or similar). This is very annoying in practice.

Re: Introducing the Windows Pseudo Console (ConPty)

#142

Post by AnIdiotOnTheNet »

wtallis wrote: post #112
AnIdiotOnTheNet wrote: post #90

Earlier quoted context omitted.

That's subjective opinion. I much prefer the Windows way because I can either specify additional characters or keep hitting tab to until what I want comes up. "Display all 1026 possibilities? (y or n)" certainly isn't an improvement. Not to mention of course that it tab completes things bash can't and doesn't, and all the other reasons your initial comment was wrong.

> I much prefer the Windows way because I can either specify additional characters or keep hitting tab to until what I want comes up. You can't specify additional characters until after you've erased all of the incorrect trailing characters that powershell filled in, and that's where powershell's completion method runs up the keystroke count unreasonably. Your comparison isn't valid if you ignore that aspect. It's al…

> you appear to have only cited the existence of concise aliases for some commands as a refutation

You claim that PowerShell is overly verbose, I point out that it is only verbose if you intentionally make it verbose for readability, you proceed to pick nits and pretend you've done some elaborate study on keystroke counts or something, then resort to tired old Linux evangelism mainstays like blaming the distro.

Re: Introducing the Windows Pseudo Console (ConPty)

#143

Post by JdeBP »

exikyut wrote: post #86
zadjii wrote: post #32

Earlier quoted context omitted.

Technically, any executable that's compiled as a commandline application is going to get a console allocated for it, no matter what on Windows. I don't believe that's something we can fix retroactively unfortunately, that's just a part of how things have to be. Now, I believe that python could have python.exe compiled as a win32 application, then call AllocateConsole as soon as the script called print() or something.…

Hmmm. MSDN doesn't really say much about AllocConsole(), if that's the right function: https://docs.microsoft.com/en-us/windows/console/allocconsol... If AllocConsole does behave in the way you say (which I understand it may{, not}), then the documentation sorely needs updating, because right now that bit of functionality (if it is there) is rather implicit. It would be really cool to effectively deprecate the curren…

> It would be really cool to effectively deprecate the current console functionality and make it relatively straightforward to use the PTY API going forward,

Please no!

The console subsystem in Windows is what terminal I/O evolved into, in the 1980s. Going back to terminal I/O is a massively retrograde step.

In any case, for addressing the problem at hand removing the console API is not the answer. rossy has explained what the problems actually are, which relate to the command interpeter waiting for processes to terminate and whether Win32 program image files are marked as GUI or console.

Personally I have always regarded that behaviour of Microsoft's command interpreter as a flaw, not a feature. I've always turned it off in JP Software command interpreters, which make it configurable. I didn't implement it in my command interpreter. However, I do appreciate that Microsoft's strong commitment to backwards compatibility hampers what can be done.

* https://jpsoft.com./help/waiting.htm

Re: Introducing the Windows Pseudo Console (ConPty)

#144

Post by JdeBP »

zadjii wrote: post #48
berbec wrote: post #17

Earlier quoted context omitted.

Will there be the ability to disown, background, nohup processes and close the console, leaving the commands running?

Those sound like they'll be more like the responsibility of the terminal emulator, unfortunately. Windows console applications aren't really able to live without being attached to a console. Now, a terminal might be able to implement those features... actually now you've got me thinking. I'll play around with that idea. Definitely non-committal, but it might be possible in the future.

The sad thing is that this was all already implemented and done in Microsoft's second POSIX subsystem for Windows NT. It provided signals and process groups support for job control shells. It had a full control sequence interpreter for output and control sequence generator for extended keys. There were termcap/terminfo database records that people had added to other operating systems. It had a line discipline with "canonical" and "raw" modes. It had pseudo-terminals, with both BSD and System 5 access semantics.

* https://technet.microsoft.com/en-gb/library/bb497016.aspx

* https://technet.microsoft.com/en-gb/library/bb463219.aspx

* https://news.ycombinator.com/item?id=12866843

* http://jdebp.info./FGA/interix-terminal-type.html

And Microsoft owns it.

Re: Introducing the Windows Pseudo Console (ConPty)

#145

Post by JdeBP »

caf wrote: post #57
zadjii wrote: post #28

Earlier quoted context omitted.

Excellent question! There are a few limitations that we have to place on the ConPty to make it work quite right. Primarily, client apps running attached to the conpty will not be able to have separate viewport and buffer sizes. On *nix, the entire "console buffer" is just the size of the window, but on Windows, technically, the buffer is much larger than the window. (as an example, when you open up a command prompt,…

*nix consoles typically have two buffers - many full-screen temrinal applications switch to the "alternate screen" on start and back to the principal screen on exit. That's why when you exit vim(1), you see the terminal state back as it was before you started it. Will ConPty support this?

The Windows console subsystem itself has supported multiple output buffers from the start.

* https://docs.microsoft.com/en-gb/windows/console/console-scr...

Re: Introducing the Windows Pseudo Console (ConPty)

#146

Post by sime2009 »

joeyaiello wrote: post #16
zokier wrote: post #10

Earlier quoted context omitted.

While it is nice that MS is focusing on console and command line now, it seems to me that you are mostly working on improving compatibility with legacy UNIXy stuff. Do you have some vision or plans to go well beyond the classic UNIXy style of console and command line? I'm thinking in the lines of projects like DomTerm http://domterm.org/ which could have nice interactions with e.g. PowerShell.

PM for PowerShell here! I haven't seen DomTerm before, but it looks pretty awesome. At a glance, it's basically a GUI-fied tmux hosted in Electron? It would be awesome to have in Windows, but wouldn't that just require that DomTerm add support for these ConPty APIs? In any case, I'm more interested in your proposed interactions. Did you have anything cool in mind? Given that we ship PowerShell on Linux, we could theo…

I'm not the person you were asking, but this should interest you.

I've been working on a terminal emulator ( Extraterm http://extraterm.org/ ) with some novel features which would dovetail nicely with how PowerShell works. The first is the ability to send files to the terminal where they can be displayed as text, images, sound, etc or as an opaque download. Extraterm also adds a command `from` which lets you use previous terminal output or files, as input in a command pipeline. See http://extraterm.org/features.html "Reusing Command Output" for a demo. This opens up other, more interactive and iterative workflows. For example, you could show some initial data and then in later commands filter and refine it while visually checking the intermediate results.

What I would like to do sometime is integrate this idea with PowerShell and its approach of processing objects instead of "streams of bytes". It should then be possible to display a PowerShell list of objects directly in the terminal, and then reuse that list in a different command while preserving the "objectness" of the data. For example, you could show a list of user objects in one tab and then in another tab (possibly a different machine) grab that list and filter it the same way as any normal list of objects in PowerShell. You could also directly show tabular data in the terminal, let the user edit it "in place" in the terminal, and then use that editted data in a new command. It allows for more hybrid and interactive workflows in the terminal while still remaining centered around the command line.

Extraterm does these features using extra (custom) vt escape codes. ConPty should allow me to extend these features to Windows too.

Re: Introducing the Windows Pseudo Console (ConPty)

#147

Post by mehrdadn »

JdeBP wrote: post #136
mehrdadn wrote: post #105

Earlier quoted context omitted.

Where are you getting this information though? I haven't worked out the bits myself yet but Wikipedia's first sentence itself says UTF-16 can encode all 1,112,064 valid code points of Unicode, which is already more than 2^(10+10) = 1,048,576.

Unicode code point space: Was 16-bit (0000 to FFFF), then 32-bit (00000000 to FFFFFFFF), and is now 21-bit (00000000 to 0010FFFF) UTF-16: Encodes the entire 21-bit range, encoding most of the first 0000 to FFFF range as-is, and using surrogate pairs in that range to encode 00010000 to 0010FFFF. The latter range is shifted to 00000000 to 000FFFFF before encoding, which can be encoded in the 20 bits that surrogate pair…

I don't see anything wrong with what you're saying, but I still don't get how it explains the original comment I replied to [1]:

> I'm not at all convinced that 2^21 codepoints will be enough, so someday it'd be nice to be able to get past UTF-16 and move to UTF-8

UTF-16 currently uses up to 2 16-bit code units per code point, whereas UTF-8 uses up to 4 8-bit code units per code point, and the latter wastes more bits for continuation than the former. How is "getting past UTF-16 and moving to UTF-8" supposed to increase the number of code points we can represent, as claimed above? If anything, UTF-16 wastes fewer bits in the current maximum number of code units, so it should have more room for expansion without increasing the number of code units.

[1] https://news.ycombinator.com/item?id=17771351

Re: Introducing the Windows Pseudo Console (ConPty)

#148

Post by JdeBP »

oblio wrote: post #122
kpil wrote: post #111

-- Those who don't understand Unix are condemned to reinvent it, poorly. Henry Spencer

This is such a cliché. Do you think that the people who implemented the Windows Console, especially the people working on Windows NT, did not know about Unix? People try different approaches, sometimes they don't work out. And it's not like Unix is the Word of God, anyway, it has plenty of flaws. (Yeah, after a long time on internet forums I get kind of touchy after someone copy-pastes the same old and tired line.)

The problem is that in this case it's not understanding Microsoft's own prior software that condemns one to reinvent it. Microsoft's second POSIX subsystem for Windows NT, a.k.a. Interix, had all of this.

Re: Introducing the Windows Pseudo Console (ConPty)

#149

Post by AnIdiotOnTheNet »

speeder wrote: post #95
mehrdadn wrote: post #83

Earlier quoted context omitted.

Confused, where does 2^21 code points come from and how is that related to the UTF-16 vs. UTF-8 distinction? Can't both of them encode all Unicode code points? Or are you thinking of code units perhaps, and UCS-2? Although even there I'm confused where the 2^21 came from.

Utf-16 is always 16 bit, utf-8 is variable, can go from 8 to 32 as needed... This does make coding for utf-8 harder, but when it works is really wonderful stuff.

UTF-16 is not always 16b/codepoint. UCS-2, its ancestor and more or less what is actually used by Windows when it talks about "Unicode", was 16b/codepoint. It turned out that 64k codepoints wasn't enough for everybody, so they increased the range and added "surrogate pairs" to UCS-2 to make UTF-16. Codepoints in UTF-16 are, therefore, either 16b/codepoint or 32b/codepoint. Many systems that adopted UCS-2 claim UTF-16 compatability but in fact allow unpaired surrogate pairs (an error in UTF-16). The necessity of encoding such invalid UTF-16 strings in UTF-8 under certain circumstances lead to the pseudo-standard known as WTF-8[0].

UTF-8 was designed (as legend has it, on the back of a napkin during a dinner break) after the increase in range and doesn't suffer from the same problem. Additionally, it's straight-forward "continuation" mechanism isn't any more difficult to deal with than surrogate pairs, and it doesn't have any endianess issues like UTF-16/UCS-2.

[0]https://simonsapin.github.io/wtf-8/

Re: Introducing the Windows Pseudo Console (ConPty)

#150

Post by JdeBP »

mehrdadn wrote: post #147
JdeBP wrote: post #136

Earlier quoted context omitted.

Unicode code point space: Was 16-bit (0000 to FFFF), then 32-bit (00000000 to FFFFFFFF), and is now 21-bit (00000000 to 0010FFFF) UTF-16: Encodes the entire 21-bit range, encoding most of the first 0000 to FFFF range as-is, and using surrogate pairs in that range to encode 00010000 to 0010FFFF. The latter range is shifted to 00000000 to 000FFFFF before encoding, which can be encoded in the 20 bits that surrogate pair…

I don't see anything wrong with what you're saying, but I still don't get how it explains the original comment I replied to [1]: > I'm not at all convinced that 2^21 codepoints will be enough, so someday it'd be nice to be able to get past UTF-16 and move to UTF-8 UTF-16 currently uses up to 2 16-bit code units per code point, whereas UTF-8 uses up to 4 8-bit code units per code point, and the latter wastes more bits…

If I had intended to explain that, I wouldn't be replying to your comment about your not working out the bits with an explanation that works out the bits for you, and shows that it really is capable of encoding all 17 planes even though surrogate pairs have only 20 bits.

And as you can see, if you do work out the bits, you find that cryptonector is wrong, since UTF-8 (as it has been standardized from almost the start of the 21st century, and as codecs in the real world have taken to implementing it since) encodes no more bits than UTF-16 does. It's 21 bits for both.

Post reply on HN