Windows Subsystem for Linux: The lost potential • Hacker News Board

Re: Windows Subsystem for Linux: The lost potential

#181

Post by m_mueller » Fri, Nov 20, 2020, 8:43 AM UTC

for_xyz wrote: post #179

wahern wrote: post #174

Earlier quoted context omitted.

Your point about the dentry cache seems like a non-sequitur. The cache isn't the bottleneck; the bottleneck is that NTFS is a crappy filesystem. Yes, I've read this comment: https://github.com/microsoft/WSL/issues/873#issuecomment-425... The argument about Windows not having a single, central dentry cache doesn't hold water. For one thing, there's no reason to think that a two-level cache would significantly decrease…

> the bottleneck is that NTFS is a crappy filesystem. Care to explain why do you think NTFS is crap? From my experience it's usually bad assumptions about files under windows and every application or library tries to stick to posix interface when dealing with files (open, read/write, close) which tends to block for longer periods on windows than on linux counterparts which results in significant perfomance loss . Lin…

so what's the 'Windows way' then? keeping a file open forever and let it lock out other programs from using that file? I'm starting to get an idea why Windows is getting on my nerves so much with file locks....

Re: Windows Subsystem for Linux: The lost potential

#182

Post by jmnicolas » Fri, Nov 20, 2020, 8:47 AM UTC

for_xyz wrote: post #179

wahern wrote: post #174

Earlier quoted context omitted.

Your point about the dentry cache seems like a non-sequitur. The cache isn't the bottleneck; the bottleneck is that NTFS is a crappy filesystem. Yes, I've read this comment: https://github.com/microsoft/WSL/issues/873#issuecomment-425... The argument about Windows not having a single, central dentry cache doesn't hold water. For one thing, there's no reason to think that a two-level cache would significantly decrease…

> the bottleneck is that NTFS is a crappy filesystem. Care to explain why do you think NTFS is crap? From my experience it's usually bad assumptions about files under windows and every application or library tries to stick to posix interface when dealing with files (open, read/write, close) which tends to block for longer periods on windows than on linux counterparts which results in significant perfomance loss . Lin…

> Care to explain why do you think NTFS is crap?

I don't know if it's crap but it's much much slower than EXT4.

I remember reading a comment here that Windows in a VM on a Linux host was faster than bare metal.

Probably not true but I decided to make a test. I have a .net core app that insert data in a sqlite db (the resulting db is about 300 GB).

So I benchmarked this app on Linux (it was previously running on Windows) and IIRC it ran about 4 times faster.

Re: Windows Subsystem for Linux: The lost potential

#183

Post by ralph87 » Fri, Nov 20, 2020, 8:51 AM UTC

wahern wrote: post #174

ralph87 wrote: post #9

Contra opinion: the VM option reduced the service interface between Windows and Linux to a single kernel implementation and a few drivers, rather than every possible userspace program ever written. It's an amazing and obvious trade off. My inner architecture astronaut appreciates all the ideas in this post, but I've been trying to kill that guy for over a decade now. The bottom line is WSLv1 design SUCKED precisely b…

Your point about the dentry cache seems like a non-sequitur. The cache isn't the bottleneck; the bottleneck is that NTFS is a crappy filesystem. Yes, I've read this comment: https://github.com/microsoft/WSL/issues/873#issuecomment-425... The argument about Windows not having a single, central dentry cache doesn't hold water. For one thing, there's no reason to think that a two-level cache would significantly decrease…

> The argument about Windows not having a single, central dentry cache doesn't hold water

This was more about cache coherency, and it's not an argument, it's a trait shared by every other Linux-over-X implementation (UML, coLinux). It is fundamental to solving a problem users want - seamless integration, i.e. no opaque ext4 blob. Why doubt the reasons given by Microsoft, when they match observations not just for WSL but every other system in the same class?

Re: Windows Subsystem for Linux: The lost potential

#184

Post by mehrdadn » Fri, Nov 20, 2020, 8:59 AM UTC

wahern wrote: post #174

ralph87 wrote: post #9

Contra opinion: the VM option reduced the service interface between Windows and Linux to a single kernel implementation and a few drivers, rather than every possible userspace program ever written. It's an amazing and obvious trade off. My inner architecture astronaut appreciates all the ideas in this post, but I've been trying to kill that guy for over a decade now. The bottom line is WSLv1 design SUCKED precisely b…

Your point about the dentry cache seems like a non-sequitur. The cache isn't the bottleneck; the bottleneck is that NTFS is a crappy filesystem. Yes, I've read this comment: https://github.com/microsoft/WSL/issues/873#issuecomment-425... The argument about Windows not having a single, central dentry cache doesn't hold water. For one thing, there's no reason to think that a two-level cache would significantly decrease…

This is not due to NTFS. (And NTFS is not crap either.)

Try it on another file system if you don't believe me.

Re: Windows Subsystem for Linux: The lost potential

#185

Post by mehrdadn » Fri, Nov 20, 2020, 9:00 AM UTC

m_mueller wrote: post #181

for_xyz wrote: post #179

Earlier quoted context omitted.

> the bottleneck is that NTFS is a crappy filesystem. Care to explain why do you think NTFS is crap? From my experience it's usually bad assumptions about files under windows and every application or library tries to stick to posix interface when dealing with files (open, read/write, close) which tends to block for longer periods on windows than on linux counterparts which results in significant perfomance loss . Lin…

so what's the 'Windows way' then? keeping a file open forever and let it lock out other programs from using that file? I'm starting to get an idea why Windows is getting on my nerves so much with file locks....

You can just use FILE_SHARE_WRITE if you don't want to lock out other programs...

Re: Windows Subsystem for Linux: The lost potential

#186

Post by dataflow » Fri, Nov 20, 2020, 9:03 AM UTC

spondyl wrote: post #127

mdoms wrote: post #122

This article is entirely wrong about why it's called the Windows Subsystem for Linux and not the Linux Subsystem for Windows (not surprising, since his argument makes no sense). The real reason is that Microsoft can't name something leading with a trademark owned by someone else. Here it is straight from the horse's mouth, senior PM of WSL Rich Turner: https://twitter.com/richturn_ms/status/1245481405947076610

You can see a similar comment from Rich Turner here on Hacker News too, under the username bitcrazed: https://news.ycombinator.com/item?id=13697918 I'd never actually read it with the apostrophe as pointing out in that tweet so the name makes a lot more sense now. Perhaps Windows Subsystem for (running) Linux would be clearer hah Also, according to his profile, Rich is former Product Manager for WSL and currently "Sr…

They could've called it the Windows Linux Subsystem and done away with the confusion.

Re: Windows Subsystem for Linux: The lost potential

#187

Post by acqq » Fri, Nov 20, 2020, 9:07 AM UTC

mehrdadn wrote: post #184

wahern wrote: post #174

Earlier quoted context omitted.

Your point about the dentry cache seems like a non-sequitur. The cache isn't the bottleneck; the bottleneck is that NTFS is a crappy filesystem. Yes, I've read this comment: https://github.com/microsoft/WSL/issues/873#issuecomment-425... The argument about Windows not having a single, central dentry cache doesn't hold water. For one thing, there's no reason to think that a two-level cache would significantly decrease…

This is not due to NTFS. (And NTFS is not crap either.) Try it on another file system if you don't believe me.

NTFS on Windows was indeed always significantly slower when doing a lot of file operations. It was noticeable even long before WSL existed: NTFS and the whole infrastructure connected to it has by design bigger maintenance cost for every opening of the file, for example. That's undeniable. One can say that NTFS could have maybe been implemented faster on some other system (with less hooks etc), but NTFS on Windows is provably slower for the file operations people do every day on Linux, that's a simple fact. Like working with big source repositories containing a lot of files (git, node.js). The people who decided to make WSL2 faster for Linux programs achieved the speedup by going full VM. I would also prefer that they worked on NTFS on Windows long enough until the file operations become faster for everybody, but it didn't happen.

That's one of the lost opportunities that the original article laments about.

Edit: a reply to dataflow’s "try your operations on another file system" answer: I don't have to, I remember: there was a time when people used FAT formatted media much more than today -- and NTFS was slower than FAT too, under the same Windows, on the same hardware, without installing anything from third parties. It can be that on Windows 10 FAT got to be comparably slow, but in earlier times, on slower machines, FAT was visibly faster compared to NTFS. But NTFS was more robust to failures, and I preferred using NTFS for anything that is not temporary. I admit that the whole Windows infrastructure around what we consider purely "a filesystem" is also a serious part of the problem, by design. And as I wrote, it's a lost opportunity that the bottlenecks weren't identified and the changes made to the benefit of everything on some future Windows.

Re: Windows Subsystem for Linux: The lost potential

#188

Post by mehrdadn » Fri, Nov 20, 2020, 9:08 AM UTC

rbanffy wrote: post #26

hapless wrote: post #22

Earlier quoted context omitted.

The original POSIX subsystem was not very useful -- a minimal set of system calls, plus a Microsoft re-branded MKS Toolkit to provide shells and utilities. Microsoft ended up buying Interix, who offered a much more pleasant UNIX-on-Windows setup. That was supported for many years as "SFU" and later "SUA" SUA was still a pretty weird UNIX though. PE-COFF binaries, and a linker that wasn't 100% GNU compatible, and a lo…

On XP and 7 (and 8, perhaps) I used Cygwin to have a Unix-like environment. Does it still work on Windows 10? I remember I could call Windows programs from it and manipulate Windows files.

I believe it does, but you might like MSYS2 more.

Re: Windows Subsystem for Linux: The lost potential

#189

Post by mehrdadn » Fri, Nov 20, 2020, 9:13 AM UTC

acqq wrote: post #187

mehrdadn wrote: post #184

Earlier quoted context omitted.

This is not due to NTFS. (And NTFS is not crap either.) Try it on another file system if you don't believe me.

NTFS on Windows was indeed always significantly slower when doing a lot of file operations. It was noticeable even long before WSL existed: NTFS and the whole infrastructure connected to it has by design bigger maintenance cost for every opening of the file, for example. That's undeniable. One can say that NTFS could have maybe been implemented faster on some other system (with less hooks etc), but NTFS on Windows is…

Again: try your operations on another file system, then tell me it's NTFS that's slow.

It's not NTFS that's slow at opening files. It's the I/O subsystem. You'll see the slowness with other file systems too.

Re: Windows Subsystem for Linux: The lost potential

#190

Post by TeMPOraL » Fri, Nov 20, 2020, 9:22 AM UTC

jasonkester wrote: post #175

alerighi wrote: post #106

Earlier quoted context omitted.

Correct, but... this is not WSL. This is a VM. Let's see this from the opposite point of view: WSL1 is to WSL2 what WINE is to a VM running Windows. Two completely different approaches. The thing that would have allowed the WSL1 to do was to really integrate between Windows and Linux: imagine writing running a Linux command, piping its output in a Windows command to pipe it again in a Linux command. Imagine sending a…

imagine writing running a Linux command, piping its output in a Windows command to pipe it again in a Linux command. But that's not most of us want to do with WSL. We just want to run the team's React app with its rube goldberg npm monstrosity that falls over when you try to run it on windows. That use case covers 90% of us. No need to ask for anything more.

This is how our industry goes to crap. Repeatedly pruning things that 90% of the current userbase doesn't use over time give you a thoroughly useless product. I mean, do it 7 times, and you've already lost 50% of your audience. If you target 80% cutoff, you're down to 20% of the users now.

I personally do want it from WSL. Hell, 90% of my use of WSL is like this - I run Emacs on my work machine under WSL, where it feels at home. I use it to drive Windows-side cmake and Visual Studio compiler, piping output back to Emacs. Because I can, it works, and it's a much nicer working experience than any app could give me.