Not sure exactly what the situation was here, but if it was an NTFS folder from Windows shared into the VM/WSL, then you are indeed running the npm operation in the inner system but paying the cost of writing thousands of tiny files across that translation layer.
Hmm. I thought the problem was that NTFS is not particularly fast at file creation. Generally not a problem, except for the folks who wrote the npm tooling which generates 10,000 files in the blink of an eye.
Most of the performance issues are not NTFS so much as the prevalence of file-system filter drivers. This is why WSL had so many file-system performance issues despite the fact that it was using a custom FS. IIRC this also motivated the move to WSL2 from what I read, since a virtual machine can use a small set of files to store a whole FS.
If you deal with many small files on Windows, you will observe much better performances in WSL2/ext4 than windows with NTFS. Even though the ext4 filesystem is stored as a file inside a NTFS partition.
If you use NodeJS for example, it’s a very obvious speed difference.
Most compilation processes create a lot of very small files. Or a metric ton of them if you're unlucky enough to do js.
NTFS is awful at doing lots of small accesses on very small files, at least compared to Ext4. Add to that the fact that if you're on WSL, it needs to write POSIX attributes (which NTFS has no support for and therefore WSL must emulate), and that's a recipe for builds being slowed massively by IO.
As for Defender, well, Defender sees new files and halts everything while it scans them. You can see how that can be a problem for a process that pops out tens of thousands of temporary files, which it will all go and scan, locking them while it does so.
To clarify the "it" here, NTFS allows you to create a file named `aux.js` just fine. The thing that injects special handling of `aux.js` is the Windows object layer that sits on top of it.
will create a file named `aux.js` just fine, since it tells the Windows layer to not normalize names, which includes disabling the special handling of files named `aux`.
If your issues were disk speed related, you should know that accessing files in NTFS using WSL2 is very slow. You'll have to move your files to the Linux FS first.
I/O is the biggest issue on NTFS (which is why we use WSL for most of our node work - lots of small files to futz with); how about native (kernel) support for ext4 or zfs? That would be sweet.
This one? I remember it as an earnest description of the difficulties the WSL team had with the speed of NTFS - and I think it was one of the reasons for the switch to virtualisation in WSL2.
So if accessing ntfs from wsl is now slow, you can put the files in wsl instead. The problem then, if you're using this as a dev machine, is how do you edit them in Windows? I want to use my JetBrains IDEs to edit wsl files, if that doesn't work I'd just stick with dual boot.
Well, there is a little bit of emulation. We have to encode and handle Linux-style file permissions, which has some overhead.
We are working on making both the WSL layer and NTFS itself better. For example, in the upcoming Windows release, we have created a fast path for stat that should yield a modest speed improvement.
I think this is because WSL uses NTFS Alternate data streams to store linux file attributes, and they don't survive when a file is copied to a non-NTFS file system.
My issue is across the whole windows platform not just WSL. If you check large repos out on windows native it sucks too. Rather than just fixing WSL and declaring victory they should fix the NTFS problem.
NTFS also has a 260 character path limit [0], which as the sibling comment says can really jam up node_modules and the current HEAD of the intellij-community repo which currently has some giant filename in a subdirectory of its repo
I'm aware of their claim about using Group Policy to remove the limit but I've never used GP in order to know its sharp edges for my gaming computer
reply