linux.conf.au
5 HN recommenders · recommended in 9 months, 2020–2025
48:04 · 161K views · Jan 16, 2020
Robert Collins explains why rustup installs were painfully slow on Windows and how changes to rustup and Windows cut install time from 3m30s to 14s. The talk covers NTFS, small files and the cost of antivirus hooks on file close.
The go-to reference for why file operations are slow on Windows and how antivirus filters, not NTFS itself, are the cause.
12 comments from Hacker News
NTFS can be especially awful if you're used to Linux behaviour, port code over to Windows and expect the same performance. Here's a talk on porting rustup to Windows: this video To begin with, it takes rustup 3m30s to install on Windows. After rejigging the code again and again away from its naive implementation which works fine on Linux, to perform sympathetically towards NTFS, it takes the same rustup 14s to install. That's quite a performance gain! But it needed a lot of changes to rustup, and to Windows itself.
Add your npm directories to the virus file exclusion lists of Windows Defender. Note that some of the directories may not be obvious (those under AppData for example). The speed improvements are massive for tools like these that touch lots of small files. Also take a look at this talk on NTFS, file system speed, and how the presenter optimized the Rust installer: this video . There is an awful lot that can be done to tools that deal with lots of files to perform well under Windows too.
And as a result of this, if you want to optimize your program to be fast on Windows, you will make very different optimization decisions than on other platforms. For example, Windows runs virus scans on close(), which makes it very slow. This means that sometimes it makes sense to have one or more background threads exclusively dedicated to closing files. There's a good talk on this at this video
I want to take this opportunity to recommend the talk "NTFS isn't that bad" ( this video ). NTFS prefers a different access pattern than most usual file systems. I remember that a part of the talk was about speed-ups on Linux as well. So even if it doesn't sway your opinion it should enhance your perspective on how file systems work.
That may be due to a combination of Malware detection + most unix programs not really written to take advantage of the features NTFS has to offer This is a great talk on the topic this video