How to save 32TB of traffic every week with one NPM package
Chokidar is a fast open-source file watcher for node.js. You give it a bunch of files, it watches them for changes and notifies you every time an old file is edited; or a new file is created.
I’ve created it in 2012, to mitigate differences between node file watching APIs such asfs.watch andfs.watchFile. In 2013, we’ve found a great way to improve MacOS experience by addingFSEvents support.
Since then, the package gained huge popularity. Build tools and all sorts of apps added Chokidar as their dependency. One of the most prominent examples of this isMicrosoft Visual Studio Code, a cross-platform source code editor.
- 11,707,948 NPM downloads every week. VS Code downloads are excluded from NPM, folks are getting it through its homepage.
- 2,421,435 GitHub repositories depend on Chokidar
- 4,483 NPM packages depend on Chokidar
Goals for v3: improve performance & UX
Chokidar 1 supported any node starting from 0.8. For version 3, we’ve decided to bump the requirement to node 8.x, which allowed to simplify architecture a lot. 8.x is the oldest node supported by node-core today, so this makes sense.
The main goal with v3 was to improve CPU + RAM usage and to reduce the number of dependencies.
- Switchingnode-fsevents toN-API, a new API for building native addons. In v2, we’ve been using node-gyp, which is buggy, sloppy, and produced useless errors while compiling.
- Rewritingreaddirp — another module of ours — to a stream API. Allowed to reduce memory consumption by a factor of 5 in some cases. By the way, Node 10+ allows using brand new
for-awaitsyntax for streams — totally great developer UX. - To handle globs, we’ve rewrittenpicomatch andbraces to use its own parsers instead of3-rd party dep. This allowed to drop dependency count significantly.
- For the remaining dependencies, we’ve walked through every package and removed all non-essential code from their NPM bundles. Which included tests, various fixtures.
package.json#filesfield helped to accomplish this — you simply specify files which need to be included there.
Every dependency is a potential security vulnerability
There is a sad trend in JS community to create packages that consume lots of dependencies. Django, a huge web-framework written in Python, uses only three deps right now. As a comparison, nest.js uses 41 dependency!
I want to emphasize that every dependency is a potential security vulnerability. Consider three cases:
- Reputable developer giving project rights to a rogue developer in a good faith(happened before).
- Reputable developer getting hacked and having a bad package published.
- Rogue developer creating new project, waiting for it to become popular and pushing a bad version after that(happened before).
With Chokidar 3, we were able to cut down a number of dependenciesfrom 201 to just 15! All dependencies which are maintained by folks that are not affiliated with chokidar have been removed.
To mitigate issue #2, Chokidar is also doing code signing right now. We’re using PGP keys to sign important commits & git tags.
Numbers
Let’s comparenpm install on a clean machine for v2 and v3:
Chokidar v2:
- Installation time: 7.753 sec
- Consumes 3.195mb of traffic (3.03mb down, 165kb up) and makes 46 network connections
- Creates 201 packages from 130 contributors and audits 2113 packages
node_modulesfile count: 1,251 (484 *.js)node_modulessize: 8.3Mb
Chokidar v3:
- Installation time: 1.575 sec(4.9x faster)
- Consumes 352Kb of traffic(9x less) (329kb down, 23kb up) and makes 16 network connections
- Creates 15 packages(13x less) from 20 contributors and audits 19 packages
node_modulesfile count: 79 (26 *.js)(15.8x less)node_modulessize: 496Kb(16.7x less)
In an ideal case, if every machine would have its caches clean, and NPM cleans them from time to time, using v3 would save us:
downloads=11707948per_pkg_bytes=3350200-360448terabyte=1024**4round(downloads*per_pkg_bytes/terabyte,2)# => 31.8432 Tb (245 bytes) of traffic is now going to be potentially saved every week!
In the end
It would be great if more software reduced its resource usage. We could improve user experience to so many people.
I look at you, a 103Mb iOS application.
At you, a web app with 950 NPM dependencies.
At you, an electron app with 1.2Gb RAM usage at startup.
And of course at you, a web page with 15 ad trackers.
Let’s work together on making our software swift, small & secure!
Thanks toPhilipp Dunkel,Jon Schlinkert and everyone else involved in releasing v3 out.
My next step is to create a set of highly auditable 0-dep cryptographic libraries for JS. I call itnoble-cryptography and the first packages are already out! Check it:noble-ed25519 &noble-secp256k1.