- Notifications
You must be signed in to change notification settings - Fork3k
Conversation
@zkat Will any of these flags become default? Some folks bump into addition flags but it doesn't reach everyone (example: I understand that it breaks expectations on how people think npm works right now (which is |
@siddharthkp There's no significant speed difference between I see you mentioned that you think users would expect Now, if you're looking for a speedup, the biggest one you can get with So, no, I probably won't use npm@5 to make |
😮 I hada conversation with@iarna a while back (npm 3) where she educated me about the network requests to the registry even when using cache + the effect of latency on it. (Significant for emerging nations - prominently 2G and latency heavy nations - Australia and such) Are you saying pacote nullifies that effect? (or at least makes it not noticeable) because that would be 💯
Ouch! My bad, I meant
This is another interesting part 🤔
💯 Is there a github issue that I can upvote for this? |
I'm on a Bay Area network, which is admittedly absurdly good. If the difference is so small now, though, I imagine it wouldn't be so big: even at 1/10th the speed, that would still be barely 10 seconds added (to a 60-second installation).
I really can't say that for sure right now. tbh, the difference is so surprisingly small that I think I'm gonna have to spend some time crawling through tests and making sure I don't have some bug. That code definitely has tests attached, though...
Ah. As far as this one goes, pacote obeys cache age settings from a given registry. This is generally happy news for all involved (if we hit the registry less without degrading user experience, that's a win!), and those settings tend to be fairly low: the main npm registry sets Iwould care more if it weren't for shrinkwrap being the main target of optimization, which brings me to...
Pretty much this. If we don't have the exact hashes for the tarballs we want to end up installing, npm needs to repeat the process of
Not quite yet. There'sa github project for npm@5 and one of my TODOs is to write a proposal for how shrinkwrap-by-default would work. From what I've talked with@iarna, though, we need to fix other stuff (like |
siddharthkp commentedApr 12, 2017
If shrinkwrap by default is likely to come in npm6, does it make sense to make |
Any reasons why yarn is so fast? I really expect npm@5 to be as fast as yarn. How can I help? |
siddharthkp commentedApr 12, 2017
@Kuroljov With npm-shrinkwrap, npm will be in serious competition with yarn. @zkat Can we start playing around with that somewhere? (Can contribute to your benchmarks?) |
@Kuroljov This release addresses a lot of performance issues. Right now, npm@5 installs are running anywhere between 2x-5x faster than Some sample benchmarks: |
@siddharthkp expect a beta release of npm@5 in the coming weeks. I can't give you the exact date, but I'm currently in the process of reviving npm's massive test suite after such a big change and fixing all my little messups along the way. What I can tell you is that if you makehttps://npm.im/make-fetch-happen andhttps://npm.im/cacache faster and better, it'll be directly reflected on the performance of npm@5. Oncehttps://npm.im/pacote gets its I'll make an announcement somewhere dev-oriented once |
@siddharthkp coming back to this: what are the network conditions where you're at? (since you asked). I'd like to get a better, more realistic perspective on how npm5 handles slow and/or unstable networks and special configurations like proxies, which is all fairly hard to simulate unless you've been deep in that situation, imo. There's now a (semi-closed) beta of npm5 that you can install with |
Previously we were doing this so we could load relationships, but now we cansimply request no fake children. This forces a fetch-metadata phase for thelockfile, but does not actually install all the modules.
This can happen when, for example, removing a module from your project thatisn't actually in your node_modules at the moment.
It used to be that we did this check in `is-extraneous` but it turns out tobe much cleaner to keep that pure. It let's us use `is-extraneous` forthings like pruning.
This was tagged and released yesterday as |
olegkorol commentedJun 12, 2017
Hi, what is the new format of Thanks in advance! |
NOTE: This PR is a WIP. npm@5 beta is currently underway. See#16510 for deets.
Wowowowowow npm@5!
This release marks months of hard work for the young, scrappy, and hungry CLI team, and includes some changes we've been hoping to do for literally years. npm@5 takes npm a pretty big step forward, significantly improving its performance in almost all common situations, fixing a bunch of old errors due to the architecture, and just generally making it more robust and fault-tolerant. It comes with changes to make life easier for people doing monorepos, for users who want consistency/security guarantees, and brings semver support to git dependencies. See below for all the deets!
Breaking Changes
One binary to rule them all:
./cli.jshas been removed in favor of./bin/npm-cli.js. In case you were doing something with./cli.jsitself. (Replace cli.js with bin/npm-cli.js and remove cli.js #12096)Stub file removed (utils: remove
lib/utils/rename.js#16204)Existing npm caches will no longer be used: you will have to redownload any cached packages. There is no tool or intention to reuse old caches. (Replace metadata + fetch + cache code #15666)
npm install ./packages/subdirwill now create a symlink instead of a regular installation.file://path/to/tarball.tgzwill not change -- only directories are symlinked. (RFC: file: specifier changes #15900)npm will now scold you if you capitalize its name. seriously it will fight you.
The "extremely legacy"
_tokencouchToken has been removed. (Remove_tokensupport altogether` #12986)npm will
--saveby default now. Additionally,package-lock.jsonwill be automatically created unless annpm-shrinkwrap.jsonexists. (Replace metadata + fetch + cache code #15666)git dependencies support semver (Resolve git dependencies with semver ranges #15308,Replace metadata + fetch + cache code #15666)
git dependencies with
preparescripts will have theirdevDependenciesinstalled, andnpm installrun in their directory before being packed.npm cachecommands have been rewritten and don't really work anything like they did before. (Replace metadata + fetch + cache code #15666)--cache-minand--cache-maxhave been deprecated. (Replace metadata + fetch + cache code #15666)running npm while offline will no longer insist on retrying network requests (Replace metadata + fetch + cache code #15666)
The contents of
~/.npmhave completely changed.A new loglevel,
notice, has been added and set as default.package locks no longer exclude
optionalDependenciesthat failed to build. This means package-lock.json and npm-shrinkwrap.json should now be cross-platform. (RFC: file: specifier changes #15900)shrinkwrap and package-lock no longer exclude
devDependencies. You can still prevent their installation with--prod/--only=prod. (RFC: file: specifier changes #15900)If you generated your package lock against registry A, and you switch to registry B, npm will now try toinstall the packages from registry B, instead of A. If you want to use different registries for different packages, use scope-specific registries (
npm config set @myscope:registry=https://myownregist.ry/packages/). Different registries for different unscoped packages are not supported anymore.shrinkwrap and package-lock no longer warn and exit without saving the lockfile
Local tarballs can now only be installed if they have a file extensions
.tar,.tar.gz, or.tgz.Feature Summary
There's a bunch of exciting stuff going in here! You can track all the changes planned for
npm@5onits project tracker.Installer changes
A new, standardised lockfile feature meant for cross-package-manager compatibility (
package-lock.json), and a new format and semantics for shrinkwrap. (spec: Describe npm-shrinkwrap.json and package-lock.json #16441)--saveis no longer necessary. All installs will be saved by default. You can prevent saving with--no-save. Installing optional and dev deps is unchanged: use-D/--save-devand-O/--save-optionalif you want them saved into those fields instead. Note that since npm@3, npm will automatically update npm-shrinkwrap.json when you save: this will also be true forpackage-lock.json. (Replace metadata + fetch + cache code #15666)Installing a package directory now ends up creating a symlink and does the Right Thing™ as far as saving to and installing from
npm-shrinkwrap.jsongoes. If you have a monorepo, this should make things much easier to work with, and probably a lot faster too. 😁 (RFC: file: specifier changes #15900)git dependencies now support semver-based matching:
npm install git://github.com/npm/npm#semver:^5(Resolve git dependencies with semver ranges #15308,Replace metadata + fetch + cache code #15666)git dependencies with
preparescripts will nowhave their devDependencies installed, and their prepare script executed as if undernpm pack.node-gyp now supports
node-gyp.cmdon Windows (Changenpm_config_node_gypto use shell script on Windows #14568)npm no longer blasts your screen with the whole installed tree. Instead, you'll see a summary report of the install that is much kinder on your shell real-estate. Specially for large projects. (Replace post install module list with summary report #15914):
--parseableand--jsonnow work more consistently across various commands, particularlyinstallandls.Indentation is nowdetected and preserved for
package.json,package-lock.json, andnpm-shrinkwrap.json. If the package lock is missing, it will default topackage.json's current indentation.Publishing
sha512andsha1checksums. Versions of npm from 5 onwards will use the strongest algorithm available to verify downloads. (feat(integrity): add integrity field to publish npm-registry-client#157)Cache Rewrite!
We've been talking about rewriting the cache for a loooong time. So here it is. Lots of exciting stuff ahead. The rewrite will also enable some exciting future features, but we'll talk about those when they're actually in the works.#15666 is the main PR for all these changes. Additional PRs/commits are linked inline.
package metadata, package download, and caching infrastructure replaced
It's a bit faster.Hopefully it will be noticeable. 🤔
With the shrinkwrap and package-lock changes, tarballs will be looked up in the cache by content address (and verified with it)
Corrupted cache entries willautomatically be removed and re-fetched on integrity check failure.
npm CLI now supports tarball hashes with any hash function supported by Node.js. That is, it willuse
sha512for tarballs from registries that send asha512checksum as the tarball hash. Publishing withsha512is added byfeat(integrity): add integrity field to publish npm-registry-client#157 and may be backfilled by the registry for older entries.remote tarball requests are now properly cached. This means that even if you're missing the
integrityfield in yournpm-shrinkwrap.json, npm will be able to install from the cache.Downloads for large packages are streamed in and out of disk. npm is now able to install packages of """any""" size without running out of memory. Support for publishing them is pending (due to registry limitations).
Automatic fallback-to-offline mode. npm will seamlessly use your cache if you are offline, or if you lose access to a particular registry (for example, if you can no longer access a private npm repo, or if your git host is unavailable).
A new
--prefer-offlineoption will make npm skip any conditional requests (304 checks) for stale cache data, andonly hit the network if something is missing from the cache.A new
--prefer-onlineoption that will force npm to revalidate cached data (with 304 checks), ignoring any staleness checks, and refreshing the cache with revalidated, fresh data.A new
--offlineoption will force npm to use the cache or exit. It will error with anENOTCACHEDcode if anything it tries to install isn't already in the cache.A new
npm cache verifycommand that will garbage collect your cache, reducing disk usage for things you don't need (-handwave-), and will do full integrity verification on both the index and the content. This will also be hooked intonpm doctoras part of its larger suite of checking tools.The new cache isvery fault tolerant and supports concurrent access.
npm cache clearis no longer useful for anything except clearing up disk space.Package metadata is cached separately per registry and package type: you can't have package name conflicts between locally-installed packages, private repo packages, and public repo packages. Identical tarball data will still be shared/deduplicated as long as their hashes match.
HTTP cache-related headers and features are "fully" (lol) supported for both metadata and tarball requests -- if you have your own registry, you can define your own cache settings the CLI will obey!
prepublishOnlynow runsbefore the tarball to publish is created, afterpreparehas run....and more!
For release task list
prepareon git dependency installationpackage-lock.jsonvsshrinkwrapcreated-with,shrinkwrap-version,package-integrity,npm updatesave files as right type: devDeps end up getting saved as depsdiff-treesbut they do end up in the lock.npm install --productiondoesn't remove dev deps from the lock.noticeandtiminglog levels directly to npmlogpreparewhen installing git dependencies.