Hey mate 👋. Have you seen ayarn.lock file? It can bepackage-lock.json,pnpm.lock ornpm-shrinkwrap, it does not matter that much. What matters - this file have a purpose.

package-lock.json is automatically generated for any operations where npm modifies either the node_modules tree, or package.json. It describes the exact tree that was generated, such that subsequent installs are able to generate identical trees, regardless of intermediate dependency updates.
–A manifestation of the manifest

In short - lock files represents and store information about dependencies other than your direct ones, and particular version of such indirect dependency to use.

There is no time to explain everything in the details, let me show you the problem.

Chapter 1 - understanding the problem

Lets create an issue

Many of us used lock files for years and used them without any troubles. Everything seems good, but there is one nuance we need to discover. A little feature we are going to reveal. Let's run an experiment.
I will useyarn to demonstration purposes.

The experiment itself is fordemonstration purposes only. Just blindly follow the steps or magic might not happen

• Step 1. Create empty repository.
  • yarn init and "install" it -yarn
  • you should get an emptyyarn.lock file and emptynode_modules folder
• Step 2. Let's add first dependency. Let it betslib@2.0.3
  • yarn add tslib@2.0.3
  • that would create a record inyarn.lock like

tslib@2.0.3:version "2.0.3"resolved "https://something-something/tslib-2.0.3.tgz"integrity sha512-long-and-ugly==

• Step 3. Let's massage our environment andmanually amendpackage.json andyarn.lock to usetslib:^2.0.3
  • inpackage.json:"tslib": "^2.0.3"
  • inyarn.lock:tslib@^2.0.3
  • then runyarn
  • ⚠️ it is important to have the following record inyarn.lock, which means "tslib@2.0.3 used for tslib:^2.0.3"
    • 🤔 the same will happen if 2.0.3 is the latest version available. Well, it's not the latestnow, but was in the past
    • 💡 pleaseremember this moment as adirect edit of the lock file

tslib@^2.0.3:version "2.0.3"

• Step 4. Add anotherdependency depending ontslib
  • yarn add focus-lock (v0.11.2). This package dependstslib:^2.0.3😉. Perfect match.

let's verify - ouryarn.lock should containonly two dependencies -focus-lock andtslib

Sothat's it. We have created a problem 😎. You don't see it? Yep, the real issue is that nobody sees this problem. But it's already here.

The problem is - we have two dependencies "perfectly" matching each other.

Is it a problem?

😉 Let's perform one more experiment.

• Step 1: removetslib
  • yarn remove tslib
• Step 2: check youryarn.lock, and it will beunchanged, as tslib is still used byfocus-lock
• Step 3: shrug 🤷‍♂️
• Step 4: removeyarn.lock
  • runrm ./yarn.lock
• Step 5: reinstall packages
  • runyarn

Now check your lock file 😉. It will contain the sametslib:^2.0.3, but this time it will beresolved to 2.4.0 (the last version available by the time of writing this article)

tslib@^2.0.3:version "2.4.0"

You might not get the joke, let me rephrase it

😳 Existing dependencies and the order of their addition affects the shape of the end artefact 😬

• In the first experiment focus-lockreused preexistingtslib
• In the second one there will no such constrain and it used the "freshest" one.

Read it another way - anynew dependency you will try reuse what you already have. Any dependency you update will try to reuse what you already have. Your's legacy, your's past, your's yesterday affects your tomorrow.

As an example - nextjs installed today andthe same version of nextjs installed tomorrow can have at least different version ofcaniuse, as it released on a daily schedule.

But that is just a half of the problem. The real one - you can still haveunexpectedly obsolete packages and hear me out -focus-lock by itself is using and is being tested withtslib 2.3.1, declaring a version below as a dependency because "it does not need any newer". Or needs, but have no idea about it, because cannot test itself agains other versions of own dependencies.

Yep, this is exactly how I broke someone one's project -import error: '__spreadArray' is not exported from 'tslib'

Another example maybe?

The example above was a little synthetic. Ok, VERY synthetic. Frankly speaking it took a while for me to find a way of reproducing such behaviour in a short and sound way, asusually everything was working well. Withoutdirect edit hack, the one I've asked you to remember, issue will be not created.
But what would be different?

Let's redo everything from scratch, but skip the hack. Depending on particular actions and events it can result in

tslib@^2.0.3, tslib@^2.4.0:version "2.4.0"

orTWO versions oftslib, which can becollapsed into one viadeduplication and there isanother story about it.

Our problem is a part of duplication one. About the same version fragmentation and bloating node_modules with, but more often without a reason.

You might have already A LOT of duplications in your lock file - try to dedupe them if you never tried and tell me the results. At least check how manytslibs you've got.

PS: deduplication is available for any package manager, as a built-in or as a separate package. Just google.

A real life example?

🤔 probably the last example was also now quite sound. Let's create something better. Something that has bitten me personally multiple times.

Let's talk about a real stuff, about consumingintertwined packages.

1️⃣cool-widget usescool-form usescool-button (and other packages)
2️⃣cool-widget usescool-button as well 😎

Again - this case is not applicable to simple projects. This case is not very applicable to monorepos.
This case requires a presence of someflexibility and a non-zerodistance between package. It needs a lag, it needs a gap, it needs a latency.

This case needs one repo consuming another to have agap in thenpm logistic layer in between

Let's create asupply chain disruption ⬇️

• 1️⃣ Imagine you have aButton. Just abutton exported as a npm package -cool-button. Every UIKit has such primitive.
• 2️⃣ and then you use thatcool-button in many other packages. For example incool-formandcool-widget
• 3️⃣ andcool-widget usescool-form
  • It is a big organism consuming other molecules(form)and atoms (button).
• ...
• ➡️ And then you introduce a new major version of your verycool-button 🙀.

The tricky part here is how "major version update"propagates through packages:

• cool-button should have get a major version bump due to public interface change
• cool-form might got a patch update, as nothing changes for it's public interface. Button is implementation details and is not affecting look-n-feel of a form
• cool-widget actually does nothave to do anything....or haven't updated all required dependenciesyet.

Ourcool-widget might be patch-updated, as it does not change public interface, just internals, and

• it may use a new version of acool-button, because some renovate-bot (or a button's developer) updated it in all consumers.
• depend on acool-form, which also was just updated to use a new button, but... 😉 there is no reason forcool-widget to update dependency oncool-form as nothing actually was changed. It's the samecool-form.

In other words - dependency `cool-form` will be kept as is

What 😕? How so? Why not updated?

Let's stop here and check if cool-widget have to react and incorporate every single move of every single piece. Any reason?

🤷‍♂️ Why someone should be bothered to bump everydependency in package.json every time everydependency, a little spare part of a bigger whole, updates? It could be ten times a day for ten packages, and it sounds like a lot of useless work! Especially some work is required to perform update, run test, verify the result, deploy the result. Oh

Ok, ok. Let's imagine you are fine with it, You want it. So abutton update will cause:

• 1️⃣ a cascade update of ALL packages depending on it,
• 2️⃣ and then of all packages depending on just changed
• 3️⃣ and then of all packages depending on just changed
• 4️⃣ and then of all packages depending on just changed
• 5️⃣ should I stop here?

It actually can be a loop -sometimes packages might depends on previous versions of themselves. Likebabel usesbabel to compile self.
So 😕 I really hope you don't want that. You probably do update in batches, or not really doing it at all, and that's ok - if stuff does work, just don't touch it.

Long story short -cool-widget have got a newcool-button@v2 directly updated, but still have the oldcool-button@v1 from thekept-as-iscool-form

So now we have two buttons, with different major versions, and this cannot bededuplicated.

😠:"HEY! Wait a minute. I've got you and I am promising to be a good lad and keep my deep dependencies up to date!"
😿:"Unfortunately this is not about you, this is __about packages created by someone else, about the packages you don't control"
😕:"I dont control?"
😾:"You don't"

You have got a problem withstale intermediate dependency. A dependency of your dependency which you don't directly control. And a dependency [of your dependency of your dependency]. And a dependency [of your dependency of your dependency of your dependency]. And ....

And that isthe problem we were chasing all this time.

First time this issue was encountered back in 2017 –Yark: How to upgrade indirect dependencies?

Chapter 2 - Solving the problem

It's an easy job to point a finger on a problem and tell everyone -"People! Here is the problem".
It is completely another job to propose a solution.

Going back and upgrading

Lets make another step back and look on what you do control - the direct dependencies:

• you can declare dependencies you need and their versions
  • as their versions might represent a wide range - the particular versions to use will be stored in yourlock file
  • if by any reason you get a newer version of already existing dep - it will be automaticallyraised to the highest one. If not automatically, then afterdedupe.
  • 😅 so by installing a new dep you can change dependencies of your existing?
    • This is why changes inyarn.lock might need a review
• you can manage and especially update those deps.
  • useyarn up, oryarn upgrade-interactive
  • or use a little bit more advancednpm-check-updates
  • in both cases onlypackage.lock gonna be changed

This is how you can controldirect dependencies. The ones explicitly declared in yourpackage lock. You have no control over indirect dependencies totallyderived from requirements of thereal ones.

Package managers are trying to preserve as much as possible in order to reduce the impact of the change and amount efforts users might need to perform in order to manage consequences. Known asPrinciple of least action.

Package manager's job is to keep things. Our goal is to _loosen some bolts...

Please welcome...

Yarn-unlock-file

yarn-unlock-file is a new library, capable of editing your lock file and addressing issues created by indirect dependencies

The simplest use case is nothing more than

npx yarn-unlock-file all# and don't forget toyarn

That is it - the command will:

• 🔐 keep any of dependencies declared by you
• 🔓 "unlock" all indirect ones
  • "unlock" means "delete" old records fromyarn.lock letting the newer, might be more correct versions to be installed

What you are dependencies?

There isone thing we forgot – the purpose of package managers. What isyarn, what isnpm and what they do...

PackageManagers are primarily managinglogistics - they provide transport layer betweennpm repository and yourproject.
Then PackageManagers are in charge to recreate the same environment, the same combination of packages across different machines to make our live morepredictable.

However, if you use a library, you probablyrely on it to be tested and behave well, but this is possibleonly if the library wasassembled in your projects in exactly same way it was built and tested in the place of the origin - it's own space.

So it's important tolet free artificial boundaries preventing the similar environments

Everything has limits, and a perfect tool capable to capture andfreeze your decisions in time might need a hand.

No tool should change your decisions and change package versions picked by you, but it might help you manage decisions you haven't made consciously - concrete versions for your indirect dependencies.

Try commands like

• npx yarn-unlock-file dev to update only devDependencies, or
• npx yarn-unlock-file matching @material-ui/* to update dependencies ofmaterial-uibut not MUI itself

Safety first

There is a reasons why we are not just deletingyarn.lock - the result can be unpredictable. Or it can be "too much" - you will get a broken build or a broken application if your tests are not great in detecting anomalies.

• One should bumping your dependencies regularly, but it's not always safe and can go unpredictable in large projects.
  • One might update versions in the same range, and "unlocking" is a good way to do that.
• One might still be cautions about updating direct dependencies, or even dependencies of dependencies. But not many layers is there?
  • npx yarn-unlock-file levels all will tell you how much.
    • I haven't seen more than 8.
    • I haven't seen less than 3.
  • 😎 updating dependencies of dependencies of dependencies should be safe
    • npx yarn-unlock-file all --min-level 3

Dry run

Well, why one should blindly trust some tool to delete only what you want. Trust, but verify. Use dry run mode.
npx yarn-unlock-file all --min-level 3 --dry-run

The picture above is a good indication of how many layers can be hidden inside a simple solution,memoize-one in this case.

One might ask the question - why level 8 dependencyis-arraysh is here, and what causes it?

• lets ask:yarn why is-arrayish> nyc#test-exclude#read-pkg-up#read-pkg#load-json-file#parse-json#error-ex#is-arrayish
• let's update only this branch:npx yarn-unlock-file matching nyc --dry-run

1️⃣ look 2️⃣ pickmin-level you are comfortable with 3️⃣ wipe

Note about Npm-check-Updates

ncu has a "doctor" mode to update dependency, run tests, and only then update another dependency.
While this works just great - it actually cantangle your lock file beyond imagination because packages you install today affects the shape of package you install tomorrow 😉

Having more control over this process is quite beneficial.

• For years I was giving an advice of"just find something in your lock file and delete".

• For years I was looking intoyarn.lock updates to correct result of deduplication sometimes by selecting a few hundred of lines todelete and regenerate.

🤷‍♂️ enough is enough. It's time to understand that deterministic builds is one problem, butsupply chain disruption is different.

Authors of packages and tools you useexpect and test their creations using some know (to them) set of other packages and tools, which are in turn rely on some other set of libraries. Any dependency specified within an accepted "range", and most of them are, is a subject for version mismatch. Or anexpectation that the end consumers will use a new patch version of a tiny library with something very important fixed.

There are dependencies you know, the ones you want to use.
Everything else has to be unlocked on a weekly basic.

You can use Renovate, Dependabot ornpm-check-updates

But it's not enough
https://github.com/theKashey/yarn-unlock-file

PS: Did you know thatdependabot can update dependencies in your lock.file? Likehere it bumps terser changing only lock file, because it's an indirect dependency.

So what?

Just try it

npx yarn-unlock-file devyarn

And then we will see 😈

💡: this is a first, partially experimental solution, currently capable to handle only yarn(v1 and v3).