Uh oh!
There was an error while loading.Please reload this page.
- Notifications
You must be signed in to change notification settings - Fork2.8k
feat(typescript-estree): switch to globby#2418
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
Uh oh!
There was an error while loading.Please reload this page.
Conversation
Thanks for the PR,@bradzacher! typescript-eslint is a 100% community driven project, and we are incredibly grateful that you are contributing to that community. The core maintainers work on this in their personal time, so please understand that it may not be possible for them to review your work immediately. Thanks again! 🙏Please, if you or your company is finding typescript-eslint valuable, help us sustain the project by sponsoring it transparently onhttps://opencollective.com/typescript-eslint. As a thank you, your profile/company logo will be added to our main README which receives thousands of unique visitorsper day. |
Fixes#2398If the user has a particularly large node_modules folder and uses globs for `parserOption.project`, then the `glob` library can spend a decent chunk of time searching the `node_modules` folder.In some cases, this can be on the order of hundreds to thousands of milliseconds.This wouldn't be a problem, but for safety and correctness during a persistent parse, we have to do this check for every call to the parser (i.e. every file that's being linted).Over a whole project, this can easily add up to many, many seconds wasted.Previously, we:- applied the project globs, one by one- then manually excluded `tsconfig`s from the list.This meant that we are always slow. I remember I did this because I had issues getting `glob`'s `ignore` option to work at all.## The solution`globby` is a better glob library:- it accepts an array of globs, which saves us doing manual looping- it supports exclusion globs (globs prefixed with `!`), which are evaluated as part of the glob process- it has caching built in by defaultThis allows us to evaluate all of the `project` globs at once, as opposed to one at a time (so should be less duplicated work).This also allows us to evaluate the `projectFolderIgnoreList` at the same time as the `project` globs (so should be no useless work done).All of these together should cut the glob evaluation time down to ~50ms for the first parse, and ~2ms for each parse after that (due to caching).For comparison, previously, in bad cases we would spend ~3-500ms, per project, per parsed file.Example to illustrate how much faster this can potentially be:For a project that provides 2 globs and has 100 files.Before: 300ms * 2 * 100 = 60,000ms (60s)After: 50ms + 2 * 100 = 250msThis should also save a non-trival amount of time in other, more optimal setups.BREAKING CHANGE:- removes the ability to supply a `RegExp` to `projectFolderIgnoreList`, and changes the meaning of the string value from a regex to a glob.
Fixes#2398If the user has a particularly large node_modules folder and uses globs for `parserOption.project`, then the `glob` library can spend a decent chunk of time searching the `node_modules` folder.In some cases, this can be on the order of hundreds to thousands of milliseconds.This wouldn't be a problem, but for safety and correctness during a persistent parse, we have to do this check for every call to the parser (i.e. every file that's being linted).Over a whole project, this can easily add up to many, many seconds wasted.Previously, we:- applied the project globs, one by one- then manually excluded `tsconfig`s from the list.This meant that we are always slow. I remember I did this because I had issues getting `glob`'s `ignore` option to work at all.## The solution`globby` is a better glob library:- it accepts an array of globs, which saves us doing manual looping- it supports exclusion globs (globs prefixed with `!`), which are evaluated as part of the glob process- it has caching built in by defaultThis allows us to evaluate all of the `project` globs at once, as opposed to one at a time (so should be less duplicated work).This also allows us to evaluate the `projectFolderIgnoreList` at the same time as the `project` globs (so should be no useless work done).All of these together should cut the glob evaluation time down to ~50ms for the first parse, and ~2ms for each parse after that (due to caching).For comparison, previously, in bad cases we would spend ~3-500ms, per project, per parsed file.Example to illustrate how much faster this can potentially be:For a project that provides 2 globs and has 100 files.Before: 300ms * 2 * 100 = 60,000ms (60s)After: 50ms + 2 * 100 = 250msThis should also save a non-trival amount of time in other, more optimal setups.BREAKING CHANGE:- removes the ability to supply a `RegExp` to `projectFolderIgnoreList`, and changes the meaning of the string value from a regex to a glob.
Fixes#2398If the user has a particularly large node_modules folder and uses globs for `parserOption.project`, then the `glob` library can spend a decent chunk of time searching the `node_modules` folder.In some cases, this can be on the order of hundreds to thousands of milliseconds.This wouldn't be a problem, but for safety and correctness during a persistent parse, we have to do this check for every call to the parser (i.e. every file that's being linted).Over a whole project, this can easily add up to many, many seconds wasted.Previously, we:- applied the project globs, one by one- then manually excluded `tsconfig`s from the list.This meant that we are always slow. I remember I did this because I had issues getting `glob`'s `ignore` option to work at all.## The solution`globby` is a better glob library:- it accepts an array of globs, which saves us doing manual looping- it supports exclusion globs (globs prefixed with `!`), which are evaluated as part of the glob process- it has caching built in by defaultThis allows us to evaluate all of the `project` globs at once, as opposed to one at a time (so should be less duplicated work).This also allows us to evaluate the `projectFolderIgnoreList` at the same time as the `project` globs (so should be no useless work done).All of these together should cut the glob evaluation time down to ~50ms for the first parse, and ~2ms for each parse after that (due to caching).For comparison, previously, in bad cases we would spend ~3-500ms, per project, per parsed file.Example to illustrate how much faster this can potentially be:For a project that provides 2 globs and has 100 files.Before: 300ms * 2 * 100 = 60,000ms (60s)After: 50ms + 2 * 100 = 250msThis should also save a non-trival amount of time in other, more optimal setups.BREAKING CHANGE:- removes the ability to supply a `RegExp` to `projectFolderIgnoreList`, and changes the meaning of the string value from a regex to a glob.
Uh oh!
There was an error while loading.Please reload this page.
Fixes#2398
If the user has a particularly large node_modules folder and uses globs for
parserOption.project, then thegloblibrary can spend a decent chunk of time searching thenode_modulesfolder.In some cases, this can be on the order of hundreds to thousands of milliseconds.
This wouldn't be a problem, but for safety and correctness during a persistent parse, we have to do this check for every call to the parser (i.e. every file that's being linted).
Over a whole project, this can easily add up to many, many seconds wasted.
Previously, we:
tsconfigs from the list.This meant that we are always slow. I remember I did this because I had issues getting
glob'signoreoption to work at all.The solution
globbyis a better glob library:!), which are evaluated as part of the glob processThis allows us to evaluate all of the
projectglobs at once, as opposed to one at a time (so should be less duplicated work).This also allows us to evaluate the
projectFolderIgnoreListat the same time as theprojectglobs (so should be no useless work done).All of these together should cut the glob evaluation time down to ~50ms for the first parse, and ~2ms for each parse after that (due to caching).
For comparison, previously, in bad cases we would spend ~3-500ms, per project, per parsed file.
Example to illustrate how much faster this can potentially be:
For a project that provides 2
projectglobs and has 100 files.Before:
300ms * 2 projects * 100 files = 60,000ms (60s)After:
50ms + 2ms * 100 files = 250msThis should also save a non-trival amount of time in other, more optimal setups.
BREAKING CHANGE:
RegExptoprojectFolderIgnoreList, and changes the meaning of the string value from a regex to a glob.