Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

Expand auto-import to all package.json dependencies#38923

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Conversation

andrewbranch
Copy link
Member

@andrewbranchandrewbranch commentedJun 3, 2020
edited
Loading

This PR creates an auxiliary program in the language service layer that contains node_modules packages that are specified as a direct dependency (includingdevDependencies peerDependencies) in a package.json file visible to the project’s root directory and not already present in the main program. This program is then supplied to the auto-imports machinery as a provider of additional modules that can be used for auto-import. Effectively, this means that packages that ship their own types should be available for auto-import immediately afternpm installing them, lifting a long-despised limitation of “you have to write out the import manually once in your project before auto-imports will work, unless it’s in@types, because@types is special.”

Performance

I compared a few operations against master on the output of @angular/cli (with routing). (I chose this as a test because it’s a realistic scenario where a user’s package.json will contain more dependencies with self-shipped types than are actually used in the program. For example,@angular/forms is a listed dependency but isn’t used in the boilerplate.) Measurements are an average of three trials run in VS Code 1.46.0-insider (b1ef2bf) with extensions disabled.

masterPRDiff
Startup time (ms)27292636-92 ms (-3%)
Type “P”*
updateGraph1815-4 ms (-20%)
completionInfo102115+13 ms (+13%)
Backspace, Type “P”**
completionInfo4438-6 ms (-13%)
Comment out import***
getApplicableRefactors4462+18 ms (+42%)

* This triggers completions, including import suggestions forPatternValidator from@angular/forms in the PR trials, a suggestion that is not included in the master trials.

** This triggers completions again without disrupting the auto-import cache populated in the previous action.

*** This changes the structure of the primary program, which triggers an update of the auto-import provider program. The time it takes to do so is included in the subsequentgetApplicableRefactors measurement.

Observations:

  • Editing operations that don’t change the structure of the program are unaffected (looking atupdateGraph time)
  • When generating completions, there’s an added cost associated with looking through more modules and generating auto-import suggestions for them. The penalty isn’t huge, though, and seems to be worth it for the added utility.
  • When something happens that triggers an update of the auto-import provider, there’s an associated cost that scales with the number of typed dependencies in package.json files that are not already in the main program (i.e., the size of the auto-import provider program). Because the Angular CLI generates boilerplate that has more unused typed dependencies than most real codebases (normally when you install a dependency, you import it shortly thereafter), I suspect that it will be rare for users to see much more delay than the 18 ms measured here.

Limitations

  • Only package.jsons found in the project’scurrentDirectory and upward are considered. So in an unusual project structure like

    tsconfig.jsonproject/  node_modules/  package.json  index.ts

    theproject/package.json file won’t be found.

  • This doesn’t currently resolve to JavaScript files—that is, to get auto imports, the package has to have.d.ts files. Processing plain JS for auto-imports could be explored if demand were high, but when I tried it, most of the added packages were CLI tools and plugins, not things people want to import from, so it was a performance hit for little (or even negative) utility.

Memory considerations

The auxiliary program is configured to be as light as possible, but in a project references editing scenario, it’s possible to have many projects open at once. Each project has its own language service, so each open project will typically have one of these auxiliary programs. Some possibilities for mitigating high memory usage in these scenarios:(updated: decided toexclude devDependencies for UX reasons)

  • The auxiliary program won’t be created if there are no packages listed that aren’t already in the main program,but an analysis of around 500 popular open source TypeScript repos shows that this rarely happens, largely due to devDependencies that ship their own types but are never used in the program (prominent example: typescript). Excluding devDependencies significantly increases the chances that creation an auxiliary program can be skipped, at the cost of those devDependencies being unavailable for auto-import (which can be useful when writing build scripts and tests).
  • The auxiliary program isn’t created until there’s an open file that belongs¹ to the project. This prevents us from creating these programs as we do a cross-project find-all-references.
  • If we detect that memory usage is a problem, we could dispose or stop creating these programs. I think@sheetalkamat has looked into this in the past.
  • We could either hard-code into TypeScript or indicate in third-party projects that a package should opt out of being eagerly auto-importable. In one real-world project I tested, the contents of the auxiliary programs were limited to typescript, typescript-eslint, ts-jest, jest, and prettier, none of which are commonly imported.

However, at this point I’m not convinced that any of these mitigations are necessary. The size of these programs is usually in the ballpark of a browser tab (around 50 MB from one early test). But we have avenues to explore if needed.

Fixes#37812
Fixes#36042


¹ “file that belongs to the project” here means “file whose default project is the project in question,” in contrast to “file that is contained by the project”

SantoJambit, harshavamsi, and Memeplexx reacted with heart emoji
e.g., don’t create them during cross-project find-all-refs
@andrewbranch
Copy link
MemberAuthor

@typescript-bot pack this

@typescript-bot
Copy link
Collaborator

typescript-bot commentedJun 3, 2020
edited
Loading

Heya@andrewbranch, I've started to run the tarball bundle task on this PR at86151e1. You can monitor the buildhere.

@typescript-bot
Copy link
Collaborator

typescript-bot commentedJun 3, 2020
edited
Loading

Hey@andrewbranch, I've packed this intoan installable tgz. You can install it for testing by referencing it in yourpackage.json like so:

{    "devDependencies": {        "typescript": "https://typescript.visualstudio.com/cf7ac146-d525-443c-b23c-0d58337efebc/_apis/build/builds/75621/artifacts?artifactName=tgz&fileId=45DA6FF44370AD15DEFF5E61EDC74241CF262D815E40DE2D87AB199B145FE63502&fileName=/typescript-4.0.0-insiders.20200603.tgz"    }}

and then runningnpm install.


There is also a playgroundfor this build.

@andrewbranch
Copy link
MemberAuthor

After further thought, I do think devDependencies should be excluded by default (we could add a configuration option to include them if enough people really want them). Including them by default means that virtually every project written in TypeScript would get auto-imports for the compiler, which nobody wants. If youdo want to import the compiler API,typescript is either going to be in your"dependencies" or"peerDependencies"—and weshould include peer dependencies by default.

@@ -1815,7 +1815,7 @@ namespace ts.server {
name,
rootFileName,
compilerOptions,
hostProject));
moduleResolutionHost));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

You dont need to pass moduleResolutionHost here but instead you want this to also handle trace (if logging enabled going to log) and currentDIrectory
ModuleResolutionHost = hostProject.projectService ?

Copy link
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

I don’t think I understand the suggestion—ProjectService isn’t assignable toModuleResolutionHost.

Copy link
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Merging for the beta, feel free to follow up with me

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

You want it to be {
boundMethods from hostProject.projectService.host
getCurrentDirectory : () => hostProject.getCurrentDirectory(),
trace: mayBeBind(hostProject, hostProject.trace)
}

Copy link
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Is there a realistic example of whyhostProject itself wouldn’t work? If it wants to mimic the existence of some files, why shouldn’t it be allowed to do that during type directive resolution?

Co-authored-by: Sheetal Nandi <shkamat@microsoft.com>
@andrewbranchandrewbranch changed the titleExperiment: expand auto-import to all package.json dependenciesExpand auto-import to all package.json dependenciesJun 22, 2020
@andrewbranchandrewbranch merged commit086e00d intomicrosoft:masterJun 22, 2020
@andrewbranchandrewbranch deleted the experiment/auto-import/program-per-package-json branchJune 22, 2020 23:34
cangSDARM added a commit to cangSDARM/TypeScript that referenced this pull requestJun 23, 2020
* upstream/master: (58 commits)  Variadic tuple types (microsoft#39094)  chore: resolve suggestions  Expand auto-import to all package.json dependencies (microsoft#38923)  inline local functions  Update bigint declaration file (microsoft#38526)  Update user baselines (microsoft#39077)  LEGO: check in for master to temporary branch.  Add missing index.ts files to user projects (microsoft#39163)  Add reason for a disabled code action (microsoft#37871)  Minor fix for assertion predicates (microsoft#38710)  Update LKG (microsoft#39173)  Reparse top level 'await' in modules (microsoft#39084)  change  chore: more change  chore: resolve review  chore: save space  fix: lint error  test: add test for it  chore: make isJsxAttr required  chore: revert change in checker  ...# Conflicts:#src/compiler/binder.ts#src/compiler/checker.ts#src/compiler/parser.ts#src/compiler/types.ts
Jack-Works pushed a commit to Jack-Works/TypeScript that referenced this pull requestJun 24, 2020
* Start experiment* Add logging* Go back to a single program* Fix forEachExternalModuleToImportFrom* Move auxiliary program to language service* Add logging* Don’t use resolution cache* Fix(?) containingProjects for ScriptInfo in auxiliary program* Fix ScriptInfo project inclusion* Add test for default project of auto-importable ScriptInfo* Add fourslash server test* Don’t create auto import provider inside node_modules* Add monorepo-like test* WIP* Naively ensure autoImportProvider is up to date after package.json change* Start limiting when auto update provider gets updated* Respond to changes in node_modules* Don’t create auto-import provider until a file is open that would use ite.g., don’t create them during cross-project find-all-refs* Clean up naming,@internal marking, and fix empty project creation bug* Drop devDependencies, include peerDependencies* Add additional compiler options* Fix interaction with importSuggestionsCache* Move option to UserPreferences, allow inclusion of devDependencies* Don’t filter out peerDependencies* Watch unparseable package.jsons* But don’t filter packages out due to an invalid package.json* Update test* Don’t use autoImportProvider in codefixes where it can never be used (or any refactors)* Add CompletionEntry property for telemetry* Add assertion for isPackageJsonImport to fourslash* Fix missing pushSymbol argument* Add isPackageJsonImport to tests and API baselines* Fix unit test* Host auto import provider in new Project kind* Fix InferredProject attaching on AutoImportProvider-included files, load eagerly* Update Public APIs* Simplify PackageJsonCache host* Remove unneeded markAsDirty* Defer project finished event until after AutoImportProvider is created* Make AutoImportProviderProject always report isOrphan = true* Close and remove AutoImportProviderProject when host project closes* Don’t set pendingEnsureProjectForOpenFiles* Use hasAddedOrRemovedFiles instead of hasNewProgram* Use host-wide watchOptions for package.json watching* Add to `printProjects`* Clean up* Get autoImportProvider directly from LanguageServiceHost* Clean up* Clean up* Close auto import provider on disableLanguageService* Move AutoImportProvider preload to project updateGraph* Clear auto import suggestion cache when provider program changes* Fix tests* Revert yet-unneeded change* Use projectService host for module resolution host* Don’t re-resolve type directives if nothing has changed* Update src/server/project.tsCo-authored-by: Sheetal Nandi <shkamat@microsoft.com>* Use ts.emptyArrayCo-authored-by: Sheetal Nandi <shkamat@microsoft.com>
@woodgear
Copy link

After further thought, I do think devDependencies should be excluded by default (we could add a configuration option to include them if enough people really want them). Including them by default means that virtually every project written in TypeScript would get auto-imports for the compiler, which nobody wants. If youdo want to import the compiler API,typescript is either going to be in your"dependencies" or"peerDependencies"—and weshould include peer dependencies by default.

sometimes we do want to auto import in devDependencies, for example, some test only package. is there some better way to avoid the "typescript" problem? maybe some kind of filter?

electrovir reacted with thumbs up emoji

@andrewbranch
Copy link
MemberAuthor

Once you manually import your devDependency once, it will be available for auto-import throughout the rest of your project. So... it's kind of already a reverse filter that you configure implicitly as you code.

electrovir reacted with confused emoji

@nandorojo
Copy link

Is there a way to specifywhichpackage.json to use for this?

I have a monorepo, structured like this:

  • apps
    • nextjs
  • packages
    • components

Mypackage.json inside ofapps/nextjs has all of my dependencies, such asreact, etc.

However, VSCode only checks therootpackage.json of my monorepo, which has zero dependencies inside of it.As a result, none of the monorepopackages get autoimport.

Is there a way to set the setting to specify apackageJsonPath or something like that?

"typescript.preferences.includePackageJsonAutoImports":"on","typescript.preferences.packageJsonPath":"./apps/nextjs",

@andrewbranch
Copy link
MemberAuthor

@nandorojo are you primarily a TypeScript or JavaScript user? (Or more to the point, is this particular project written in TypeScript?) The expected structure is that each package of a monorepo should have a tsconfig.json file. As a JS user, you can still have a tsconfig.json or jsconfig.json file there, and even just putting{} as its contents should be enough for package.json auto-imports to work. Note that we have an open issue for investigating inferring this structure for JS users without any tsconfig/jsconfig files, but it’s complicated:#45558.

@nandorojo
Copy link

@andrewbranch really appreciate the fast reply.

I'm a TS user exclusively. I tried adding atsconfig.json to eachpackages/* folder as well, but no dice on autoimports.

Interestingly, autoimport does get solved if I duplicate mydependencies fromapps/nextjs/package.json → rootpackage.json'speerDependencies. But it's a definite antipattern to put dependencies in the root of a workspace like that.

I don't mind having atsconfig in eachpackages/* andapps/* folder, but I don't think this solves it for me.

I am able to autoimport my monorepo packages (@beatgig/components does autoimport). But the dependencies fromapps/nextjs, such asreact, still don't autoimport inside ofpackages/*.

Let me know if you need me to clarify anything there, it might be a bit hard to follow.

@andrewbranch
Copy link
MemberAuthor

Are you able to either share the project you’re working in or reproduce this in a little example? The only possible explanation I can think of besides this being a bug is that if we discover too many unused dependencies, we disable the package.json auto imports:

VS Code preference TypeScript - Preferences: Include Package JSON Auto Imports

This would happen if you have a bunch of dependencies listed in a package.json but the project in question has not already imported them. You can try toggling this setting toon and see if that makes a difference. Otherwise, go ahead and open a new issue with repro steps. Thanks!

@nandorojo
Copy link

nandorojo commentedNov 16, 2021
edited
Loading

I did already set that toon inside of.vscode/settings.json, actually, but unfortunately it didn't help. I don't think I can share the current repo since it's private, but I'll try to reproduce it in one you can use and will post an issue. Thank you again!

@nandorojo
Copy link

@andrewbranch I have a repro here:https://github.com/nandorojo/vscode-monorepo-autoimport-repro

It's very minimal (these are all the dependencies):

Screen Shot 2021-11-16 at 1 56 15 PM

Do you need me to open an issue, or is this sufficient? I added a few comments in the only twots files that should give you what you need.

@andrewbranch
Copy link
MemberAuthor

This is working as intended—react is not listed inany package.json visible topackages/components/index.ts. We filter out transitive dependencies from auto-imports because importing from packages you don’texplicitly depend on is fragile—see#30033 and#38768. You don’t need to addreact to theroot package.json, but it should be inpackages/components/package.json. I believe yarn/npm/lerna will all hoist this to the root node_modules so the dependency isn’t duplicated, so there’s no reason not to specify it as a dependency.

nandorojo reacted with thumbs up emoji

@nandorojo
Copy link

nandorojo commentedNov 16, 2021
edited
Loading

I see, thanks for clarifying.

It works fine if I manually import it, though, so it's as if the autoimport doesn't match the actual behavior. Both the Next.js app and the type checker work without explicitly adding these dependencies to each package.

My hesitation with adding it topackages/components/package.json is that I'll have tons of duplicate items in eachpackage.json when it isn't actually required, either at runtime or for types. I would only add these dependencies to each package to get auto-import to work.

Is there a way to possiblyextend the dependencies fromapps/nextjs? (My guess is there isn't, but just in case...)

I believe yarn/npm/lerna will all hoist this to the root node_modules so the dependency isn’t duplicated, so there’s no reason not to specify it as a dependency.

I agree that practically speaking, since they won't duplicate, there's no real downside to this.

But in a large monorepo with over a dozenpackages/* folders and tons of dependencies, it's a pretty rough DX to copy every dependency into so many places, given that it's all only to get the auto-imports to work.

If there are no intentions to allow transitive dependencies (even behind a flag) for autoimports, then I'll just add each dependency to each subpackage. My guess is, you've thought about the downsides more than I have. But I thought I would leave my thoughts on the DX.

I don't necessarily want all transitive dependencies to autoimport – only the ones I've explicitly put in a subpackage'spackage.json.

@nandorojo
Copy link

I did try explicitly adding these to my monorepo packages, as recommended, and it does solve the autoimports.

However, to prevent duplicate dependencies, I have to fix the versions inpackages/components/package.json to the same exact ones inapps/nextjs/package.json. If I don't do this, then my lockfile will reflect an additional dependency.

I was hoping this would result in no changes to my lockfile, but it seems it does:

{"name":"@repo/components","version":"0.0.0","main":"index.ts","devDependencies": {"moti":"*",  }}

For example,moti now has0.16.1 released. Originally,apps/nextjs had version0.16.0. But now that I added it to@repo/components, I now have both versions installed:

image

I can put versions there explicitly instead to avoid duplication, but it makes updating a dependency a bit of a hassle. I now have to increment the version in 15+ folders each time I want to update.

I know this is intended, but I figured it would be useful to share the downside. Thanks again!

TomasSestak reacted with thumbs up emoji

@andrewbranch
Copy link
MemberAuthor

I think package/workspace managers have tools for doing these kind of bulk install operations. I also think that if you’re not publishing these packages individually, but rather the monorepo structure is just for code organization, there’s no great downside to putting common dependencies in the root package.json. And if youare publishing each package individually, then I think itis an antipattern to import transitive dependencies. There are people who vehemently disagree with me on that point in#38768, but I’m pretty firm on it.

@nandorojo
Copy link

nandorojo commentedNov 16, 2021
edited
Loading

I also think that if you’re not publishing these packages individually, but rather the monorepo structure is just for code organization, there’s no great downside to putting common dependencies in the root package.json.

Interesting, good to know. I'm only concerned for the times that I'm using this to organize shared packages across our internal apps. When it comes to open source monorepos I publish, I do in fact copy over the dependencies in each package. But for an app with code organization, I don't know why importing transitive dependencies would really be an issue. I suppose I'll just put the deps in the root package.json, and hopefully that works fine across the apps themselves.

I think package/workspace managers have tools for doing these kind of bulk install operations.

Maybe lerna does this, I've only used it for publishable packages so I'm not sure. If there aren't downsides to using the root package on internal projects, I might just do that. But yarn workspaces warns against it whenever you install something in the root package, so I figured it wasn't a good pattern.

andrewbranch reacted with thumbs up emoji

@nandorojo
Copy link

nandorojo commentedNov 16, 2021
edited
Loading

In case anyone else lands, here, I found a nice monorepo tool that extends Lerna's capabilities to keep versions tied together between packages when you install/upgrade:https://www.npmjs.com/package/lerna-update-wizard

Heard throughlerna/lerna#2142 (comment)

I wrote up a guide here:https://github.com/nandorojo/ts-monorepo-autoimport-guide

Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment
Reviewers

@sheetalkamatsheetalkamatsheetalkamat requested changes

@uniqueiniquityuniqueiniquityuniqueiniquity left review comments

@amcaseyamcaseyamcasey approved these changes

@mjbvzmjbvzmjbvz approved these changes

@DanielRosenwasserDanielRosenwasserAwaiting requested review from DanielRosenwasser

@sandersnsandersnAwaiting requested review from sandersn

Assignees

@andrewbranchandrewbranch

Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Expand auto-import to all dependencies in package.json Automatic imports don't work for packages that provide their own types
9 participants
@andrewbranch@typescript-bot@amcasey@woodgear@nandorojo@DanielRosenwasser@sheetalkamat@uniqueiniquity@mjbvz

[8]ページ先頭

©2009-2025 Movatter.jp