You signed in with another tab or window.Reload to refresh your session.You signed out in another tab or window.Reload to refresh your session.You switched accounts on another tab or window.Reload to refresh your session.Dismiss alert
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
Untillibsyncrpc is set up to publish to npm, this PR takes a git dependency on it, which will build the binary from source duringnpm install.You needRust 1.85 or higher to have a successfulnpm install in typescript-go.
Note
Takeaways from design meeting:
Investigate some kind of query system for composing batched requests. Muffled screams of “Not GraphQL!” and “I want GraphQL!” could be heard on the other end of the line.
FFI via napi-go still on the table to investigate as an additional option (not replacing IPC), but may need a different API surface or refactoring since the one included here returns shallow serializable objects.
No consensus on whether remoting ASTs means we don’t need to provide a client-side parser. The performance of remoting is pretty good, but requires loading the large tsgo binary, and the perf maybe isn’t good enough for linters that need to remote thousands of ASTs. There are other JS parsers though, and not having to maintain one with identical behavior to the Go-based parser is seen as a real win.
This PR is the start of a JavaScript API client and Go API server that communicate over STDIO. Only a few methods are implemented; the aim of this PR is to be the basis for discussions around the general architecture, then additional functionality can be filled in.
Same backend, different clients
This PR includes a synchronous JavaScript client for Node.js. It useslibsyncrpc to block during IPC calls to the server. Relatively small changes to the client could produce an asynchronous variant without Node.js-specific native bindings that could work in Deno or Bun. I don’t want to make specific promises about WASM without doing those experiments, but using the same async client with an adapter for calling WASM exports seems possible. I’m imagining that eventually we’ll publish the Node.js-specific sync client as a standalone library for those who need a sync API, and an async version adaptable to other use cases, ideally codegen’d from the same source. The same backend is intended to be used with any out-of-process client.
Client structure
This PR creates two JavaScript packages,@typescript/ast and@typescript/api (which may make more sense as@typescript/api-sync or@typescript/api-node eventually). The former contains a copy of TS 5.9’s AST node definitions, related enums, and node tests (e.g.isIdentifier()), with the minor changes that TS 7 has made to those definitions applied. The latter contains the implementation of the Node.js API client. It currently takes a path to the tsgo executable and spawns it as a child process. (I imagine eventually, the TypeScript 7.0+ compiler npm package will be a peerDependency of the API client, and resolution of the executable can happen automatically.)
Backend structure
tsgo api starts the API server communicating over STDIO. The server initializes theapi.API struct which is responsible for handling requests and managing state, like a stripped-downproject.Service. In fact, it uses the other components of the project system, storing documents and projects the same way. (As the project service gets built out with things like file watchers and optimizations for find-all-references, it would get increasingly unwieldy to use directly as an API service, but a future refactor might extract the basic project and document storage to a shared component.)
The API already has methods that return projects, symbols, and types. These are returned as IDs plus bits of easily serializable info, like name and flags. When one of these objects is requested, the API server stores it with its ID so follow-up requests can be made against those IDs. This does create some memory management challenges, which I’ll discuss a bit later.
Implemented functionality
Here’s a selection of the API client type definitions that shows what methods exist as of this PR:
exportinterfaceAPIOptions{tsserverPath:string;cwd?:string;logFile?:string;fs?:FileSystem;}exportinterfaceFileSystem{directoryExists?:(directoryName:string)=>boolean|undefined;fileExists?:(fileName:string)=>boolean|undefined;getAccessibleEntries?:(directoryName:string)=>FileSystemEntries|undefined;readFile?:(fileName:string)=>string|null|undefined;realpath?:(path:string)=>string|undefined;}exportdeclareclassAPI{constructor(options:APIOptions);parseConfigFile(fileName:string):ConfigResponse;loadProject(configFileName:string):Project;}exportinterfaceConfigResponse{options:Record<string,unknown>;fileNames:string[];}exportdeclareclassProject{configFileName:string;compilerOptions:Record<string,unknown>;rootFiles:readonlystring[];reload():void;getSourceFile(fileName:string):SourceFile|undefined;getSymbolAtLocation(node:Node):Symbol|undefined;getSymbolAtLocation(nodes:readonlyNode[]):Symbol|undefined;getSymbolAtPosition(fileName:string,position:number):Symbol|undefined;getSymbolAtPosition(fileName:string,positions:readonlynumber[]):(Symbol|undefined)[];getTypeOfSymbol(symbol:Symbol):Type|undefined;getTypeOfSymbol(symbols:readonlySymbol[]):(Type|undefined)[];}exportinterfaceNode{readonlyid:number;readonlypos:number;readonlyend:number;readonlykind:SyntaxKind;readonlyparent:Node;forEachChild<T>(visitor:(node:Node)=>T):T|undefined;getSourceFile():SourceFile;}exportinterfaceSourceFileextendsNode{readonlykind:SyntaxKind.SourceFile;// Node types are basically same as Strada, without additional methodsreadonlystatements:NodeArray<Statement>;readonlytext:string;}exportdeclareclassSymbol{id:string;name:string;flags:SymbolFlags;checkFlags:number;}exportdeclareclassType{flags:TypeFlags;}
Client-side virtual file systems are also supported. There’s a helper for making a very simple one from a record:
import{API}from"@typescript/api";import{createVirtualFileSystem}from"@typescript/api/fs";import{SyntaxKind}from"@typescript/ast";constapi=newAPI({cwd:newURL("../../../",import.meta.url).pathname,tsserverPath:newURL("../../../built/local/tsgo",import.meta.url).pathname,fs:createVirtualFileSystem({"/tsconfig.json":"{}","/src/index.ts":`import { foo } from './foo';`,"/src/foo.ts":`export const foo = 42;`,}),});
Performance
These are the results of the included benchmarks on my M2 Mac. Note that IPC isvery fast on Apple Silicon, and Windows seems to see significantly more overhead per call. Tasks prefixedTS - refer to the rough equivalent with the TypeScript 5.9 API. ThegetSymbolAtPosition tasks are operating on TypeScript’sprogram.ts, which has 10893 identifiers.
To editorialize these numbers a bit: in absolute terms, this is pretty fast, even transferring large payloads like a binary-encodedchecker.ts (10). On the order of tens, hundreds, or thousands of API calls, most applications probably wouldn’t notice a per-call regression over using the TypeScript 5.9 API, and may speed up if program creation / parsing multiple files is a significant portion of their API consumption today (5–7). However, the IPC overhead is pretty noticeable when looking at hundreds of thousands of back-to-back calls on an operation that would be essentially free in a native JavaScript API, like getting the symbol for every identifier in a large file (15, 18). For that reason, we’ll be very open to including bulk/batch/composite API methods that reduce the number of round trips needed to retrieve lots of information for common scenarios (16, 17).
Memory management
The current API design uses opaque IDs for objects like symbols and types, so the client can receive a handle to one of these objects and then query for additional information about it. For example, implemented in this PR isgetTypeOfSymbol, which takes a symbol ID. The server has to store the symbol in a map so it can be quickly retrieved when the client asks for its type. This client/server split presents two main challenges:
When the client makes two calls that result in the same symbol or same type, the client should return the strict-equal same object, while allowing garbage collection to work on those objects.
When one of those client objects goes out of scope, it should eventually be released from the server, so server memory doesn’t grow indefinitely.
To accomplish this, there is a client-side object registry that stores objects by their IDs. API users will need to explicitly dispose those objects to release them both from the client-side store and from the server. (Server objects may be automatically released in response to program updates, and making additional queries against them will result in an error.) This can be done with the.dispose() method:
From offline chats it sounds likets-loader would probably not work with the new Go API in type checking mode, asts-loader goes deep into the guts of the TypeScript APIs.
FWIW I think this is the opposite of what I was trying to say; I can't imagine us offering a public API that doesn't let one construct a program, get its errors, and get the outputs. But I don't know which "guts" you're speaking to exactly. Of course, downstream API consumers will require modification for sure, so "not work" is true from that perspective.
John is referring to offline chats with me, because I do know the guts in question, and ts-loader’s API usage is extremely broad, low-level, and implementation-specific.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading.Please reload this page.
Important
Untillibsyncrpc is set up to publish to npm, this PR takes a git dependency on it, which will build the binary from source during
npm install.You needRust 1.85 or higher to have a successfulnpm installin typescript-go.Note
Takeaways from design meeting:
This PR is the start of a JavaScript API client and Go API server that communicate over STDIO. Only a few methods are implemented; the aim of this PR is to be the basis for discussions around the general architecture, then additional functionality can be filled in.
Same backend, different clients
This PR includes a synchronous JavaScript client for Node.js. It useslibsyncrpc to block during IPC calls to the server. Relatively small changes to the client could produce an asynchronous variant without Node.js-specific native bindings that could work in Deno or Bun. I don’t want to make specific promises about WASM without doing those experiments, but using the same async client with an adapter for calling WASM exports seems possible. I’m imagining that eventually we’ll publish the Node.js-specific sync client as a standalone library for those who need a sync API, and an async version adaptable to other use cases, ideally codegen’d from the same source. The same backend is intended to be used with any out-of-process client.
Client structure
This PR creates two JavaScript packages,
@typescript/astand@typescript/api(which may make more sense as@typescript/api-syncor@typescript/api-nodeeventually). The former contains a copy of TS 5.9’s AST node definitions, related enums, and node tests (e.g.isIdentifier()), with the minor changes that TS 7 has made to those definitions applied. The latter contains the implementation of the Node.js API client. It currently takes a path to the tsgo executable and spawns it as a child process. (I imagine eventually, the TypeScript 7.0+ compiler npm package will be a peerDependency of the API client, and resolution of the executable can happen automatically.)Backend structure
tsgo apistarts the API server communicating over STDIO. The server initializes theapi.APIstruct which is responsible for handling requests and managing state, like a stripped-downproject.Service. In fact, it uses the other components of the project system, storing documents and projects the same way. (As the project service gets built out with things like file watchers and optimizations for find-all-references, it would get increasingly unwieldy to use directly as an API service, but a future refactor might extract the basic project and document storage to a shared component.)The API already has methods that return projects, symbols, and types. These are returned as IDs plus bits of easily serializable info, like name and flags. When one of these objects is requested, the API server stores it with its ID so follow-up requests can be made against those IDs. This does create some memory management challenges, which I’ll discuss a bit later.
Implemented functionality
Here’s a selection of the API client type definitions that shows what methods exist as of this PR:
Here’s some example usage from benchmarks:
Client-side virtual file systems are also supported. There’s a helper for making a very simple one from a record:
Performance
These are the results of the included benchmarks on my M2 Mac. Note that IPC isvery fast on Apple Silicon, and Windows seems to see significantly more overhead per call. Tasks prefixed
TS -refer to the rough equivalent with the TypeScript 5.9 API. ThegetSymbolAtPositiontasks are operating on TypeScript’sprogram.ts, which has 10893 identifiers.To editorialize these numbers a bit: in absolute terms, this is pretty fast, even transferring large payloads like a binary-encoded
checker.ts(10). On the order of tens, hundreds, or thousands of API calls, most applications probably wouldn’t notice a per-call regression over using the TypeScript 5.9 API, and may speed up if program creation / parsing multiple files is a significant portion of their API consumption today (5–7). However, the IPC overhead is pretty noticeable when looking at hundreds of thousands of back-to-back calls on an operation that would be essentially free in a native JavaScript API, like getting the symbol for every identifier in a large file (15, 18). For that reason, we’ll be very open to including bulk/batch/composite API methods that reduce the number of round trips needed to retrieve lots of information for common scenarios (16, 17).Memory management
The current API design uses opaque IDs for objects like symbols and types, so the client can receive a handle to one of these objects and then query for additional information about it. For example, implemented in this PR is
getTypeOfSymbol, which takes a symbol ID. The server has to store the symbol in a map so it can be quickly retrieved when the client asks for its type. This client/server split presents two main challenges:To accomplish this, there is a client-side object registry that stores objects by their IDs. API users will need to explicitly dispose those objects to release them both from the client-side store and from the server. (Server objects may be automatically released in response to program updates, and making additional queries against them will result in an error.) This can be done with the
.dispose()method:or withexplicit resource management: