Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

tools: port js2c.py to C++#46997

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Closed
joyeecheung wants to merge4 commits intonodejs:mainfromjoyeecheung:js2c-cpp
Closed

Conversation

joyeecheung
Copy link
Member

@joyeecheungjoyeecheung commentedMar 7, 2023
edited
Loading

tools: port js2c.py to C++

This makes it easier to use third-party dependencies in this tool
(e.g. adding compression using algorithms not available in Python).
It is also much faster - locally js2c.py takes ~1.5s to generate the
output whereas this version takes ~0.1s - and consumes less memory
(~110MB v.s. 66MB).

This also modifies the js2c.py a bit to simplify the output, making
it easier to compare with one generated by the C++ version. Locally
the output from the two are identical. We'll remove js2c.py in a
subsequent commit when the C++ version is used by default.

build: replace js2c.py with js2c.cc

Locally, running the two version for the first time (when the output does not yet exist so no reading is done):

❯ time tools/js2c.py --directory lib --target out2.cc config.gypi deps/v8/tools/splaytree.mjs deps/v8/tools/codemap.mjs deps/v8/tools/consarray.mjs deps/v8/tools/csvparser.mjs deps/v8/tools/profile.mjs deps/v8/tools/profile_view.mjs deps/v8/tools/logreader.mjs deps/v8/tools/arguments.mjs deps/v8/tools/tickprocessor.mjs deps/v8/tools/sourcemap.mjs deps/v8/tools/tickprocessor-driver.mjs deps/acorn/acorn/dist/acorn.js deps/acorn/acorn-walk/dist/walk.js deps/cjs-module-lexer/lexer.js deps/cjs-module-lexer/dist/lexer.js deps/undici/undici.jsuser time1.36ssystem time0.16stotal time1.532smax memory94924K❯ time out/Release/node_js2c out.cc lib config.gypi deps/v8/tools/splaytree.mjs deps/v8/tools/codemap.mjs deps/v8/tools/consarray.mjs deps/v8/tools/csvparser.mjs deps/v8/tools/profile.mjs deps/v8/tools/profile_view.mjs deps/v8/tools/logreader.mjs deps/v8/tools/arguments.mjs deps/v8/tools/tickprocessor.mjs deps/v8/tools/sourcemap.mjs deps/v8/tools/tickprocessor-driver.mjs deps/acorn/acorn/dist/acorn.js deps/acorn/acorn-walk/dist/walk.js deps/cjs-module-lexer/lexer.js deps/cjs-module-lexer/dist/lexer.js deps/undici/undici.jsuser time0.06ssystem time0.03stotal time0.092smax memory51048K❯ diff out.cc out2.cc

And running for the second time (reading the existing output and not overwriting since the content isn't going to change):

❯ time tools/js2c.py --directory lib --target out2.cc config.gypi deps/v8/tools/splaytree.mjs deps/v8/tools/codemap.mjs deps/v8/tools/consarray.mjs deps/v8/tools/csvparser.mjs deps/v8/tools/profile.mjs deps/v8/tools/profile_view.mjs deps/v8/tools/logreader.mjs deps/v8/tools/arguments.mjs deps/v8/tools/tickprocessor.mjs deps/v8/tools/sourcemap.mjs deps/v8/tools/tickprocessor-driver.mjs deps/acorn/acorn/dist/acorn.js deps/acorn/acorn-walk/dist/walk.js deps/cjs-module-lexer/lexer.js deps/cjs-module-lexer/dist/lexer.js deps/undici/undici.jsuser time1.08ssystem time0.12stotal time1.208smax memory104720K❯ time out/Release/node_js2c out.cc lib config.gypi deps/v8/tools/splaytree.mjs deps/v8/tools/codemap.mjs deps/v8/tools/consarray.mjs deps/v8/tools/csvparser.mjs deps/v8/tools/profile.mjs deps/v8/tools/profile_view.mjs deps/v8/tools/logreader.mjs deps/v8/tools/arguments.mjs deps/v8/tools/tickprocessor.mjs deps/v8/tools/sourcemap.mjs deps/v8/tools/tickprocessor-driver.mjs deps/acorn/acorn/dist/acorn.js deps/acorn/acorn-walk/dist/walk.js deps/cjs-module-lexer/lexer.js deps/cjs-module-lexer/dist/lexer.js deps/undici/undici.jsuser time0.06ssystem time0.04stotal time0.101smax memory64924K

lin72h reacted with thumbs up emojiaduh95 and lin72h reacted with heart emojilin72h reacted with rocket emoji
@nodejs-github-bot
Copy link
Collaborator

Review requested:

  • @nodejs/gyp

@nodejs-github-botnodejs-github-bot added buildIssues and PRs related to build files or the CI. dependenciesPull requests that update a dependency file. libuvIssues and PRs related to the libuv dependency or the uv binding. needs-ciPRs that need a full CI run. pythonPRs and issues that require attention from people who are familiar with Python. toolsIssues and PRs related to the tools directory. labelsMar 7, 2023
@nodejs-github-bot
Copy link
Collaborator

@nodejs-github-bot
Copy link
Collaborator

@joyeecheung
Copy link
MemberAuthor

cc @nodejs/build @nodejs/build-files

@kvakil
Copy link
Contributor

@joyeecheung do you mind if I take over this PR and rebase it? I have some ideas I would like to build on top of this :)

This makes it easier to use third-party dependencies in this tool(e.g. adding compression using algorithms not available in Python).It is also much faster - locally js2c.py takes ~1.5s to generate theoutput whereas this version takes ~0.1s - and consumes less memory(~110MB v.s. 66MB).This also modifies the js2c.py a bit to simplify the output, makingit easier to compare with one generated by the C++ version. Locallythe output from the two are identical. We'll remove js2c.py in asubsequent commit when the C++ version is used by default.
@joyeecheung
Copy link
MemberAuthor

I actually had the rebase locally, and was just waiting for the two other js2c changes to land before I could push it (and also, waiting for reviews, which this still doesn't have...so ping @nodejs/build @nodejs/build-files again..)

@anonriganonrig added the request-ciAdd this label to start a Jenkins CI on a PR. labelMay 21, 2023
@github-actionsgithub-actionsbot removed the request-ciAdd this label to start a Jenkins CI on a PR. labelMay 21, 2023
@nodejs-github-bot
Copy link
Collaborator

@anonrig
Copy link
Member

cc @nodejs/cpp-reviewers

@anonrig
Copy link
Member

@joyeecheung can we merge this PR?

@joyeecheung
Copy link
MemberAuthor

Landed inceb2a21...4da7bc9

joyeecheung added a commit that referenced this pull requestMay 24, 2023
This makes it easier to use third-party dependencies in this tool(e.g. adding compression using algorithms not available in Python).It is also much faster - locally js2c.py takes ~1.5s to generate theoutput whereas this version takes ~0.1s - and consumes less memory(~110MB v.s. 66MB).This also modifies the js2c.py a bit to simplify the output, makingit easier to compare with one generated by the C++ version. Locallythe output from the two are identical. We'll remove js2c.py in asubsequent commit when the C++ version is used by default.PR-URL:#46997Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
joyeecheung added a commit that referenced this pull requestMay 24, 2023
PR-URL:#46997Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
targos pushed a commit that referenced this pull requestMay 30, 2023
This makes it easier to use third-party dependencies in this tool(e.g. adding compression using algorithms not available in Python).It is also much faster - locally js2c.py takes ~1.5s to generate theoutput whereas this version takes ~0.1s - and consumes less memory(~110MB v.s. 66MB).This also modifies the js2c.py a bit to simplify the output, makingit easier to compare with one generated by the C++ version. Locallythe output from the two are identical. We'll remove js2c.py in asubsequent commit when the C++ version is used by default.PR-URL:#46997Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
targos pushed a commit that referenced this pull requestMay 30, 2023
PR-URL:#46997Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
@targostargos mentioned this pull requestJun 4, 2023
Ceres6 pushed a commit to Ceres6/node that referenced this pull requestAug 14, 2023
This makes it easier to use third-party dependencies in this tool(e.g. adding compression using algorithms not available in Python).It is also much faster - locally js2c.py takes ~1.5s to generate theoutput whereas this version takes ~0.1s - and consumes less memory(~110MB v.s. 66MB).This also modifies the js2c.py a bit to simplify the output, makingit easier to compare with one generated by the C++ version. Locallythe output from the two are identical. We'll remove js2c.py in asubsequent commit when the C++ version is used by default.PR-URL:nodejs#46997Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
Ceres6 pushed a commit to Ceres6/node that referenced this pull requestAug 14, 2023
PR-URL:nodejs#46997Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
Ceres6 pushed a commit to Ceres6/node that referenced this pull requestAug 14, 2023
This makes it easier to use third-party dependencies in this tool(e.g. adding compression using algorithms not available in Python).It is also much faster - locally js2c.py takes ~1.5s to generate theoutput whereas this version takes ~0.1s - and consumes less memory(~110MB v.s. 66MB).This also modifies the js2c.py a bit to simplify the output, makingit easier to compare with one generated by the C++ version. Locallythe output from the two are identical. We'll remove js2c.py in asubsequent commit when the C++ version is used by default.PR-URL:nodejs#46997Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
Ceres6 pushed a commit to Ceres6/node that referenced this pull requestAug 14, 2023
PR-URL:nodejs#46997Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
codebytere added a commit to electron/electron that referenced this pull requestNov 2, 2023
codebytere added a commit to electron/electron that referenced this pull requestNov 6, 2023
codebytere added a commit to electron/electron that referenced this pull requestNov 14, 2023
codebytere added a commit to electron/electron that referenced this pull requestNov 15, 2023
codebytere added a commit to electron/electron that referenced this pull requestNov 16, 2023
codebytere added a commit to electron/electron that referenced this pull requestNov 21, 2023
codebytere added a commit to electron/electron that referenced this pull requestNov 22, 2023
codebytere added a commit to electron/electron that referenced this pull requestNov 28, 2023
codebytere added a commit to electron/electron that referenced this pull requestNov 29, 2023
jkleinsc pushed a commit to electron/electron that referenced this pull requestNov 30, 2023
* chore: upgrade to Node.js v20* src: allow embedders to override NODE_MODULE_VERSIONnodejs/node#49279* src: fix missing trailing ,nodejs/node#46909* src,tools: initialize cppgcnodejs/node#45704* tools: allow passing absolute path of config.gypi in js2cnodejs/node#49162* tools: port js2c.py to C++nodejs/node#46997* doc,lib: disambiguate the old term, NativeModulenodejs/node#45673* chore: fixup Node.js BSSL tests*nodejs/node#49492*nodejs/node#44498* deps: upgrade to libuv 1.45.0nodejs/node#48078* deps: update V8 to 10.7nodejs/node#44741* test: use gcUntil() in test-v8-serialize-leaknodejs/node#49168* module: make CJS load from ESM loadernodejs/node#47999* src: make BuiltinLoader threadsafe and non-globalnodejs/node#45942* chore: address changes to CJS/ESM loading* module: make CJS load from ESM loader (nodejs/node#47999)* lib: improve esm resolve performance (nodejs/node#46652)* bootstrap: optimize modules loaded in the built-in snapshotnodejs/node#45849* test: mark test-runner-output as flakynodejs/node#49854* lib: lazy-load deps in modules/run_main.jsnodejs/node#45849* url: use private properties for brand checknodejs/node#46904* test: refactor `test-node-output-errors`nodejs/node#48992* assert: deprecate callTrackernodejs/node#47740* src: cast v8::Object::GetInternalField() return value to v8::Valuenodejs/node#48943* test: adapt test-v8-stats for V8 updatenodejs/node#45230* tls: ensure TLS Sockets are closed if the underlying wrapclosesnodejs/node#49327* test: deflake test-tls-socket-closenodejs/node#49575* net: fix crash due to simultaneous close/shutdown on JS Stream Socketsnodejs/node#49400* net: use asserts in JS Socket Stream to catch races in futurenodejs/node#49400* lib: fix BroadcastChannel initialization locationnodejs/node#46864* src: create BaseObject with node::Realmnodejs/node#44348* src: implement DataQueue and non-memory resident Blobnodejs/node#45258* sea: add support for V8 bytecode-only cachingnodejs/node#48191* chore: fixup patch indices* gyp: put filenames in variablesnodejs/node#46965* build: modify js2c.py into GN executable* fix: (WIP) handle string replacement of fs -> original-fs* [v20.x] backport vm-related memoryfixesnodejs/node#49874* src: make BuiltinLoader threadsafe and non-globalnodejs/node#45942* src: avoid copying string in fs_permissionnodejs/node#47746* look upon my works ye mightyand dispair* chore: patch cleanup* [api] Remove AllCan Read/Writehttps://chromium-review.googlesource.com/c/v8/v8/+/5006387* fix: missing include for NODE_EXTERN* chore: fixup patch indices* fix: fail properly when js2c fails in Node.js* build: fix js2c root_gen_dir* fix: lib/fs.js -> lib/original-fs.js* build: fix original-fs file xforms* fixup! module: make CJS load from ESM loader* build: get rid of CppHeap for now* build: add patch to prevent extra fs lookup on esm load* build: greatly simplify js2c modificationsMoves our original-fs modifications back into a super simple python script action, wires up the output of that action into our call to js2c* chore: update to handle moved internal/modules/helpers file* test: update @types/node test* feat: enable preventing cppgc heap creation* feat: optionally prevent calling V8::EnableWebAssemblyTrapHandler* fix: no cppgc initialization in the renderer* gyp: put filenames in variablesnodejs/node#46965* test: disable single executable tests* fix: nan tests failing on node headers missing file* tls,http2: send fatal alert on ALPN mismatchnodejs/node#44031* test: disable snapshot tests*nodejs/node#47887*nodejs/node#49684*nodejs/node#44193* build: use deps/v8 for v8/toolsNode.js hard depends on these in their builtins* test: fix edge snapshot stack tracesnodejs/node#49659* build: remove js2c //base dep* build: use electron_js2c_toolchain to build node_js2c* fix: don't create SafeSet outside packageResolveFixes failure in parallel/test-require-delete-array-iterator:=== release test-require-delete-array-iterator ===Path: parallel/test-require-delete-array-iteratornode:internal/per_context/primordials:426    constructor(i) { super(i); } // eslint-disable-line no-useless-constructor                     ^TypeError: object is not iterable (cannot read property Symbol(Symbol.iterator))    at new Set (<anonymous>)    at new SafeSet (node:internal/per_context/primordials:426:22)* fix: failing crashReporter tests on LinuxThese were failing because our change from node::InitializeNodeWithArgs tonode::InitializeOncePerProcess meant that we now inadvertently calledPlatformInit, which reset signal handling. This meant that our intentionalcrash function ElectronBindings::Crash no longer worked and the renderer processno longer crashed when process.crash() was called. We don't want to use Node.js'default signal handling in the renderer process, so we disable it by passingkNoDefaultSignalHandling to node::InitializeOncePerProcess.* build: only create cppgc heap on non-32 bit platforms* chore: clean up util:CompileAndCall* src: fix compatility with upcoming V8 12.1 APIsnodejs/node#50709* fix: use thread_local BuiltinLoader* chore: fixup v8 patch indices---------Co-authored-by: Keeley Hammond <vertedinde@electronjs.org>Co-authored-by: Samuel Attard <marshallofsound@electronjs.org>
MrHuangJser pushed a commit to MrHuangJser/electron that referenced this pull requestDec 11, 2023
* chore: upgrade to Node.js v20* src: allow embedders to override NODE_MODULE_VERSIONnodejs/node#49279* src: fix missing trailing ,nodejs/node#46909* src,tools: initialize cppgcnodejs/node#45704* tools: allow passing absolute path of config.gypi in js2cnodejs/node#49162* tools: port js2c.py to C++nodejs/node#46997* doc,lib: disambiguate the old term, NativeModulenodejs/node#45673* chore: fixup Node.js BSSL tests*nodejs/node#49492*nodejs/node#44498* deps: upgrade to libuv 1.45.0nodejs/node#48078* deps: update V8 to 10.7nodejs/node#44741* test: use gcUntil() in test-v8-serialize-leaknodejs/node#49168* module: make CJS load from ESM loadernodejs/node#47999* src: make BuiltinLoader threadsafe and non-globalnodejs/node#45942* chore: address changes to CJS/ESM loading* module: make CJS load from ESM loader (nodejs/node#47999)* lib: improve esm resolve performance (nodejs/node#46652)* bootstrap: optimize modules loaded in the built-in snapshotnodejs/node#45849* test: mark test-runner-output as flakynodejs/node#49854* lib: lazy-load deps in modules/run_main.jsnodejs/node#45849* url: use private properties for brand checknodejs/node#46904* test: refactor `test-node-output-errors`nodejs/node#48992* assert: deprecate callTrackernodejs/node#47740* src: cast v8::Object::GetInternalField() return value to v8::Valuenodejs/node#48943* test: adapt test-v8-stats for V8 updatenodejs/node#45230* tls: ensure TLS Sockets are closed if the underlying wrapclosesnodejs/node#49327* test: deflake test-tls-socket-closenodejs/node#49575* net: fix crash due to simultaneous close/shutdown on JS Stream Socketsnodejs/node#49400* net: use asserts in JS Socket Stream to catch races in futurenodejs/node#49400* lib: fix BroadcastChannel initialization locationnodejs/node#46864* src: create BaseObject with node::Realmnodejs/node#44348* src: implement DataQueue and non-memory resident Blobnodejs/node#45258* sea: add support for V8 bytecode-only cachingnodejs/node#48191* chore: fixup patch indices* gyp: put filenames in variablesnodejs/node#46965* build: modify js2c.py into GN executable* fix: (WIP) handle string replacement of fs -> original-fs* [v20.x] backport vm-related memoryfixesnodejs/node#49874* src: make BuiltinLoader threadsafe and non-globalnodejs/node#45942* src: avoid copying string in fs_permissionnodejs/node#47746* look upon my works ye mightyand dispair* chore: patch cleanup* [api] Remove AllCan Read/Writehttps://chromium-review.googlesource.com/c/v8/v8/+/5006387* fix: missing include for NODE_EXTERN* chore: fixup patch indices* fix: fail properly when js2c fails in Node.js* build: fix js2c root_gen_dir* fix: lib/fs.js -> lib/original-fs.js* build: fix original-fs file xforms* fixup! module: make CJS load from ESM loader* build: get rid of CppHeap for now* build: add patch to prevent extra fs lookup on esm load* build: greatly simplify js2c modificationsMoves our original-fs modifications back into a super simple python script action, wires up the output of that action into our call to js2c* chore: update to handle moved internal/modules/helpers file* test: update @types/node test* feat: enable preventing cppgc heap creation* feat: optionally prevent calling V8::EnableWebAssemblyTrapHandler* fix: no cppgc initialization in the renderer* gyp: put filenames in variablesnodejs/node#46965* test: disable single executable tests* fix: nan tests failing on node headers missing file* tls,http2: send fatal alert on ALPN mismatchnodejs/node#44031* test: disable snapshot tests*nodejs/node#47887*nodejs/node#49684*nodejs/node#44193* build: use deps/v8 for v8/toolsNode.js hard depends on these in their builtins* test: fix edge snapshot stack tracesnodejs/node#49659* build: remove js2c //base dep* build: use electron_js2c_toolchain to build node_js2c* fix: don't create SafeSet outside packageResolveFixes failure in parallel/test-require-delete-array-iterator:=== release test-require-delete-array-iterator ===Path: parallel/test-require-delete-array-iteratornode:internal/per_context/primordials:426    constructor(i) { super(i); } // eslint-disable-line no-useless-constructor                     ^TypeError: object is not iterable (cannot read property Symbol(Symbol.iterator))    at new Set (<anonymous>)    at new SafeSet (node:internal/per_context/primordials:426:22)* fix: failing crashReporter tests on LinuxThese were failing because our change from node::InitializeNodeWithArgs tonode::InitializeOncePerProcess meant that we now inadvertently calledPlatformInit, which reset signal handling. This meant that our intentionalcrash function ElectronBindings::Crash no longer worked and the renderer processno longer crashed when process.crash() was called. We don't want to use Node.js'default signal handling in the renderer process, so we disable it by passingkNoDefaultSignalHandling to node::InitializeOncePerProcess.* build: only create cppgc heap on non-32 bit platforms* chore: clean up util:CompileAndCall* src: fix compatility with upcoming V8 12.1 APIsnodejs/node#50709* fix: use thread_local BuiltinLoader* chore: fixup v8 patch indices---------Co-authored-by: Keeley Hammond <vertedinde@electronjs.org>Co-authored-by: Samuel Attard <marshallofsound@electronjs.org>
Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment
Reviewers

@anonriganonriganonrig approved these changes

Assignees
No one assigned
Labels
buildIssues and PRs related to build files or the CI.dependenciesPull requests that update a dependency file.libuvIssues and PRs related to the libuv dependency or the uv binding.needs-ciPRs that need a full CI run.pythonPRs and issues that require attention from people who are familiar with Python.toolsIssues and PRs related to the tools directory.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

5 participants
@joyeecheung@nodejs-github-bot@kvakil@anonrig@danielleadams

[8]ページ先頭

©2009-2025 Movatter.jp