Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

Fix fewest modules OOM#4986

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Open
steinybot wants to merge16 commits intoscala-js:main
base:main
Choose a base branch
Loading
fromsteinybot:fix/4985-fewest-modules-oom

Conversation

steinybot
Copy link

@steinybotsteinybot commentedMay 20, 2024
edited
Loading

Fixes#4985

This is a first attempt. I'm not sure if it is doing the right thing yet but its fast and doesn't throw an OOME and it passes the various assertions inModuleSplitter (which caught one bug).

I ended up creating a separate tagger as keeping track of themaxExcludedHopCount was getting pretty complicated and I was struggling to keep the logic correct. It isn't even used forFewestModules.

maxExcludedHopCount actually doesn't look right to me. AFAICT the value you end up with depends on the order that the paths are traversed (which is undefined). If a longer path with more exclusions is traversed first you might get a longer path than if a shorter path is traversed first because theif (updated) check will prevent the dependencies being traversed if a shorter or equal path already exists.

He-Pin and dispalt reacted with thumbs up emoji
@gzm0
Copy link
Contributor

TY for taking a stab at this. I'll start working through it.

For starters:

If a longer path with more exclusions is traversed first you might get a longer path than if a shorter path is traversed first because the if (updated) check will prevent the dependencies being traversed if a shorter or equal path already exists.

IIUCupdated is true even if only the hop counts changed:

But indeed, this means we might perform a traversal merely for the purpose of tracking the excluded hop count (which we will not need inFewestModules).

@steinybot
Copy link
Author

Yep, you are right. I was incorrectly thinking that it was possible to get it to not propagate the new excluded hop count to a previously visited path but that is impossible. It does mean that it has to traverse all paths to find the maximum but we can simply do that as a second pass without worrying about the actual path and blowing up the heap.

@gzm0gzm0 mentioned this pull requestMay 21, 2024
@steinybot
Copy link
Author

Some rough benchmarks with 6G of memory runningclient/clean;client/fullLinkJS with-Dsbt.task.timings=true:

[info]   client / Compile / fullLinkJS                                        :  4625 ms
  • This is after 5 runs. It starts at ~13s and gets faster every run.

I'm fairly confident that it is doing the right thing.

Here is what our modules look like on our master branch with 1.6.0 before I ran into this bug:
Screenshot 2024-05-21 at 1 20 49 PM

Here is after my big refactor which started triggering the OOMes with the changes in this PR:
Screenshot 2024-05-20 at 11 33 01 PM

@steinybotsteinybot changed the titleFix/4985 fewest modules oomFix fewest modules OOMMay 22, 2024
Comment on lines 206 to 209
// Revisit static dependencies again when the module id changes.
// We do not need to revisit dynamic dependencies because by definition they are not used by public
// modules, the first time we tag them is the shortest path so that path end never changes and only the
// ends are used for the module ids.
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

This is the key. If we revisit dynamic dependencies then it blows up. I'm pretty sure that this holds up. I tried coming with a counter example but I couldn't.

@steinybotsteinybot marked this pull request as ready for reviewJune 4, 2024 22:28
@steinybot
Copy link
Author

Dunno what's up with those two targets that fail to find the logs. Is that flaky?

@sjrd
Copy link
Member

sjrd commentedJun 5, 2024

Logs are automatically deleted after a while (one week I think?) because they're really huge and fill up our hard disks pretty quickly. The pass/fail information is kept and is still relevant. I'll rerun the tests so you get the logs back. If you want to look at them later, make a copy. ;)

steinybot reacted with thumbs up emoji

@steinybot
Copy link
Author

Ah Scalastyle. Pretty cryptic error messageexpected start of definition, but was Token(VAL,val,1619,val). I think it doesn't like 2locally blocks withvals with the same name 😞 .

@steinybot
Copy link
Author

steinybot commentedJun 5, 2024
edited
Loading

Oh nope there were trailing commas. Lovely...

assertNotEquals(initialStaticModuleID, finalStaticModuleID)
assertNotEquals(finalPublicModuleID, finalStaticModuleID)
// TODO: Why are there 3 modules and not 2?
assertNotEquals(finalDynamicModuleID, finalStaticModuleID)
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

I thought there should have been 2 modules but there are 3. Is that right?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Yes:

  • The public module with the public class
  • The dynamic module with the dynamic class
  • The core module with the shared dependencies of the public and the dynamic class (this cannot be in the public module, because it must be imported by the dynamic module).

steinybot reacted with thumbs up emoji
Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment
Reviewers

@gzm0gzm0gzm0 left review comments

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

FewestModules OOMs with dynamic dependencies
3 participants
@steinybot@gzm0@sjrd

[8]ページ先頭

©2009-2025 Movatter.jp