Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

[MAJOR VERSION] Async iterators#1135

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Merged
Siemienik merged 76 commits intoexceljs:masterfromalubbe:async-iterators
Jun 2, 2020

Conversation

alubbe
Copy link
Member

@alubbealubbe commentedFeb 18, 2020•
edited
Loading

This PR is the result of a lot of groundwork laid by previous PRs#1127#1125#829#1142#1190#1140#1139#1142

I'm happy to report that this PR adds 200 lines of documentation and deletes over 400 lines of code while keeping all tests green, maintaining performance and at the same time making our code base much easier to reason about and maintain.

I've thrown out everything related to streams & custom flow control logic around that and replaced it with generators (sync wherever possible, async when necessary). I exposed one external stream API at the end, built on the generators underneath.

This PR is now ready to be merged. It supports node v8 via the es5 import.

Here is the new usage:

constworkbookReader=newExcelJS.stream.xlsx.WorkbookReader('./file.xlsx');forawait(constworksheetReaderofworkbookReader){forawait(constrowofworksheetReader){// ...}}

staskolukasz, fmenis, ntpbnh15, and FITSteam reacted with thumbs up emojivitaliytv reacted with rocket emoji
@alubbe
Copy link
MemberAuthor

All the tests are green - I'm really excited what you think of this approach@Siemienik@guyonroche

If we like it, I would love to add support for it our streaming reader, as well, so that the consumers of our library can use for-await-of on the streams that we expose

... and make our streams destroyable without creating memory leaks ;)

@alubbe
Copy link
MemberAuthor

And here are the benchmarking numbers. First, currentmaster:

####################################################WARMUP: Current memory usage: 8.63 MBWARMUP: huge xlsx file profiling startedReading worksheet 1Reading row 50000Reading row 100000Reading worksheet 2Reading row 150000Processed 2 worksheets and 150002 rowsWARMUP: huge xlsx file profiling finished in 6961msWARMUP: Current memory usage (before GC): 134.96 MBWARMUP: Current memory usage (after GC): 51.7 MB####################################################RUN 1: huge xlsx file profiling startedReading worksheet 1Reading row 50000Reading row 100000Reading worksheet 2Reading row 150000Processed 2 worksheets and 150002 rowsRUN 1: huge xlsx file profiling finished in 6490msRUN 1: Current memory usage (before GC): 177.75 MBRUN 1: Current memory usage (after GC): 51.78 MB####################################################RUN 2: huge xlsx file profiling startedReading worksheet 1Reading row 50000Reading row 100000Reading worksheet 2Reading row 150000Processed 2 worksheets and 150002 rowsRUN 2: huge xlsx file profiling finished in 6683msRUN 2: Current memory usage (before GC): 148.7 MBRUN 2: Current memory usage (after GC): 51.67 MB####################################################RUN 3: huge xlsx file profiling startedReading worksheet 1Reading row 50000Reading row 100000Reading worksheet 2Reading row 150000Processed 2 worksheets and 150002 rowsRUN 3: huge xlsx file profiling finished in 6350msRUN 3: Current memory usage (before GC): 176.76 MBRUN 3: Current memory usage (after GC): 51.63 MB

And after this PR:

####################################################WARMUP: Current memory usage: 8.61 MBWARMUP: huge xlsx file profiling startedReading worksheet 1Reading row 50000Reading row 100000Reading worksheet 2Reading row 150000Processed 2 worksheets and 150002 rowsWARMUP: huge xlsx file profiling finished in 6823msWARMUP: Current memory usage (before GC): 148.66 MBWARMUP: Current memory usage (after GC): 41.22 MB####################################################RUN 1: huge xlsx file profiling startedReading worksheet 1Reading row 50000Reading row 100000Reading worksheet 2Reading row 150000Processed 2 worksheets and 150002 rowsRUN 1: huge xlsx file profiling finished in 6749msRUN 1: Current memory usage (before GC): 115.94 MBRUN 1: Current memory usage (after GC): 25.15 MB####################################################RUN 2: huge xlsx file profiling startedReading worksheet 1Reading row 50000Reading row 100000Reading worksheet 2Reading row 150000Processed 2 worksheets and 150002 rowsRUN 2: huge xlsx file profiling finished in 6653msRUN 2: Current memory usage (before GC): 97.41 MBRUN 2: Current memory usage (after GC): 25.12 MB####################################################RUN 3: huge xlsx file profiling startedReading worksheet 1Reading row 50000Reading row 100000Reading worksheet 2Reading row 150000Processed 2 worksheets and 150002 rowsRUN 3: huge xlsx file profiling finished in 6797msRUN 3: Current memory usage (before GC): 145.61 MBRUN 3: Current memory usage (after GC): 41.11 MB

In short, we have a slight slowdown (3-4%), but lower memory usage and much simpler control flow in our code, making it more maintainable.

@alubbealubbe mentioned this pull requestApr 8, 2020
Copy link
Member

@SiemienikSiemienik left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

ok, so we are waiting for v3.9.3 has been published

@alubbe
Copy link
MemberAuthor

I found and fixed another issue with the zip iteration (seeZJONSSON/node-unzipper#191) and have also deprecated CSV#createInputStream

@alubbe
Copy link
MemberAuthor

I've resolved all merge conflicts, this PR is now ready to be merged


it('should fail fast on a huge file',function(){
this.timeout(20000);
this.timeout(5000);
Copy link
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

This test is currently super flaky onmaster, this PR fixes it - and so confidently, that we can decrease the timeout from 20s to 5s

@alubbe
Copy link
MemberAuthor

@guyonroche can we please merge this?

Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment
Reviewers

@AlanscutAlanscutAlanscut left review comments

@SiemienikSiemienikSiemienik approved these changes

@guyonrocheguyonrocheAwaiting requested review from guyonroche

+2 more reviewers

@sohaisohaisohai left review comments

@AdrianSkierniewskiAdrianSkierniewskiAdrianSkierniewski left review comments

Reviewers whose approvals may not affect merge requirements
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

5 participants
@alubbe@Siemienik@sohai@AdrianSkierniewski@Alanscut
[8]ページ先頭
©2009-2025 Movatter.jp