Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

mpremote: Support bytecode raw paste for 'mpremote run module.mpy'#8744

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Conversation

projectgus
Copy link
Contributor

@projectgusprojectgus commentedJun 9, 2022
edited
Loading

For resource constrained devices withoutmpremote mount support, it is useful to be able to directly run precompiled bytecode (mpremote run) when developing, rather having to send source code each time.

Currently this is possible using pyboard.py - the implementation loads some loader Python and injects the bytecode as a variable. This approach reduces RAM usage further by using the "raw paste" window to paste into the same code path that loads and executes .mpy files (thanks Damien for this tip).

Summary of changes

  • Adds a new raw paste command 'B' for 'raw paste bytecode' (raw paste command 'A' is the current/default paste command)
  • Adds an escaping mechanism (<char> is escaped asCtrl-F<char + 8>) during raw paste so byte values less than 8 can be sent without triggering Ctrl-C or Ctrl-D handlers. The two-byte escape sequence still counts as one byte in the paste window.
  • Adds relevant support to mpremote.py
  • Adds the same "loader Python" approach to mpremote.py which is already in pyboard.py, for devices without raw paste support. Unclear how useful this is, I think it's only needed if using newer mpremote.py with older firmware, but there's no device-side changes and minimal host changes.

Impact

Building for PYBV11 with default settings, .text segment +128 bytes.

TODOs

  • My understanding might be wrong, but this still currently streams the entire module into a memory buffer (mp_raw_code_load, mp_make_function_from_raw_code) and then executes it ( mp_call_function_0). Maybe this can be made to execute it in chunks as each Python statement completes, meaning less RAM usage for modules which execute some statements on import and don't need to keep those around. But I'm very new to this and haven't dug right into it, could be way off. (I think the soft reset before run makes this a bit of a moot point, although there are still potential savings for 'run'-ing a bytecode that doesn't define any code.)
  • Set up some protocol macro defines in the code so all of the possible command sequences are grouped together, to make the protocol easier to understand from the top down.
  • Test this works withMICROPY_REPL_EVENT_DRIVEN (Seems only JavaScript port uses this method, at least by default, and it doesn't support mpremote - if I should test this on a different port, let me know.)
  • Test this works with 'Ctrl-K inject file'
  • Set up compilation guards for ports which don't support .mpy loading
  • Check for regressions due to the new escape sequences anywhere that pyboard.py calls raw paste routines
  • Implement sending of bytecode without raw paste in mpremote (can add the 'injected variable' approach from pyboard.py into mpremote)
  • Measure actual memory usage (currently testing on a board with no mem_info)
  • Check for memory leaks (I'm not yet across how this rawcode buffer gets cleaned up if you load another one...)

Compatibility

I believe this approach is compatible between different mpremote.py and MicroPython versions:

  • Newer mpremote.py will detect that older MicroPython can't support the "bytecode raw paste" command ("B")
  • Old mpremote.py will not try to send this data at all.

Except for the case of anyone who was sending a raw Ctrl-F character as-is in a raw paste, in which case that will stop working as expected if the mprempote.py + firmware versions don't match (due to the Ctrl-F escape handler).

@projectgus
Copy link
ContributorAuthor

@jimmo@andrewleech@dpgeorge Do you folks have any suggestions about this?

@dpgeorgedpgeorge added py-coreRelates to py/ directory in source toolsRelates to tools/ directory in source, or other tooling labelsJun 9, 2022
@dpgeorge
Copy link
Member

I think@andrewleech had an idea (and maybe somewhere in a PR) to build .mpy files on the fly in mpremote (calling out to mpy-cross). That would fit well here.

projectgus reacted with rocket emoji

@codecov-commenter
Copy link

Codecov Report

Merging#8744 (a933f69) intomaster (5bb2a85) willdecrease coverage by0.00%.
The diff coverage isn/a.

@@            Coverage Diff             @@##           master    #8744      +/-   ##==========================================- Coverage   98.31%   98.31%   -0.01%==========================================  Files         155      156       +1       Lines       20304    20326      +22     ==========================================+ Hits        19962    19983      +21- Misses        342      343       +1
Impacted FilesCoverage Δ
py/compile.c99.69% <0.00%> (-0.07%)⬇️
py/objmodule.c100.00% <0.00%> (ø)
extmod/modurandom.c100.00% <0.00%> (ø)
ports/unix/moduos.c18.91% <0.00%> (ø)
ports/unix/mpconfigport.h100.00% <0.00%> (ø)

Continue to review full report at Codecov.

Legend -Click here to learn more
Δ = absolute <relative> (impact),ø = not affected,? = missing data
Powered byCodecov. Last update5bb2a85...a933f69. Read thecomment docs.

@andrewleech
Copy link
Contributor

I've got an open draft MR which adds support for manifest files to mpremote - with this any python files referenced in a manifest.py are bytecompiled into a local folder with mpy-cross, exposed via mount, then added to sys.path so that the device can easily import everything.
I want to refactor this such that when using regularmpremote mount, any time the device tries to importblah.mpy from the mount, mpremote would check forblay.py, mpy-cross it, then deliver it to the device instead.

So this would reduce ram use and increase import speed (less data to transfer) but still requires mount. These changes wouldn't need any new support within micropython either, but don't address all of your needs - specifically directly running an mpy file without mount support.

Would it be feasible to include vfs support in a build without any specific filesystem libraries? That way mount could be used without needing the bulk of any one of the fs formats. That being said, while I haven't looked at your code in detail I presume a raw paste mode can be added with less code than the VFS framework.

I haven't looked into the ram buffer size needs while importing a mpy file over mount but expect that bytecode would be read "line by line" so to speak by the vm? I don't know if it buffers the entire mpy file in ram when running from a (mount) filesystem?

@peterhinch
Copy link
Contributor

I want to refactor this such that when using regular mpremote mount, any time the device tries to import blah.mpy from the mount, mpremote would check for blay.py, mpy-cross it, then deliver it to the device instead.

That sounds awesome. Well worth doing!

A problem could occur with "incompatible mpy format" messages. Would it be possible to check the firmware version on the target against the cross-compiler version? A mismatch could prevent pre-compilation and issue a warning.

@projectgusprojectgusforce-pushed thefeature/mpremote_run_mpy branch froma933f69 to717a1bdCompareJuly 6, 2022 23:26
@projectgusprojectgus changed the titlempremote: Experiment with bytecode raw paste for 'mpremote run module.mpy'mpremote: Support bytecode raw paste for 'mpremote run module.mpy' and 'mpremote repl --inject-file'Jul 6, 2022
@projectgusprojectgus changed the titlempremote: Support bytecode raw paste for 'mpremote run module.mpy' and 'mpremote repl --inject-file'mpremote: Support bytecode raw paste for 'mpremote run module.mpy'Jul 6, 2022
@projectgusprojectgusforce-pushed thefeature/mpremote_run_mpy branch from717a1bd to249cb20CompareJuly 7, 2022 02:16
- Adds a new raw paste command 'B' for 'raw paste bytecode' ('A' is'paste source')- Adds an escaping mechanism (Ctrl-F <char + 8>) during raw paste sobytes less than 8 can be sent without triggering Ctrl-C or Ctrl-Dhandlers. The two-byte escape sequence still counts as one byte inthe paste window.- Adds relevant support to mpremote.pySigned-off-by: Angus Gratton <gus@projectgus.com>
Signed-off-by: Angus Gratton <gus@projectgus.com>
Signed-off-by: Angus Gratton <gus@projectgus.com>
The raw repl language is gradually getting more complex. To structurethe code a bit more, introduce some names for the different controlcharacters and the "init command" sequence which is currentlyonly used for starting a paste.Signed-off-by: Angus Gratton <gus@projectgus.com>
@projectgusprojectgusforce-pushed thefeature/mpremote_run_mpy branch from249cb20 to02826f0CompareJuly 7, 2022 02:49
@projectgus
Copy link
ContributorAuthor

Hi@andrewleech,

Sorry I didn't reply to these earlier on:

Would it be feasible to include vfs support in a build without any specific filesystem libraries? That way mount could be used without needing the bulk of any one of the fs formats. That being said, while I haven't looked at your code in detail I presume a raw paste mode can be added with less code than the VFS framework.

This is a good question! Actually the target I'm focusing on, B_L072Z_LRWAN1 (32KB RAM), has the necessary base vfs support, but on master it runs out of memory whenmpremote mount tries to push the_fs_hook_code which implements the dummy filesystem! This is 5124 bytes of Python source.

With this change and ahacky patch to pre-compile the _fs_hook_code on top then I can dompremote mount successfully, but it still uses almost half the available heap once the FS class is loaded:

❯ mpremote mount .Local directory . is mounted at /remoteConnected to MicroPython at /dev/ttyACM0Use Ctrl-] to exit this shell>MicroPython v1.19.1-106-g249cb207d-dirty on 2022-07-07; B-L072Z-LRWAN1 with STM32L072CZType "help()" for more information.>>> import gc, micropython; gc.collect(); micropython.mem_info()stack: 588 out of 3072GC: total: 12096, used: 5360, free: 6736 No. of 1-blocks: 76, 2-blocks: 11, max blk sz: 19, max free sz: 279

Compared to a plain REPL:

❯ mpremoteConnected to MicroPython at /dev/ttyACM0Use Ctrl-] to exit this shellMicroPython v1.19.1-106-g249cb207d-dirty on 2022-07-07; B-L072Z-LRWAN1 with STM32L072CZType "help()" for more information.>>> import gc, micropython; gc.collect(); micropython.mem_info()stack: 588 out of 3072GC: total: 12096, used: 352, free: 11744 No. of 1-blocks: 6, 2-blocks: 2, max blk sz: 5, max free sz: 725

So I'm thinking "mpremote mount" may not ever be viable on these very small systems.

I haven't looked into the ram buffer size needs while importing a mpy file over mount but expect that bytecode would be read "line by line" so to speak by the vm? I don't know if it buffers the entire mpy file in ram when running from a (mount) filesystem?

Honestly I don't fully understand this yet, I'm going to take another look through it soon.

@projectgus
Copy link
ContributorAuthor

projectgus commentedJul 7, 2022
edited
Loading

I haven't looked into the ram buffer size needs while importing a mpy file over mount but expect that bytecode would be read
"line by line" so to speak by the vm? I don't know if it buffers the entire mpy file in ram when running from a (mount) filesystem?

My reading of persistentcode.c is that it reads all of the byte code into memory before it executes. This seems probably necessary to keep complexity down as the bytecode may contain jumps, calls to child scopes, etc. that don't happen linearly.

One interesting thing I noticed is that the "outer scope" of bytecode (i.e. the code which is executed as the module imports) is kept in memory after the execution/import completes (for any call that goes viamp_raw_code_load, which includes both importing from a source or bytecode file orrepl --inject-file). As far as I can tell, this bytecode isn't needed again so therc->fun_data buffer could be freed after execution/import completes.

Downside, in most cases freeing this won't save a lot - for example in_fs_code_hook it's 75 bytes of bytecode out of a 3000 byte file. It'd only be significant for modules which execute a lot of one-time code on import.

@dpgeorge@jimmo Does that sound right, or am I missing something?

@projectgusprojectgus marked this pull request as ready for reviewJuly 8, 2022 00:13
RetiredWizard pushed a commit to RetiredWizard/micropython that referenced this pull requestDec 30, 2023
@projectgus
Copy link
ContributorAuthor

This is an automated heads-up that we've just merged a Pull Request
that removes the STATIC macro from MicroPython's C API.

See#13763

A search suggests this PR might apply the STATIC macro to some C code. If it
does, then next time you rebase the PR (or merge from master) then you should
please replace all theSTATIC keywords withstatic.

Although this is an automated message, feel free to @-reply to me directly if
you have any questions about this.

Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
py-coreRelates to py/ directory in sourcetoolsRelates to tools/ directory in source, or other tooling
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

5 participants
@projectgus@dpgeorge@codecov-commenter@andrewleech@peterhinch

[8]ページ先頭

©2009-2025 Movatter.jp