Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

Integrated handling for filesystem and gzipped binaries#8266

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Closed
gneiss15 wants to merge25 commits intoesp8266:masterfromgneiss15:Integrated-handling-for-filesystem-and-gzipped-Binaries
Closed

Integrated handling for filesystem and gzipped binaries#8266

gneiss15 wants to merge25 commits intoesp8266:masterfromgneiss15:Integrated-handling-for-filesystem-and-gzipped-Binaries

Conversation

@gneiss15
Copy link
Contributor

ATENTION: This is tested on Linux only! See below for open questions/prerequisites.

Currently there are some manual steps necessary to prepare the files necessary for OTA updates (especially for ESP8266HTTPUpdateServer)

The steps are:

  1. Compile the sketch
  2. Export the generated sketch-binary
  3. Generate the filesystem
  4. Export the fs-binary
  5. Generate gzipped versions of binaries
  6. Generate signed versions of gzipped binaries

Which of the above steps are necessary depends on the sketch (does it use a fs?) and the kind of OTA chosen (signed binaries necessary?) or preferred usage (use of gzipped binaries for faster upload?).

There are some problems/inconveniences:
2. The signed sketch-binary is not exported when using "Export compiled Binary".
(This point will be solved with my Pull-Request#8255)
3. & 4. AFAIK, the only “integrated” way to generate the fs is using one of the tools:
“arduino-esp8266littlefs-plugin” or “arduino-esp8266fs-plugin”.
Both tools are designed to create & upload the fs, but not export them.
So if preparing the files for an OTA when no board is connected, you have to:

  • ignore errors shown because upload is not possible
  • find & navigate to the “build.path” and copy the fs-binary manually
  1. & 6. AFAIK, there is no “integrated” way to perform these steps.

This Pull-Request will integrate all of the above steps into the “compile and/or upload” action of Arduino.
This is done by adding two new entries (“Filesystem” and “Binaries”) to the tools-menu.
These entries are presented just like any other options for esp8266 (“Upload speed” ... “Non-32-Bit Access”) and placed right after the last.

Menu-Entry “Filesystem” let You choose between:

  • Off
  • LitteFs: Create & Upload
  • LitteFs: Create only
  • SPIFFS: Create & Upload
  • SPIFFS: Create only

Menu-Entry “Binaries” let You choose between:

  • Off
  • Export (.bin & .bin.signed)
  • Create & Export gzipped Binaries too

Now You can choose with of the steps mentioned above will be automatically performed.
After a (successful) compile, the files will be exported to a subdir “bin/{variant}” of Your sketch directory (“{variant}” replaced with the name of the board as shown behind “Tools -> Board:”).
Signed variants of all Binaries are gererated similar to the “automatic signing” done for normal sketch binary.

Open questions/prerequisites:
As stated above I had tested this on Linux only.
Even under this os there are some prerequisites:

  • gzip must be available (inside path)
    if it is not, a message is given and creating of gzipped binaries is skipped.
  • tkinter should be available
    if it is not, and creating of fs is requested AND the data dir is empty, I didn't know how to ask the user: “Are you sure you want to create an empty ... image?”.
    So in this case a message is given and the fs will not be created.

This “should” work under Windows/MacOs the same, but can't be tested by me (I had no PC with this OSes)

An additional question for developers:
As mentioned inside function main of “postbuild.py”, there is a strange behavior of the arg "{build.path}" supplied via “platform.txt”. If someone can explain me why this “string” behave as described in the comments, I would be happy ;-)

gneiss15and others added7 commitsAugust 2, 2021 16:53
The current platform.txt doesn't define the entries to copy signed binaries when performing a "Export compiled Binary".This pull request adds the nessesary enties.
ATENTION: This is tested on Linux only! See below for open questions/prerequisites.Currently there are some manual steps necessary to prepare the files necessary for OTA updates (especially for ESP8266HTTPUpdateServer)The steps are:1. Compile the sketch2. Export the generated sketch-binary3. Generate the filesystem4. Export the fs-binary5. Generate gzipped versions of binaries6. Generate signed versions of gzipped binariesWhich of the above steps are necessary depends on the sketch (does it use a fs?) and the kind of OTA chosen (signed binaries necessary?) or preferred usage (use of gzipped binaries for faster upload?).There are some problems/inconveniences:2. The signed sketch-binary is not exported when using "Export compiled Binary".(This point will be solved with my Pull-Request#8255)3. & 4. AFAIK, the only “integrated” way to generate the fs is using one of the tools:“arduino-esp8266littlefs-plugin” or  “arduino-esp8266fs-plugin”.Both tools are designed to create & upload the fs, but not export them.So if preparing the files for an OTA when no board is connected, you have to:- ignore errors shown because upload is not possible- find & navigate to the “build.path” and copy the fs-binary manually5. & 6. AFAIK, there is no “integrated” way to perform these steps.This Pull-Request will integrate all of the above steps into the “compile and/or upload” action of Arduino.This is done by adding two new entries (“Filesystem” and “Binaries”) to the tools-menu.These entries are presented just like any other options for esp8266 (“Upload speed” ... “Non-32-Bit Access”) and placed right after the last.Menu-Entry “Filesystem” let You choose between:- Off- LitteFs: Create & Upload- LitteFs: Create only- SPIFFS: Create & Upload- SPIFFS: Create onlyMenu-Entry “Binaries” let You choose between:- Off- Export (.bin & .bin.signed)- Create & Export gzipped Binaries tooNow You can choose with of the steps mentioned above will be automatically performed.After a (successful) compile, the files will be exported to a subdir “bin/{variant}” of Your sketch directory (“{variant}” replaced with the name of the board as shown behind “Tools -> Board:”).Signed variants of all Binaries are gererated similar to the “automatic signing” done for normal sketch binary.Open questions/prerequisites:As stated above I had tested this on Linux only.Even under this os there are some prerequisites:- gzip must be available (inside path)if it is not, a message is given and creating of gzipped binaries is skipped.- tkinter should be availableif it is not, and creating of fs is requested AND the data dir is empty, I didn't know how to ask the user: “Are you sure you want to create an empty ... image?”.So in this case a message is given and the fs will not be created.This “should” work under Windows/MacOs the same, but can't be tested by me (I had no PC with this OSes)An additional question for developers:As mentioned inside function main of “postbuild.py”, there is a strange behavior of the arg "{build.path}" supplied via “platform.txt”. If someone can explain me why this “string” behave as described in the comments, I would be happy ;-)
@gneiss15
Copy link
ContributorAuthor

gneiss15 commentedAug 13, 2021
edited
Loading

Ok, bug found. It seams, that the python version used is a bit more strict (than mine) and imports only if there are no errors inside. There was an ident error inside utilities.py that must have raise the error.

@gneiss15
Copy link
ContributorAuthor

A try to get someone review this.
Acording tohttps://webapps.stackexchange.com/questions/29197/any-way-to-contact-a-user-on-github/45733#45733 this should work..
@earlephilhower Can You please review this ?
@d-a-v Can You please review this ?

@d-a-v
Copy link
Collaborator

d-a-v commentedAug 18, 2021
edited
Loading

edit Thanks for allowing to automate these steps !

gzip must be available (inside path)

Can you replace this dependancy by some python code ?

tkinter should be available
if it is not, and creating of fs is requested AND the data dir is empty, I didn't know how to ask the user: “Are you sure you want to create an empty ... image?”.

I'm not sure tkinter can be easily provided by the embedded python. Why not bailing out with an error when FS is empty, and/or add a another menu entry for empty FS ?

@gneiss15
Copy link
ContributorAuthor

gneiss15 commentedAug 18, 2021
edited
Loading

Thanks for reviewing this.

I'm not sure tkinter can be easily provided by the embedded python.

Yes it can and I show how to install it in the message given if it's not already installed... see line 84 of utilities.py

Why not bailing out...

That's what I already do. I give the message "can't ask because..." and then simply skip generation of the fs.

gzip .. Can you replace this dependancy by some python code ?

Yeah, that's a good point.
I didn't know that python already has a buildin module for gzip files.
Tried it now and found that this can be used.
I found, that files generated with the module differ a bit from the one generated by the linux gzip, but if they are generated by one and extracted by the other the result is always the original file.
So I had uploaded this change right now..

gneiss15and others added3 commitsAugust 18, 2021 14:27
* Integrated handling for filesystem and gzipped BinariesATENTION: This is tested on Linux only! See below for open questions/prerequisites.Currently there are some manual steps necessary to prepare the files necessary for OTA updates (especially for ESP8266HTTPUpdateServer)The steps are:1. Compile the sketch2. Export the generated sketch-binary3. Generate the filesystem4. Export the fs-binary5. Generate gzipped versions of binaries6. Generate signed versions of gzipped binariesWhich of the above steps are necessary depends on the sketch (does it use a fs?) and the kind of OTA chosen (signed binaries necessary?) or preferred usage (use of gzipped binaries for faster upload?).There are some problems/inconveniences:2. The signed sketch-binary is not exported when using "Export compiled Binary".(This point will be solved with my Pull-Request#8255)3. & 4. AFAIK, the only “integrated” way to generate the fs is using one of the tools:“arduino-esp8266littlefs-plugin” or  “arduino-esp8266fs-plugin”.Both tools are designed to create & upload the fs, but not export them.So if preparing the files for an OTA when no board is connected, you have to:- ignore errors shown because upload is not possible- find & navigate to the “build.path” and copy the fs-binary manually5. & 6. AFAIK, there is no “integrated” way to perform these steps.This Pull-Request will integrate all of the above steps into the “compile and/or upload” action of Arduino.This is done by adding two new entries (“Filesystem” and “Binaries”) to the tools-menu.These entries are presented just like any other options for esp8266 (“Upload speed” ... “Non-32-Bit Access”) and placed right after the last.Menu-Entry “Filesystem” let You choose between:- Off- LitteFs: Create & Upload- LitteFs: Create only- SPIFFS: Create & Upload- SPIFFS: Create onlyMenu-Entry “Binaries” let You choose between:- Off- Export (.bin & .bin.signed)- Create & Export gzipped Binaries tooNow You can choose with of the steps mentioned above will be automatically performed.After a (successful) compile, the files will be exported to a subdir “bin/{variant}” of Your sketch directory (“{variant}” replaced with the name of the board as shown behind “Tools -> Board:”).Signed variants of all Binaries are gererated similar to the “automatic signing” done for normal sketch binary.Open questions/prerequisites:As stated above I had tested this on Linux only.Even under this os there are some prerequisites:- gzip must be available (inside path)if it is not, a message is given and creating of gzipped binaries is skipped.- tkinter should be availableif it is not, and creating of fs is requested AND the data dir is empty, I didn't know how to ask the user: “Are you sure you want to create an empty ... image?”.So in this case a message is given and the fs will not be created.This “should” work under Windows/MacOs the same, but can't be tested by me (I had no PC with this OSes)An additional question for developers:As mentioned inside function main of “postbuild.py”, there is a strange behavior of the arg "{build.path}" supplied via “platform.txt”. If someone can explain me why this “string” behave as described in the comments, I would be happy ;-)* Forgot to add new tools to git* Minor bugfix* Debug for failed check in Pull-Request* Ignore files generated by boards.txt.py & my extra files* More Debug for failed check in Pull-Request* Bugfix for check* And another bigfix :-(* Changed mod a+x (as all other .py)* Syntax error in utilities.py may be the reason for failed import. Correted* instead of external commd, use python module gzip
…pped-BinariesConflicts:tools/postbuild.pytools/utillities.py
@gneiss15
Copy link
ContributorAuthor

gneiss15 commentedAug 18, 2021
edited
Loading

I made the info shown if tkinter was not installed a bit better, so I hope that's now all OK
@d-a-v Can You please review this again?

@JAndrassy
Copy link
Contributor

this is PR8266 :-)

d-a-v reacted with thumbs up emoji

@d-a-v
Copy link
Collaborator

On the way to #8285 (#8284 is the current last one)

@gneiss15
Copy link
ContributorAuthor

@d-a-v@earlephilhower: I thing that (part off) my original comment should be integrated into the documentation of the esp toolkit (when this PR was merged) and although mentioned in the tools “arduino-esp8266littlefs-plugin” or “arduino-esp8266fs-plugin”.
I did not know how this can be done. Can someone either guide me or do it by himself ?

@d-a-v
Copy link
Collaborator

You can integrate documentation changes as part of your pull request. Everything is indoc/

Added description of changes to docu
@gneiss15
Copy link
ContributorAuthor

@d-a-v I had done as suggested + make some improvements inspired by some of my fiends.

@gneiss15
Copy link
ContributorAuthor

@d-a-v: Is there any reason why this isn't merged till now ?
If so, please let me know what I should do

Copy link
Collaborator

@d-a-vd-a-v left a comment
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

There is a lot of work in this PR, thanks for the feature addition !

There are some unrelated changes which should not be part of it : some breaking or unexplained changes and also a softwareserial commit which should be removed.


Uploading files to file system
------------------------------
Uploading files to file system (new)
Copy link
Collaborator

@d-a-vd-a-vSep 14, 2021
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

(new) =>(combined firmware and filesystem)

Copy link
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

OK; good point (Thanks)

Copy link
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Hm, I don't understand what happend here. You changed this, but inside the "resolved" view I still see my old version ???
I prefer You Version (with "combined firmware and filesystem" instead of "new"), but that's not shown here ???

Copy link
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

OK, because that seams to be true for all such comments here, I integrate theses changes into my local copy and commit theses changes later..
If this isn't the right way, please let me know.

# Next, do the binaryupload
# Combined in one rule because Arduino doesn't supportupload.1.pattern/upload.3.pattern
tools.esptool.upload.pattern="{cmd}" -I "{runtime.platform.path}/tools/upload.py" --chip esp8266 --port"{serial.port}" --baud "{upload.speed}" "{upload.verbose}"{upload.erase_cmd} {upload.resetmethod} write_flash 0x0 "{build.path}/{build.project_name}.bin"
tools.esptool.upload.network_pattern="{network_cmd}" -I "{runtime.platform.path}/tools/espota.py" -i "{serial.port}" -p "{network.port}" "--auth={network.password}" -f "{build.path}/{build.project_name}.bin"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Is removingespota.py really part of this PR ?
This one is very sensitive and any change must be backward compatible.
It must be part of another dedicated PR.

Copy link
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

It's not removed... See You comment above (about old behaviour) there espota.py is still contained.
It's only modified to additionally upload the fs within one call to espota !

#!/usr/bin/env python3

# Network-Upload-Wrapper
# 2021-08-08: G.N.: Written to support (optional) uploading of sketch & filesystem in one step
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Should be part of another PR

Copy link
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

No, that is an essential part of this PR.
It does the combined upload when using netupload instead of espota, so it MUST be part of this PR.

returnpath

defTkDialog(title,text,mbName ):
try:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Is it a leftover ?

Copy link
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

No, it is the base of (for ex.) ConfirmDialog & WarningDialog which are used inside postbuild.py

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Anything with python must be cross-compatible with all archs.
A portable version of python is shipped by the installer and I doubt tkinter will work in that case.

@gneiss15
Copy link
ContributorAuthor

gneiss15 commentedSep 15, 2021
edited
Loading

@d-a-v wrotes: There are some unrelated changes which should not be part of it : some breaking or unexplained changes and also a softwareserial commit which should be removed.
Sorry, maybe I misunderstood something (git isn't my favorite VCS, I am more trained to use Subversion).
The "unrelated changes" are from my try to keep this repro in sync with the master branch. This is the reason for the softwareserial part.. As far as I know, I had done it correct, but maybe I am wrong. If so, can You pleas help me to correct this ?

Because I am although a bit confused what "resolve conversation" above means/does.. I had done:

  • Integrate changes You suggested into my local copy. These changes I will commit soon.
  • Changes I had signed "not my change" (or similar) I had ignored for now, these changes happens because i made a pull from master.... Maybe I had done this wrong, but I don't know how to revert...

Some of the changes that are shown inside my branch are although shown inside the main repro (as already committed), but when comparing my branch against the master these changes are shown a additional changes ??? confusing..

master...gneiss15:Integrated-handling-for-filesystem-and-gzipped-Binaries

@gneiss15
Copy link
ContributorAuthor

gneiss15 commentedSep 15, 2021
edited
Loading

OK, I hope I had found the correct way of removing unwanted changes.
I clone the master repro, compare it with my local repro and remove any changes that aren't mine. I had committed the result now...

As far as I could see, the only "open" difference is the removal of "SoftwareSerial" here I don't know how to revert that..

@d-a-v
Copy link
Collaborator

Reviewing is made against current core's source (master branch).
Can you merge master's head with your branch and re-push ?
About the CI error, you should notgit rm libaries/SoftwareSerial but simply update it (withgit submodule update --init).

@gneiss15
Copy link
ContributorAuthor

@d-a-v:

Reviewing is made against current core's source (master branch).
Yes, that's clear to me, but what is a bit curious is, that I had done a pull of the master branch, then these changes are integrated into my repo. Now these changes seams to be no more inside the master repro (was this repro rolled back ?)

Can you merge master's head with your branch and re-push ?
It seams that does not work, the changes are still there..

About the CI error, you should not git rm libaries/SoftwareSerial but simply update it (with git submodule update --init).
Does nothing ??

All this trouble comes from the merge mentioned above (05.09) with tag8c06e83 which now links to a "page not found" ???

I still have no idea how to resolve the mentioned problems...

Should I remove my whole repro and redo all changes ?

@d-a-v
Copy link
Collaborator

If you are not familiar with git, best is probably to reapply your changes to a fresh master.

@gneiss15
Copy link
ContributorAuthor

gneiss15 commentedSep 19, 2021
edited by d-a-v
Loading

OK, I will do so (reapply your changes to a fresh master). Unfortunately that means I have to make a new fork... I will open a new pull request when I had done this..
Just one additionally remark about Your comment above:

A portable version of python is shipped by the installer and I doubt tkinter will work in that case.

No, tkinter is not shipped with this python, it must be installed separate (at least under linux)

@d-a-v
Copy link
Collaborator

No, tkinter is not shipped with this python, it must be installed separate (at least under linux)

That's the reason why such change can't be accepted by maintainers.
Is there another solution with menus ?

@gneiss15
Copy link
ContributorAuthor

gneiss15 commentedSep 21, 2021
edited
Loading

As I already mentioned the presense of tkinter is not mandatory. If it isn't installed I simply can't ask the user what is his choice in the situation when the FS is/will be empty. I then simply give a message and don't create the FS.
So I seen no reason why this can't be accepted. It just an option that can be ignored.

BTW.: It will take some time to create the new repro (I am away for some days)..

@gneiss15
Copy link
ContributorAuthor

I close this now, to open a new one with a new fresh fork

Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment

Reviewers

@d-a-vd-a-vd-a-v requested changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@gneiss15@d-a-v@JAndrassy

[8]ページ先頭

©2009-2025 Movatter.jp