- Notifications
You must be signed in to change notification settings - Fork13.3k
Tools - mkbuildoptglobals refactoring & attempt to fix caching#9249
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
Open
mcspr wants to merge22 commits intoesp8266:masterChoose a base branch frommcspr:buildopts/refactoring
base:master
Could not load branches
Branch not found:{{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline, and old review comments may become outdated.
Uh oh!
There was an error while loading.Please reload this page.
Open
Changes fromall commits
Commits
Show all changes
22 commits Select commitHold shift + click to select a range
8253d94 Tools - mkbuildoptglobals refactoring & attempt to fix caching
mcspr8cdc906 compile args typo
mcspreac15af build_out & no explicit clean
mcspr60689bc drop partial data, handle missing closing comment
mcspr9be7e67 check cache state dir state in ci
mcsprc3b64a2 missing file
mcspra527244 typo, fix err trap for "test ...something..."
mcsprfe3102f mention indirect and direct dependency on globals .h contents
mcsprd72ca55 update doc
mcspr35c6221 update docs
mcsprd1c9c7b restore command-line opts info, newlined debugging
mcspra3f9218 mkdir parent when writing
mcspr56bac44 useless from None
mcspreb1a542 multiple quoting fixes from sc & naming
mcspra9afc33 spacing
mcspra91da86 $cache_dir & $root quoting
mcspr7f06e65 debug flag passed as fqbn
mcspr664e9d8 direct types
mcspr78380e8 warnings fix v2, propogate errors instead
mcspr2e63e3e debugging in ci tool checks
mcspre023926 invalid debug lvl=
mcspr0c67f33 rebase unused var
mcsprFile filter
Filter by extension
Conversations
Failed to load comments.
Loading
Uh oh!
There was an error while loading.Please reload this page.
Jump to
Jump to file
Failed to load files.
Loading
Uh oh!
There was an error while loading.Please reload this page.
Diff view
Diff view
There are no files selected for viewing
2 changes: 1 addition & 1 deletion.github/workflows/build-host.yml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
23 changes: 21 additions & 2 deletions.github/workflows/build-ide.yml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
150 changes: 150 additions & 0 deletionsdoc/buildopt/aggressive-caching.rst
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,150 @@ | ||
| :orphan: | ||
| Aggressively cache compiled core | ||
| ================================ | ||
| .. attention:: | ||
| This article applies ONLY to IDE 1.x and original version | ||
| of the mkbuildoptglobals.py script shipped with Core 3.1.x | ||
| This feature appeared with the release of Arduino IDE 1.8.2. The feature | ||
| “Aggressively Cache Compiled core” refers to sharing a single copy of | ||
| ``core.a`` across all Arduino IDE Sketch windows. This feature is on by | ||
| default. ``core.a`` is an archive file containing the compiled objects | ||
| of ``./core/esp8266/*``. Created after your 1ST successful compilation. | ||
| All other open sketch builds use this shared file. When you close all | ||
| Arduino IDE windows, the core archive file is deleted. | ||
| This feature is not compatible with using global defines or compiler | ||
| command-line options. Without mediation, bad builds could result, when | ||
| left enabled. When ``#define`` changes require rebuilding ``core.a`` and | ||
| multiple Sketches are open, they can no longer reliably share one cached | ||
| ``core.a``. In a simple case: The 1st Sketch to be built has its version | ||
| of ``core.a`` cached. Other sketches will use this cached version for | ||
| their builds. | ||
| There are two solutions to this issue: | ||
| 1. Do nothing, and rely on aggressive cache workaround built into the | ||
| script. | ||
| 2. Turn off the “Aggressively Cache Compiled core” feature, by setting | ||
| ``compiler.cache_core=false``. | ||
| Using “compiler.cache_core=false” | ||
| --------------------------------- | ||
| There are two ways to turn off the “Aggressively Cache Compiled core” | ||
| feature: This can be done with the Arduino IDE command-line or a text | ||
| editor. | ||
| Using the Arduino IDE command-line from a system command line, enter the | ||
| following: | ||
| .. code-block:: console | ||
| $ arduino --pref compiler.cache_core=false --save-prefs | ||
| For the text editor, you need to find the location of | ||
| ``preferences.txt``. From the Arduino IDE, go to *File->Preferences*. | ||
| Make note of the path to ``prefereces.txt``. You *cannot* edit the file | ||
| while the Arduino IDE is running. Close all Arduino IDE windows and edit | ||
| the file ``preferences.txt``. Change ``compiler.cache_core=true`` to | ||
| ``compiler.cache_core=false`` and save. Then each sketch will maintain | ||
| its *own* copy of ``core.a`` built with the customization expressed by | ||
| their respective ``build.opt`` file. | ||
| The “workaround” | ||
| ---------------- | ||
| When the “Aggressively Cache Compiled core” feature is enabled and the | ||
| global define file is detected, a workaround will turn on and stay on. | ||
| When you switch between Sketch windows, core will be recompiled and the | ||
| cache updated. The workaround logic is reset when Arduino IDE is | ||
| completely shutdown and restarted. | ||
| The workaround is not perfect. These issues may be of concern: | ||
| 1. Dirty temp space. Arduino build cache files left over from a previous | ||
| run or boot. | ||
| 2. Arduino command-line options: | ||
| - override default preferences.txt file. | ||
| - override a preference, specifically ``compiler.cache_core``. | ||
| 3. Multiple versions of the Arduino IDE running | ||
| **Dirty temp space** | ||
| A minor concern, the workaround is always on. Not an issue for build | ||
| accuracy, but ``core.a`` maybe rebuild more often than necessary. | ||
| Some operating systems are better at cleaning up their temp space than | ||
| others at reboot after a crash. At least for Windows®, you may need to | ||
| manually delete the Arduino temp files and directories after a crash. | ||
| Otherwise, the workaround logic may be left on. There is no harm in the | ||
| workaround being stuck on, the build will be correct; however, the core | ||
| files will occasionally be recompiled when not needed. | ||
| For some Windows® systems the temp directory can be found near | ||
| ``C:\Users\<user id>\AppData\Local\Temp\arduino*``. Note ``AppData`` is | ||
| a hidden directory. For help with this do an Internet search on | ||
| ``windows disk cleanup``. Or, type ``disk cleanup`` in the Windows® | ||
| taskbar search box. | ||
| With Linux, this problem could occur after an Arduino IDE crash. The | ||
| problem would be cleared after a reboot. Or you can manually cleanup the | ||
| ``/tmp/`` directory before restarting the Arduino IDE. | ||
| **Arduino command-line option overrides** | ||
| If you are building with ``compiler.cache_core=true`` no action is | ||
| needed. If ``false`` the script would benefit by knowing that. | ||
| When using either of these two command-line options: | ||
| .. code-block:: console | ||
| $ arduino --preferences-file other-preferences.txt | ||
| $ arduino --pref compiler.cache_core=false | ||
| Hints for discovering the value of ``compiler.cache_core``, can be | ||
| provided by specifying ``mkbuildoptglobals.extra_flags=...`` in | ||
| ``platform.local.txt``. | ||
| Examples of hints: | ||
| .. code-block:: ini | ||
| mkbuildoptglobals.extra_flags=--preferences_sketch # assume file preferences.txt in the sketch folder | ||
| mkbuildoptglobals.extra_flags=--preferences_sketch "pref.txt" # is relative to the sketch folder | ||
| mkbuildoptglobals.extra_flags=--no_cache_core | ||
| mkbuildoptglobals.extra_flags=--cache_core | ||
| mkbuildoptglobals.extra_flags=--preferences_file "other-preferences.txt" # relative to IDE or full path | ||
| If required, remember to quote file or file paths. | ||
| Multiple versions of the Arduino IDE running | ||
| -------------------------------------------- | ||
| You can run multiple Arduino IDE windows as long as you run one version | ||
| of the Arduino IDE at a time. When testing different versions, | ||
| completely exit one before starting the next version. For example, | ||
| Arduino IDE 1.8.19 and Arduino IDE 2.0 work with different temp and | ||
| build paths. With this combination, the workaround logic sometimes fails | ||
| to enable. | ||
| At the time of this writing, when Arduino IDE 2.0 rc5 exits, it leaves | ||
| the temp space dirty. This keeps the workaround active the next time the | ||
| IDE is started. If this is an issue, manually delete the temp files. | ||
| Also note when a ``.h`` file is renamed in the sketch folder, a copy of the old | ||
| file remains in the build sketch folder. This can create confusion if | ||
| you missed an edit in updating an ``#include`` in one or more of your | ||
| modules. That module will continue to use the stale version of the | ||
| ``.h`` until you restart the IDE or other major changes that would | ||
| cause the IDE to delete and recopy the contents from the source | ||
| Sketch directory. Changes on the IDE Tools board settings may cause a | ||
| complete rebuild, clearing the problem. This may be the culprit for | ||
| “What! It built fine last night!” | ||
Oops, something went wrong.
Uh oh!
There was an error while loading.Please reload this page.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.