Uh oh!
There was an error while loading.Please reload this page.
- Notifications
You must be signed in to change notification settings - Fork8.3k
esp32: auto-detect the SPI flash size and automatically size the filesystem#17337
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
Uh oh!
There was an error while loading.Please reload this page.
Conversation
An observation and suggestions. It took me a long time to understand how variants worked, and what options there are for configuration as there is little documentation, other than in comments in individual make and header files, and in PR comments such as this, which are hard to find later. In this case there is a new macro defined, that can be overridden, and that is clearly described here. I think it would be very helpful to add that to the Documentation, on a common or port level. |
@dpgeorge You might also be interested inthis commit, which I'm using for some custom boards that require on-the-fly definitions of partitions. The advantage of using the IDF API's is that it will (a) do some sanity checks such as throwing an error if new partition you're defining overlaps with any existing partition and (b) adds the partition to the in-memory list of partitions so all other IDF partitions API's recognize it as if it was defined in the partition table. |
// Query the physical size of the SPI flash and store it in the size | ||
// variable of the global, default SPI flash handle. | ||
esp_flash_get_physical_size(NULL, &esp_flash_default_chip->size); |
DvdGiessenMay 26, 2025 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Note the existence ofCONFIG_SPI_FLASH_SIZE_OVERRIDE=y
in thesdkconfig
which makers the IDF at not use the flash size value from the bootloader header but from the application header, which might be better (in my case the bootloader was wrong). Whichever value it picked will work as a fallback for when this function returns an error.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
I didn't see that option. I guess it won't hurt to enable it, although I doubt it'll make any difference for MicroPython firmware which always includes the bootloader and app.
There's alsoESPTOOLPY_HEADER_FLASHSIZE_UPDATE
which I didn't use because it disables image protection via SHA256.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
I've updated to useCONFIG_SPI_FLASH_SIZE_OVERRIDE=y
.
DvdGiessenJun 12, 2025 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Having thought about it some more I don't think we should enableCONFIG_SPI_FLASH_SIZE_OVERRIDE=y
by default.
Note it only matters when users are only flashing the application (else the bootloader and application would contain the same value anyway) and whenesp_flash_get_physical_size
fails to detect the flash size. Then we can make a choice: Do we want the size stored in the bootloader, or the size stored in the application?
In cases where the application was built specifically for a certain board/chip it makes sense to default to the value stored in the application. I've specifically needed that in some cases when building a application for a specific board with an incorrect bootloader.
However, in the general case for end-users flashing MicroPython on their boards, the value in the bootloader has a better chance of being correct. After all, the value in the application is most likely completely unrelated to their actual flash size, as our application images are per this change generic and not specific to any flash size.
And if they're not flashing the bootloader (the only case in which this matters), there's a chance that their original bootloader has a correct value because unlike the application itwas set specifically for their board. Thus, I think by default we should prefer the bootloader value, and thus not enable this option.
There's also ESPTOOLPY_HEADER_FLASHSIZE_UPDATE which I didn't use because it disables image protection via SHA256.
Hm, I wonder if that comment is up to date. Seems likeesptool.py
re-calculates the SHA256 if the header was updated, as I expect it would (since that's fairly easy to do if you're already modifying the image before flashing).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Your reasoning makes sense. And I would feel more comfortable leaving this option disabled.
Oh, I didn't know about |
@Josverl what exactly are you referring to to add to the documentation? How board variants work, or how this new auto-sizing feature works? This PR is intended to simplify things so there are less variants and less things the user needs to worry about (ie selecting the right firmware based on the flash size of the board). But I agree it should have a little documentation added. At least on the download page it should say that the firmware works with flash size 4MiB and above. |
@dpgeorge , I think that it would be useful to document how the auto-sizing feature works. So that developers can read and understand how it works icw the defied boards and variants. While I think it's good that fewer variants and boards are needed, this also increases the cognitive effort needed to make use of the existing functionality and (auto) configuration |
af01615
to8d2dfbc
CompareI've now updated this to be more transparent to the user. It now creates a partition called "vfs" only if there's no such existing partition, and there is either a "factory" or "ota_1" partition. The latter is needed to work out where the filesystem starts. This works for all boards/partitions: standard, 2MiB, OTA and ROMFS. |
c220da0
to10a8275
CompareI've now added some documentation to the esp32 README, and also updated the relevant |
ports/esp32/README.md Outdated
ESP32 firmware contains a bootloader, partition table and main application | ||
firmware, which are all stored in (external) SPI flash. The user filesystem | ||
is also store in the same SPI flash. By default, MicroPython does not have |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
typo: store --> stored
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Fixed, thanks.
is also store in the same SPI flash. By default, MicroPython does not have | ||
a fixed entry in the ESP32 partition table for the filesystem. Instead it | ||
will automatically determine the size of the SPI flash upon boot and then | ||
create a partition for the filesystem called "vfs" which takes all of the |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
I would suggest
"On boot the firmware will look for an existing virtual file system ("vfs" or "ffat") , and use that. If it cannot find a file system, then it will automatically ....
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
I've adjusted the text, but not exactly as you suggested (because in the paragraph below, it mentions in more detail about overriding the behaviour).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
This is a really neat change, simplifies so much!
I left one possible suggestion although I'm not sure if it's worth the change, LGTM either way.
Uh oh!
There was an error while loading.Please reload this page.
ports/esp32/README.md Outdated
will automatically determine the size of the SPI flash upon boot, and then -- | ||
so long as there is no existing partition called "vfs" or "ffat" -- it will | ||
create a partition for the filesystem called "vfs" which takes all of the | ||
remaining flash between the end of the application up until the end of flash. |
projectgusJun 13, 2025 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
remaining flash between the end of theapplication up until the end of flash. | |
remaining flash between the end of thelast partition defined in the partition table up until the end of flash. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Thanks, now updated.
Currently in the esp32 port the size of the SPI flash must be configured atbuild time, eg 4MiB, 8MiB, etc. Also, the esp32 partition table must beconfigured at build time, which depends on the size of the SPI flash. Abigger flash means more can be allocated to the user filesystem.This commit makes it so the SPI flash size is automatically determined atruntime, and the filesystem size is automatically set to take up as muchroom as possible (a "vfs" partition is created automatically if it doesn'texist).This works by:- Setting the SPI flash size to be 4MiB in the build (or some other value, as long as the firmware app fits).- Removing the vfs partition from the esp32 partition table (only nvs, phy_init and firmware, and maybe romfs, remain in the partition table).- At boot, query the physical size of the SPI flash and use that as the actual size in the code.- If it doesn't already exist, automatically create a "vfs" partition which takes up the flash from the end of all existing partitions to the end of flash.This allows simplifying a lot of board configurations, and removing someboard variants that just change the flash size (to be done in a followingcommit).It's also fully backwards compatible, in the following sense:- Existing boards with MicroPython firmware will continue to work with the same filesystem, ie the filesystem won't be erased when the firmware is updated.- If a user has a custom esp32 partition table and installs MicroPython as a bare app into the app partition, the new MicroPython firmware will honour the esp32 partition table and use either "vfs" or "ffat" partitions as the filesystem.Signed-off-by: Damien George <damien@micropython.org>
Remove the "vfs" entry from all partitions-*.csv files, and then removeduplicated files.And remove the ESP32_GENERIC_S3-FLASH_4M variant, because it's no longerneeded.Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
16b20a1
tofa393fe
CompareTested again on a board using |
fa393fe
intomicropython:masterUh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
Summary
Currently in the esp32 port the size of the SPI flash must be configured at build time, eg 4MiB, 8MiB, etc. Also, the esp32 partition table must be configured at build time, which depends on the size of the SPI flash. A bigger flash means more can be allocated to the user filesystem.
This PR makes it so the SPI flash size is automatically determined at runtime, and the filesystem size is automatically set to take up as much room as possible.
This works by:
vfs
partition from the esp32 partition table (only nvs, phy_init and firmware remains in the partition table).vfs
partition which takes up the flash from the end of the firmware partition to the end of flash.This allows simplifying a lot of board configurations, and removing some board variants that just change the flash size.
It's also fully backwards compatible, in the following sense:
vfs
orffat
partitions as the filesystem.Eventually this mechanism will be extended:
Testing
Tested with ESP32_GENERIC on a board with 4MiB of flash, and ESP32_GENERIC_S2 on a board with 16MiB flash. The filesystem size was correctly detected.
Trade-offs and Alternatives
Boards configured for OTA have not been changed, they still define the filesystem partition explicitly.PR updated to now cover all boards.MICROPY_BOARD_STARTUP
macro.