Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

esp32: auto-detect the SPI flash size and automatically size the filesystem#17337

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Merged

Conversation

dpgeorge
Copy link
Member

@dpgeorgedpgeorge commentedMay 22, 2025
edited
Loading

Summary

Currently in the esp32 port the size of the SPI flash must be configured at build time, eg 4MiB, 8MiB, etc. Also, the esp32 partition table must be configured at build time, which depends on the size of the SPI flash. A bigger flash means more can be allocated to the user filesystem.

This PR makes it so the SPI flash size is automatically determined at runtime, and the filesystem size is automatically set to take up as much room as possible.

This works by:

  1. Setting the SPI flash size to be 4MiB in the build (or some other value, as long as the firmware app fits).
  2. Removing thevfs partition from the esp32 partition table (only nvs, phy_init and firmware remains in the partition table).
  3. At boot, query the physical size of the SPI flash and use that as the actual size in the code.
  4. If it doesn't already exist, automatically create avfs partition which takes up the flash from the end of the firmware partition to the end of flash.

This allows simplifying a lot of board configurations, and removing some board variants that just change the flash size.

It's also fully backwards compatible, in the following sense:

  1. Existing boards with MicroPython firmware will continue to work with the same filesystem, ie the filesystem won't be erased when the firmware is updated.
  2. If a user has a custom esp32 partition table and installs MicroPython as a bare app into the app partition, the new MicroPython firmware will honor the esp32 partition table and use eithervfs orffat partitions as the filesystem.

Eventually this mechanism will be extended:

  • to other ports, probably rp2 next
  • to allow users to customise the filesystem layout, and enable ROMFS partition(s) without rebuilding firmware

Testing

Tested with ESP32_GENERIC on a board with 4MiB of flash, and ESP32_GENERIC_S2 on a board with 16MiB flash. The filesystem size was correctly detected.

Trade-offs and Alternatives

  • Boards configured for OTA have not been changed, they still define the filesystem partition explicitly. PR updated to now cover all boards.
  • It may be that the physical flash size cannot be detected correctly, but I think the IDF has covered most cases by now.
  • Custom board definitions can override the auto detect feature by defining their ownMICROPY_BOARD_STARTUP macro.

robert-hh and Tangerino reacted with thumbs up emojiprojectgus reacted with rocket emoji
@dpgeorgedpgeorge requested review fromprojectgus and removed request forprojectgusMay 22, 2025 01:25
@Josverl
Copy link
Contributor

An observation and suggestions.

It took me a long time to understand how variants worked, and what options there are for configuration as there is little documentation, other than in comments in individual make and header files, and in PR comments such as this, which are hard to find later.
(Or perhaps I just failed to find the docs)

In this case there is a new macro defined, that can be overridden, and that is clearly described here.

I think it would be very helpful to add that to the Documentation, on a common or port level.
That would help tick the 'How can we teach this' checkbox.

@DvdGiessen
Copy link
Contributor

@dpgeorge You might also be interested inthis commit, which I'm using for some custom boards that require on-the-fly definitions of partitions.

The advantage of using the IDF API's is that it will (a) do some sanity checks such as throwing an error if new partition you're defining overlaps with any existing partition and (b) adds the partition to the in-memory list of partitions so all other IDF partitions API's recognize it as if it was defined in the partition table.

Comment on lines +219 to +221
// Query the physical size of the SPI flash and store it in the size
// variable of the global, default SPI flash handle.
esp_flash_get_physical_size(NULL, &esp_flash_default_chip->size);
Copy link
Contributor

@DvdGiessenDvdGiessenMay 26, 2025
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Note the existence ofCONFIG_SPI_FLASH_SIZE_OVERRIDE=y in thesdkconfig which makers the IDF at not use the flash size value from the bootloader header but from the application header, which might be better (in my case the bootloader was wrong). Whichever value it picked will work as a fallback for when this function returns an error.

Copy link
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

I didn't see that option. I guess it won't hurt to enable it, although I doubt it'll make any difference for MicroPython firmware which always includes the bootloader and app.

There's alsoESPTOOLPY_HEADER_FLASHSIZE_UPDATE which I didn't use because it disables image protection via SHA256.

Copy link
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

I've updated to useCONFIG_SPI_FLASH_SIZE_OVERRIDE=y.

Copy link
Contributor

@DvdGiessenDvdGiessenJun 12, 2025
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Having thought about it some more I don't think we should enableCONFIG_SPI_FLASH_SIZE_OVERRIDE=y by default.

Note it only matters when users are only flashing the application (else the bootloader and application would contain the same value anyway) and whenesp_flash_get_physical_size fails to detect the flash size. Then we can make a choice: Do we want the size stored in the bootloader, or the size stored in the application?

In cases where the application was built specifically for a certain board/chip it makes sense to default to the value stored in the application. I've specifically needed that in some cases when building a application for a specific board with an incorrect bootloader.

However, in the general case for end-users flashing MicroPython on their boards, the value in the bootloader has a better chance of being correct. After all, the value in the application is most likely completely unrelated to their actual flash size, as our application images are per this change generic and not specific to any flash size.

And if they're not flashing the bootloader (the only case in which this matters), there's a chance that their original bootloader has a correct value because unlike the application itwas set specifically for their board. Thus, I think by default we should prefer the bootloader value, and thus not enable this option.

There's also ESPTOOLPY_HEADER_FLASHSIZE_UPDATE which I didn't use because it disables image protection via SHA256.

Hm, I wonder if that comment is up to date. Seems likeesptool.pyre-calculates the SHA256 if the header was updated, as I expect it would (since that's fairly easy to do if you're already modifying the image before flashing).

Copy link
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Your reasoning makes sense. And I would feel more comfortable leaving this option disabled.

@dpgeorge
Copy link
MemberAuthor

You might also be interested inthis commit, which I'm using for some custom boards that require on-the-fly definitions of partitions.

Oh, I didn't know aboutesp_partition_register_external()! That looks very useful, thanks. I'll see if I can switch to using that.

@dpgeorge
Copy link
MemberAuthor

I think it would be very helpful to add that to the Documentation, on a common or port level.

@Josverl what exactly are you referring to to add to the documentation? How board variants work, or how this new auto-sizing feature works?

This PR is intended to simplify things so there are less variants and less things the user needs to worry about (ie selecting the right firmware based on the flash size of the board). But I agree it should have a little documentation added. At least on the download page it should say that the firmware works with flash size 4MiB and above.

@Josverl
Copy link
Contributor

@dpgeorge , I think that it would be useful to document how the auto-sizing feature works. So that developers can read and understand how it works icw the defied boards and variants.

While I think it's good that fewer variants and boards are needed, this also increases the cognitive effort needed to make use of the existing functionality and (auto) configuration

projectgus reacted with thumbs up emoji

@dpgeorgedpgeorgeforce-pushed theesp32-detect-flash-size branch 2 times, most recently fromaf01615 to8d2dfbcCompareJune 10, 2025 07:31
@dpgeorge
Copy link
MemberAuthor

I've now updated this to be more transparent to the user. It now creates a partition called "vfs" only if there's no such existing partition, and there is either a "factory" or "ota_1" partition. The latter is needed to work out where the filesystem starts.

This works for all boards/partitions: standard, 2MiB, OTA and ROMFS.

@dpgeorgedpgeorgeforce-pushed theesp32-detect-flash-size branch 2 times, most recently fromc220da0 to10a8275CompareJune 12, 2025 00:57
@dpgeorge
Copy link
MemberAuthor

I've now added some documentation to the esp32 README, and also updated the relevantboard.md files to account for the changes.

Josverl reacted with thumbs up emoji


ESP32 firmware contains a bootloader, partition table and main application
firmware, which are all stored in (external) SPI flash. The user filesystem
is also store in the same SPI flash. By default, MicroPython does not have
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

typo: store --> stored

Copy link
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Fixed, thanks.

is also store in the same SPI flash. By default, MicroPython does not have
a fixed entry in the ESP32 partition table for the filesystem. Instead it
will automatically determine the size of the SPI flash upon boot and then
create a partition for the filesystem called "vfs" which takes all of the
Copy link
Contributor

@JosverlJosverlJun 12, 2025
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

I would suggest
"On boot the firmware will look for an existing virtual file system ("vfs" or "ffat") , and use that. If it cannot find a file system, then it will automatically ....

Copy link
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

I've adjusted the text, but not exactly as you suggested (because in the paragraph below, it mentions in more detail about overriding the behaviour).

Josverl reacted with thumbs up emoji
Copy link
Contributor

@projectgusprojectgus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

This is a really neat change, simplifies so much!

I left one possible suggestion although I'm not sure if it's worth the change, LGTM either way.

will automatically determine the size of the SPI flash upon boot, and then --
so long as there is no existing partition called "vfs" or "ffat" -- it will
create a partition for the filesystem called "vfs" which takes all of the
remaining flash between the end of the application up until the end of flash.
Copy link
Contributor

@projectgusprojectgusJun 13, 2025
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Suggested change
remaining flash between the end of theapplication up until the end of flash.
remaining flash between the end of thelast partition defined in the partition table up until the end of flash.

Copy link
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Thanks, now updated.

Currently in the esp32 port the size of the SPI flash must be configured atbuild time, eg 4MiB, 8MiB, etc.  Also, the esp32 partition table must beconfigured at build time, which depends on the size of the SPI flash.  Abigger flash means more can be allocated to the user filesystem.This commit makes it so the SPI flash size is automatically determined atruntime, and the filesystem size is automatically set to take up as muchroom as possible (a "vfs" partition is created automatically if it doesn'texist).This works by:- Setting the SPI flash size to be 4MiB in the build (or some other value,  as long as the firmware app fits).- Removing the vfs partition from the esp32 partition table (only nvs,  phy_init and firmware, and maybe romfs, remain in the partition table).- At boot, query the physical size of the SPI flash and use that as the  actual size in the code.- If it doesn't already exist, automatically create a "vfs" partition which  takes up the flash from the end of all existing partitions to the end of  flash.This allows simplifying a lot of board configurations, and removing someboard variants that just change the flash size (to be done in a followingcommit).It's also fully backwards compatible, in the following sense:- Existing boards with MicroPython firmware will continue to work with the  same filesystem, ie the filesystem won't be erased when the firmware is  updated.- If a user has a custom esp32 partition table and installs MicroPython as  a bare app into the app partition, the new MicroPython firmware will  honour the esp32 partition table and use either "vfs" or "ffat"  partitions as the filesystem.Signed-off-by: Damien George <damien@micropython.org>
Remove the "vfs" entry from all partitions-*.csv files, and then removeduplicated files.And remove the ESP32_GENERIC_S3-FLASH_4M variant, because it's no longerneeded.Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
@dpgeorgedpgeorgeforce-pushed theesp32-detect-flash-size branch from16b20a1 tofa393feCompareJune 13, 2025 15:07
@dpgeorge
Copy link
MemberAuthor

Tested again on a board usingpartitions-4MiB-romfs.csv, it works correctly, the ROMFS is intact and the "vfs" partition starts after ROMFS.

@dpgeorgedpgeorge merged commitfa393fe intomicropython:masterJun 13, 2025
8 checks passed
@dpgeorgedpgeorge deleted the esp32-detect-flash-size branchJune 13, 2025 15:29
Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment
Reviewers

@projectgusprojectgusprojectgus approved these changes

@DvdGiessenDvdGiessenDvdGiessen left review comments

@JosverlJosverlJosverl approved these changes

Assignees
No one assigned
Labels
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

4 participants
@dpgeorge@Josverl@DvdGiessen@projectgus

[8]ページ先頭

©2009-2025 Movatter.jp