Uh oh!
There was an error while loading.Please reload this page.
- Notifications
You must be signed in to change notification settings - Fork12
Fix collision between macOS workflow artifacts in release workflows#803
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
Merged
Uh oh!
There was an error while loading.Please reload this page.
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
The established convention is to pad the GitHub Actions context identifier in references.In this workflow code, the right hand padding was missing.
GitHub Workflows are used to automatically generate and publish production and nightly releases of the project. This isdone for a range of host architectures, including macOS. The macOS builds are then put through a notarization process ina dedicated workflow job.GitHub Actions workflow artifacts are used to transfer the generated files between sequential jobs in the workflow. The"actions/upload-artifact" and "actions/download-artifact" actions are used for this purpose.The workflow artifact handling had to be reworked recently in order to handle a breaking change in the 4.0.0 release ofthe "actions/upload-artifact" action. Previously, a single artifact was used for the transfer of the builds for allhosts. However, support for uploading multiple times to a single artifact was dropped in version 4.0.0 of the"actions/upload-artifact" action. So it is now necessary to use a dedicated artifact for each of the builds. These aredownloaded in aggregate in a subsequent job by using the artifact name globbing and merging features which wereintroduced in version 4.1.0 of the "actions/download-artifact" action.A regression was introduced at that time. The chosen approach was to use a separate set of artifacts for thenon-notarized and notarized files. An overview of the sequence (the prefixes are the workflow job names):1. create-release-artifacts/create-nightly-artifacts: Generate builds.2. create-release-artifacts/create-nightly-artifacts: Upload builds to workflow artifacts3. notarize-macos: Download workflow artifacts.4. notarize-macos: Notarize macOS build from downloaded artifact.5. notarize-macos: Upload notarized build to workflow artifact with a different name than the source artifact.6. create-release/publish-nightly: Download workflow artifacts.7. create-release/publish-nightly: Publish builds.The problem with this is that the artifacts for the non-notarized (uploaded by thecreate-release-artifacts/create-nightly-artifacts job) and notarized (created by the notarize-macos job) files are thendownloaded and merged by the create-release/publish-nightly job. Since each artifact contains a file with the same pathin the merged output, the contents of the last downloaded artifact overwrite the contents of the first. It happens thatthe non-notarized artifact is downloaded after the notarized artifact, so this file path collision results innon-notarized macOS builds being published instead of the notarized builds as intended, and as done by the workflowprior to the regression. For example, this bug was also present in the Arduino CLI repository and you can see the impacton its releases:```% wgethttps://downloads.arduino.cc/arduino-cli/nightly/arduino-cli_nightly-latest_macOS_ARM64.tar.gz[...]% tar -xf arduino-cli_nightly-latest_macOS_ARM64.tar.gz% spctl -a -vvv -t install arduino-cliarduino-cli: rejected``````% wgethttps://downloads.arduino.cc/arduino-cli/arduino-cli_latest_macOS_ARM64.tar.gz[..]% tar -xf arduino-cli_latest_macOS_ARM64.tar.gz% spctl -a -vvv -t install arduino-cliarduino-cli: rejected```The chosen solution is that, instead of using new artifacts for the notarized builds, to overwrite the same macOSartifacts containing the no-notarized builds, which were generated by thecreate-release-artifacts/create-nightly-artifacts job with the notarized builds from the notarize-macos jobs. This ismade possible by the overwriting capability of the "actions/upload-artifact" action (which was introduced in version4.2.0).An overview of the new sequence (the prefixes are the workflow job names):1. create-release-artifacts/create-nightly-artifacts: Generate builds.2. create-release-artifacts/create-nightly-artifacts: Upload builds to workflow artifacts3. notarize-macos: Download macOS x86 or Apple Silicon workflow artifact.5. notarize-macos: Notarize macOS build from downloaded artifact.6. notarize-macos: Upload notarized build to same workflow artifact, overwriting the non-notarized contents.7. create-release/publish-nightly: Download workflow artifacts.8. create-release/publish-nightly: Publish builds.
2536ae5 toe2ee58fCompare6529e20 intoarduino:main 7 checks passed
Uh oh!
There was an error while loading.Please reload this page.
Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment
Labels
os: macosSpecific to macOS operating system topic: infrastructureRelated to project infrastructure type: imperfectionPerceived defect in any part of project
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
GitHub Workflows are used to automatically generate and publish production and nightly releases of the project. This is done for a range of host architectures, including macOS. The macOS builds are then put through a notarization process in a dedicated workflow job.
GitHub Actions workflow artifacts are used to transfer the generated files between sequential jobs in the workflow. Theactions/upload-artifact andactions/download-artifact actions are used for this purpose.
The workflow artifact handling had to be reworked recently in order to handle a breaking change in the 4.0.0 release of theactions/upload-artifact action (#662 /3289cb8). Previously, a single artifact was used for the transfer of the builds for all hosts. However, support for uploading multiple times to a single artifact was dropped in version 4.0.0 of theactions/upload-artifact action. So it is now necessary to use a dedicated artifact for each of the builds. These are downloaded in aggregate in a subsequent job by using the artifact name globbing and merging features which were introduced in version 4.1.0 of theactions/download-artifact action.
A regression was introduced at that time. The chosen approach was to use a separate set of artifacts for the non-notarized and notarized files. An overview of the sequence (the prefixes are the workflow job names):
create-release-artifacts/create-nightly-artifacts: Generate builds.create-release-artifacts/create-nightly-artifacts: Upload builds to workflow artifactsnotarize-macos: Download workflow artifacts.notarize-macos: Notarize macOS build from downloaded artifact.notarize-macos: Upload notarized build to workflow artifact with a different name than the source artifact.create-release/publish-nightly: Download workflow artifacts.create-release/publish-nightly: Publish builds.The problem with this is that the artifacts for the non-notarized (uploaded by the
create-release-artifacts/create-nightly-artifactsjob) and notarized (created by thenotarize-macosjob) files are then downloaded and merged by thecreate-release/publish-nightlyjob. Since each artifact contains a file with the same path in the merged output, the contents of the last downloaded artifact overwrite the contents of the first. It happens that the non-notarized artifact is downloaded after the notarized artifact, so this file path collision results in non-notarized macOS builds being published instead of the notarized builds as intended, and as done by the workflow prior to the regression. For example, this bug was also present in the Arduino CLI repository and you can see the impact on its releases:The chosen solution is that, instead of using new artifacts for the notarized builds, to overwrite the same macOS artifacts containing the no-notarized builds, which were generated by the
create-release-artifacts/create-nightly-artifactsjob with the notarized builds from thenotarize-macosjobs. This is made possible by the overwriting capability of theactions/upload-artifact action (which was introduced in version 4.2.0:actions/upload-artifact@11ff42c).An overview of the new sequence (the prefixes are the workflow job names):
create-release-artifacts/create-nightly-artifacts: Generate builds.create-release-artifacts/create-nightly-artifacts: Upload builds to workflow artifactsnotarize-macos: Download macOS x86 or Apple Silicon workflow artifact.notarize-macos: Notarize macOS build from downloaded artifact.notarize-macos: Upload notarized build to same workflow artifact, overwriting the non-notarized contents.create-release/publish-nightly: Download workflow artifacts.create-release/publish-nightly: Publish builds.The result is that there is no file path collision when the
create-release/publish-nightlyjob downloads and merges the artifacts.