- Notifications
You must be signed in to change notification settings - Fork1.5k
Validate any SQL migration scripts are the most recent#6652
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
base:main
Are you sure you want to change the base?
Changes fromall commits
1ce311f2f1fe5098c28941ab6ad372aef86File filter
Filter by extension
Conversations
Uh oh!
There was an error while loading.Please reload this page.
Jump to
Uh oh!
There was an error while loading.Please reload this page.
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| @@ -262,3 +262,178 @@ jobs: | ||||||||||||||||
| working-directory: "dev" | ||||||||||||||||
| run: docker compose down | ||||||||||||||||
| shell: pwsh | ||||||||||||||||
| validate-migration-naming: | ||||||||||||||||
| name: Validate new migration naming and order | ||||||||||||||||
| runs-on: ubuntu-22.04 | ||||||||||||||||
| steps: | ||||||||||||||||
| - name: Check out repo | ||||||||||||||||
| uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2 | ||||||||||||||||
| with: | ||||||||||||||||
| fetch-depth: 0 | ||||||||||||||||
| persist-credentials: false | ||||||||||||||||
| - name: Validate new migrations for pull request | ||||||||||||||||
| if: github.event_name == 'pull_request' | ||||||||||||||||
| run: | | ||||||||||||||||
| # Use C locale for consistent, locale-independent string comparison | ||||||||||||||||
| export LC_ALL=C | ||||||||||||||||
| # Get list of migrations from main branch | ||||||||||||||||
| git fetch origin main:main | ||||||||||||||||
Contributor There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others.Learn more. ❌Finding 1 continued: This fetch command should have error handling. If it fails, the script will continue with a stale or non-existent local Suggestion: if! git fetch origin main:main;thenecho"ERROR: Failed to fetch main branch"exit 1fi Contributor There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others.Learn more.
Suggested change
| ||||||||||||||||
| MAIN_MIGRATIONS=$(git ls-tree -r --name-only main -- util/Migrator/DbScripts/*.sql | sort) | ||||||||||||||||
| # Get list of migrations from current branch | ||||||||||||||||
| CURRENT_MIGRATIONS=$(git ls-tree -r --name-only HEAD -- util/Migrator/DbScripts/*.sql | sort) | ||||||||||||||||
| # Save to temporary files for comparison | ||||||||||||||||
| echo "$MAIN_MIGRATIONS" > /tmp/main_migrations.txt | ||||||||||||||||
| echo "$CURRENT_MIGRATIONS" > /tmp/current_migrations.txt | ||||||||||||||||
Contributor There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others.Learn more. ♻️Finding 4 - Cleanup: Consider adding cleanup for temporary files at the end of the script: rm -f /tmp/main_migrations.txt /tmp/current_migrations.txt | ||||||||||||||||
| # Find added migrations | ||||||||||||||||
| ADDED_MIGRATIONS=$(comm -13 /tmp/main_migrations.txt /tmp/current_migrations.txt) | ||||||||||||||||
| if [ -z "$ADDED_MIGRATIONS" ]; then | ||||||||||||||||
| echo "No new migration files added." | ||||||||||||||||
| exit 0 | ||||||||||||||||
| fi | ||||||||||||||||
| echo "New migration files detected:" | ||||||||||||||||
| echo "$ADDED_MIGRATIONS" | ||||||||||||||||
| echo "" | ||||||||||||||||
| # Check that all added migrations come after all existing migrations | ||||||||||||||||
| LAST_MAIN_MIGRATION=$(echo "$MAIN_MIGRATIONS" | tail -n 1 | xargs basename) | ||||||||||||||||
| echo "Last migration in main branch: $LAST_MAIN_MIGRATION" | ||||||||||||||||
| echo "" | ||||||||||||||||
| VALIDATION_FAILED=0 | ||||||||||||||||
| while IFS= read -r migration; do | ||||||||||||||||
| MIGRATION_NAME=$(basename "$migration") | ||||||||||||||||
| # Validate NEW migration filename format - enforce strict format for new migrations | ||||||||||||||||
| # Required format: YYYY-MM-DD_NN_Description.sql (with leading zeros in date and sequence) | ||||||||||||||||
| if ! [[ "$MIGRATION_NAME" =~ ^[0-9]{4}-[0-9]{2}-[0-9]{2}_[0-9]{2}_.+\.sql$ ]]; then | ||||||||||||||||
Contributor There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others.Learn more. ❌Finding 1 - Format enforcement issue: This regex requires leading zeros (
This will cause validation failures on PRs that rebase against main if the problematic migration gets included. Options:
Contributor There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others.Learn more. ❌Critical: Regex allows non-compliant migration that breaks sorting The repository contains Problem: Lexicographic string comparison produces incorrect ordering:
Impact:
Fix required before merging: Rename the non-compliant migration: git mv util/Migrator/DbScripts/2025-10-3_00_AddOrganizationIntegration_ReadByTeamsConfigurationTenantIdTeamIdStoredProcedure.sql \ util/Migrator/DbScripts/2025-10-03_00_AddOrganizationIntegration_ReadByTeamsConfigurationTenantIdTeamIdStoredProcedure.sql Then restore strict regex validation: Suggested change
| ||||||||||||||||
| echo "ERROR: Migration '$MIGRATION_NAME' does not match required format" | ||||||||||||||||
| echo "Required format: YYYY-MM-DD_NN_Description.sql" | ||||||||||||||||
Comment on lines +318 to +319 Member There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others.Learn more. 👏 Also a nice improvement here. | ||||||||||||||||
| echo " - YYYY: 4-digit year" | ||||||||||||||||
| echo " - MM: 2-digit month with leading zero (01-12)" | ||||||||||||||||
| echo " - DD: 2-digit day with leading zero (01-31)" | ||||||||||||||||
| echo " - NN: 2-digit sequence number (00, 01, 02, etc.)" | ||||||||||||||||
| echo "Example: 2025-01-15_00_MyMigration.sql" | ||||||||||||||||
| VALIDATION_FAILED=1 | ||||||||||||||||
| continue | ||||||||||||||||
| fi | ||||||||||||||||
| # Compare migration name with last main migration | ||||||||||||||||
| if [[ "$MIGRATION_NAME" < "$LAST_MAIN_MIGRATION" ]]; then | ||||||||||||||||
Contributor There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others.Learn more. 💭Finding 5 - Multiple new migrations: When a PR adds multiple migrations (e.g., The out-of-order new migrations would be caught (B would fail since it's < A alphabetically), but the error message might be confusing since it references the main branch rather than the other new migration. Is this the intended behavior, or should we also validate ordering within the set of new migrations? | ||||||||||||||||
| echo "ERROR: New migration '$MIGRATION_NAME' is not chronologically after '$LAST_MAIN_MIGRATION'" | ||||||||||||||||
| VALIDATION_FAILED=1 | ||||||||||||||||
| else | ||||||||||||||||
| echo "OK: '$MIGRATION_NAME' is chronologically after '$LAST_MAIN_MIGRATION'" | ||||||||||||||||
| fi | ||||||||||||||||
| done <<< "$ADDED_MIGRATIONS" | ||||||||||||||||
| echo "" | ||||||||||||||||
| if [ $VALIDATION_FAILED -eq 1 ]; then | ||||||||||||||||
| echo "FAILED: One or more migrations are incorrectly named or not in chronological order" | ||||||||||||||||
| echo "" | ||||||||||||||||
| echo "All new migration files must:" | ||||||||||||||||
| echo " 1. Follow the naming format: YYYY-MM-DD_NN_Description.sql" | ||||||||||||||||
| echo " 2. Use leading zeros in dates (e.g., 2025-01-05, not 2025-1-5)" | ||||||||||||||||
| echo " 3. Include a 2-digit sequence number (e.g., _00, _01)" | ||||||||||||||||
| echo " 4. Have a filename that sorts after the last migration in main" | ||||||||||||||||
| echo "" | ||||||||||||||||
| echo "To fix this issue:" | ||||||||||||||||
Contributor There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others.Learn more. 🎨Finding 6 - Enhanced error message: Consider adding the specific git command to help developers fix the issue: echo"To fix this issue:"echo" 1. Rename using git:"echo" git mv util/Migrator/DbScripts/$MIGRATION_NAME util/Migrator/DbScripts/YYYY-MM-DD_NN_NewName.sql"echo" 2. Choose a date after$LAST_MAIN_MIGRATION"echo" 3. Commit the change: git commit -m 'Rename migration to fix ordering'" Contributor There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others.Learn more. 🎨Enhancement: Add specific git commands to help developers fix issues: Suggested change
| ||||||||||||||||
| echo " 1. Locate your migration file(s) in util/Migrator/DbScripts/" | ||||||||||||||||
| echo " 2. Rename to follow format: YYYY-MM-DD_NN_Description.sql" | ||||||||||||||||
| echo " 3. Ensure the date is after $LAST_MAIN_MIGRATION" | ||||||||||||||||
| echo "" | ||||||||||||||||
| echo "Example: 2025-01-15_00_AddNewFeature.sql" | ||||||||||||||||
| exit 1 | ||||||||||||||||
| fi | ||||||||||||||||
| echo "SUCCESS: All new migrations are correctly named and in chronological order" | ||||||||||||||||
| shell: bash | ||||||||||||||||
| - name: Validate new migrations for push | ||||||||||||||||
Member There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others.Learn more. I assume these 2 jobs are identical except for the source of the current migrations? I suggest that the comparison logic could be moved into a script for reuse, e.g. | ||||||||||||||||
| if: github.event_name == 'push' || github.event_name == 'workflow_dispatch' | ||||||||||||||||
| run: | | ||||||||||||||||
| # Use C locale for consistent, locale-independent string comparison | ||||||||||||||||
| export LC_ALL=C | ||||||||||||||||
| # Get list of migrations from previous commit | ||||||||||||||||
| PREV_MIGRATIONS=$(git ls-tree -r --name-only HEAD~1 -- util/Migrator/DbScripts/*.sql 2>/dev/null | sort || echo "") | ||||||||||||||||
| # Get list of migrations from current commit | ||||||||||||||||
| CURRENT_MIGRATIONS=$(git ls-tree -r --name-only HEAD -- util/Migrator/DbScripts/*.sql | sort) | ||||||||||||||||
| # Save to temporary files for comparison | ||||||||||||||||
| echo "$PREV_MIGRATIONS" > /tmp/prev_migrations.txt | ||||||||||||||||
| echo "$CURRENT_MIGRATIONS" > /tmp/current_migrations.txt | ||||||||||||||||
| # Find added migrations | ||||||||||||||||
| ADDED_MIGRATIONS=$(comm -13 /tmp/prev_migrations.txt /tmp/current_migrations.txt) | ||||||||||||||||
| if [ -z "$ADDED_MIGRATIONS" ]; then | ||||||||||||||||
| echo "No new migration files added in this commit." | ||||||||||||||||
| exit 0 | ||||||||||||||||
| fi | ||||||||||||||||
| echo "New migration files detected:" | ||||||||||||||||
| echo "$ADDED_MIGRATIONS" | ||||||||||||||||
| echo "" | ||||||||||||||||
| # Check that all added migrations come after all existing migrations | ||||||||||||||||
| if [ -n "$PREV_MIGRATIONS" ]; then | ||||||||||||||||
| LAST_PREV_MIGRATION=$(echo "$PREV_MIGRATIONS" | tail -n 1 | xargs basename) | ||||||||||||||||
| echo "Last migration in previous commit: $LAST_PREV_MIGRATION" | ||||||||||||||||
| echo "" | ||||||||||||||||
| VALIDATION_FAILED=0 | ||||||||||||||||
| while IFS= read -r migration; do | ||||||||||||||||
| MIGRATION_NAME=$(basename "$migration") | ||||||||||||||||
| # Validate NEW migration filename format - enforce strict format for new migrations | ||||||||||||||||
| # Required format: YYYY-MM-DD_NN_Description.sql (with leading zeros in date and sequence) | ||||||||||||||||
| if ! [[ "$MIGRATION_NAME" =~ ^[0-9]{4}-[0-9]{2}-[0-9]{2}_[0-9]{2}_.+\.sql$ ]]; then | ||||||||||||||||
| echo "ERROR: Migration '$MIGRATION_NAME' does not match required format" | ||||||||||||||||
| echo "Required format: YYYY-MM-DD_NN_Description.sql" | ||||||||||||||||
| echo " - YYYY: 4-digit year" | ||||||||||||||||
| echo " - MM: 2-digit month with leading zero (01-12)" | ||||||||||||||||
| echo " - DD: 2-digit day with leading zero (01-31)" | ||||||||||||||||
| echo " - NN: 2-digit sequence number (00, 01, 02, etc.)" | ||||||||||||||||
| echo "Example: 2025-01-15_00_MyMigration.sql" | ||||||||||||||||
| VALIDATION_FAILED=1 | ||||||||||||||||
| continue | ||||||||||||||||
| fi | ||||||||||||||||
| # Compare migration name with last previous migration | ||||||||||||||||
| if [[ "$MIGRATION_NAME" < "$LAST_PREV_MIGRATION" ]]; then | ||||||||||||||||
| echo "ERROR: New migration '$MIGRATION_NAME' is not chronologically after '$LAST_PREV_MIGRATION'" | ||||||||||||||||
| VALIDATION_FAILED=1 | ||||||||||||||||
| else | ||||||||||||||||
| echo "OK: '$MIGRATION_NAME' is chronologically after '$LAST_PREV_MIGRATION'" | ||||||||||||||||
| fi | ||||||||||||||||
| done <<< "$ADDED_MIGRATIONS" | ||||||||||||||||
| echo "" | ||||||||||||||||
| if [ $VALIDATION_FAILED -eq 1 ]; then | ||||||||||||||||
| echo "FAILED: One or more migrations are incorrectly named or not in chronological order" | ||||||||||||||||
| echo "" | ||||||||||||||||
| echo "All new migration files must:" | ||||||||||||||||
| echo " 1. Follow the naming format: YYYY-MM-DD_NN_Description.sql" | ||||||||||||||||
| echo " 2. Use leading zeros in dates (e.g., 2025-01-05, not 2025-1-5)" | ||||||||||||||||
| echo " 3. Include a 2-digit sequence number (e.g., _00, _01)" | ||||||||||||||||
| echo " 4. Have a filename that sorts after the last migration" | ||||||||||||||||
| echo "" | ||||||||||||||||
| echo "Migration files are located in: util/Migrator/DbScripts/" | ||||||||||||||||
| exit 1 | ||||||||||||||||
| fi | ||||||||||||||||
| else | ||||||||||||||||
| echo "No previous migrations found (initial commit?). Skipping validation." | ||||||||||||||||
| fi | ||||||||||||||||
| echo "SUCCESS: All new migrations are correctly named and in chronological order" | ||||||||||||||||
| shell: bash | ||||||||||||||||
Uh oh!
There was an error while loading.Please reload this page.