Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

DOT NOT MERGE -- RDS/Aurora logical refresh component#262

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Open
NikolayS wants to merge11 commits intomaster
base:master
Choose a base branch
Loading
fromclaude/rds-aurora-component-012oeo6D7EyNwxoupNfefuCS

Conversation

@NikolayS
Copy link
Contributor

@NikolaySNikolayS commentedDec 10, 2025
edited by cursorbot
Loading

Note

Introduces a standalonerds-refresh tool that clones RDS/Aurora from snapshots, updates DBLab source, triggers full refresh, and cleans up—plus Docker/CLI, configs, and docs.

  • New component:rds-refresh
    • CLI tool to orchestrate DBLab full refresh via temporary RDS/Aurora clones from snapshots (main.go,refresher.go).
    • DBLab API client to update source config and trigger/monitor refresh (dblab.go).
    • AWS RDS client for snapshot discovery, clone create/wait/delete for RDS and Aurora (rds.go).
    • Config loader with env expansion, validation, and defaults; example configs (config.go,config.example.yaml).
    • Packaging and tooling:Dockerfile,Makefile, module files (go.mod,go.sum).
    • Documentation with usage, scheduling, IAM, and networking (README.md).
  • Engine
    • Add example config:engine/configs/rds-refresh.example.yaml.
    • Updateengine/go.mod/go.sum with AWS SDK (v2) indirect dependencies.

Written byCursor Bugbot for commit0104395. Configurehere.

Add a new component that automates DBLab full refresh using temporaryRDS/Aurora database clones created from snapshots. This enables ahassle-free data sync workflow that doesn't impact production.Features:- Create temporary RDS/Aurora clones from latest automated snapshots- Wait for clone availability with proper timeout handling- Trigger DBLab full refresh via API- Poll refresh status until completion- Clean up temporary clones automaticallyDeployment options:- AWS Lambda with SAM template and EventBridge scheduling- Standalone CLI binary for cron/manual execution- Docker container for containerized environmentsIncludes comprehensive documentation with IAM policy examplesand example configuration files.
Add a fully standalone version of the RDS/Aurora refresh tool that canbe built and deployed immediately without requiring a DBLab Enginerelease. This version has no dependencies on DBLab internal packages.The standalone component includes:- Self-contained Go module with minimal dependencies- Makefile for easy building (CLI, Lambda, Docker)- SAM template for AWS Lambda deployment- Dockerfile for container builds- Comprehensive documentationCan be used immediately by:1. go build -o rds-refresh .2. ./rds-refresh -config config.yaml
- Add UpdateSourceConfig method to update DBLab source connection  before triggering refresh, ensuring DBLab knows the clone endpoint- Add source database credentials (dbName, username, password) to  config for DBLab connection configuration- Remove Lambda-specific code, focusing on container/CLI deployment  which better suits the long-running nature of refresh operations- Remove duplicate engine-integrated code (engine/internal/rdsrefresh  and engine/cmd/rds-refresh) since standalone component is preferred- Update documentation with container deployment options (Docker,  ECS Task, Kubernetes CronJob, CLI with cron)- Update workflow to include config update step before refresh trigger
DBLab expects a flat JSON structure matching ConfigProjection fields,not a nested structure. The API automatically reloads the config afterupdate (calls reloadFn internally), so no SIGHUP is needed.Changed from nested:  {"retrieval": {"dbSource": {"host": "..."}}}To flat projection format:  {"host": "...", "port": 5432, "dbname": "...", ...}
@NikolayS
Copy link
ContributorAuthor

@cursor review

@NikolaySNikolayS changed the titleRDS/Aurora logical refresh componentDOT NOT MERGE -- RDS/Aurora logical refresh componentDec 12, 2025
Copy link

@cursorcursorbot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

This PR is being reviewed by Cursor Bugbot

Details

You are on the Bugbot Free tier. On this plan, Bugbot will review limited PRs each billing cycle.

To receive Bugbot reviews on all of your PRs, visit theCursor dashboard to activate Pro and start your 14-day free trial.

The function could return success prematurely if DBLab's status stillshowed StatusFinished from a previous refresh. Since /full-refreshtriggers the operation in a goroutine, polling immediately after couldsee stale status.Now tracks whether refresh has actually started before acceptingStatusFinished as success. This prevents premature clone deletion.
@NikolayS
Copy link
ContributorAuthor

@cursor review again

@cursor
Copy link

cursorbot commentedDec 13, 2025

Cursor Agent can help with this pull request. Just@cursor in comments and I'll start working on changes in this branch.
Learn more about Cursor Agents

…tates- Add immediate status check before ticker loop in WaitForRefreshComplete  to avoid unnecessary delay of up to pollInterval (30s) on first check- Align IsRefreshInProgress with WaitForRefreshComplete by including  StatusPending and StatusRenewed as in-progress states, preventing  potential duplicate refresh triggers
@NikolayS
Copy link
ContributorAuthor

@cursor check it again

@cursor
Copy link

cursorbot commentedDec 13, 2025
edited
Loading

I've reviewed therds-refresh component again.All previously identified bugs have been fixed:

Review summary

✅ Bug 1: Race condition (premature completion on stale status) - FIXED

TheWaitForRefreshComplete function properly uses therefreshStarted flag to prevent returning success on staleStatusFinished status.

✅ Bug 2: Inconsistent refresh status check - FIXED

TheIsRefreshInProgress function now includes all active states, matchingWaitForRefreshComplete:

// IsRefreshInProgress checks if a refresh is currently in progress.// Considers all active states: refreshing, snapshotting, pending, renewed.func (c*DBLabClient)IsRefreshInProgress(ctx context.Context) (bool,error) {status,err:=c.GetStatus(ctx)iferr!=nil {returnfalse,err}switchstatus.Retrieving.Status {caseStatusRefreshing,StatusSnapshotting,StatusPending,StatusRenewed:returntrue,nildefault:returnfalse,nil}}

✅ Bug 3: Delayed first status check - FIXED

TheWaitForRefreshComplete now performs an immediate status check before starting the polling loop:

// immediate first checkifdone,err:=checkStatus();err!=nil {returnerr}elseifdone {returnnil}ticker:=time.NewTicker(pollInterval)

No further issues found. The code looks ready for review from a correctness standpoint.


Open in Cursor Open in Web

Learn more about Cursor Agents

@NikolayS
Copy link
ContributorAuthor

@agneum could you please have a look at this (of course, we need to bring this to gitlab and test thoroughly)

- Add problem/solution section with ASCII diagram- Add prerequisites section- Restructure quick start with numbered steps- Add configuration reference table- Simplify scheduling examples (Docker cron, K8s, ECS)- Add network requirements diagram- Add cost estimates- Improve troubleshooting section
@NikolayS
Copy link
ContributorAuthor

@cursor review the instructions in README, and scrutinize it -- any potential issues?

Copy link

@cursorcursorbot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

✅ Bugbot reviewed your changes and found no bugs!


- Add 'Why?' section explaining xmin horizon / bloat problem- Reduce README from ~400 to ~200 lines- Reduce CLI help from ~70 to ~40 lines- Use collapsible sections for K8s/ECS examples- Add troubleshooting table format- Focus on essential info only
- Use 'RDS clone' instead of 'clone' to avoid confusion with DBLab clones- Use 'RDS snapshot' instead of 'snapshot' to avoid confusion with DBLab snapshots- Update description: 'Perform full refresh from RDS/Aurora snapshots (logical mode)'
Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment

Reviewers

@cursorcursor[bot]cursor[bot] left review comments

At least 2 approving reviews are required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@NikolayS@claude

[8]ページ先頭

©2009-2025 Movatter.jp