- Notifications
You must be signed in to change notification settings - Fork70
DOT NOT MERGE -- RDS/Aurora logical refresh component#262
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
base:master
Are you sure you want to change the base?
DOT NOT MERGE -- RDS/Aurora logical refresh component#262
Uh oh!
There was an error while loading.Please reload this page.
Conversation
Add a new component that automates DBLab full refresh using temporaryRDS/Aurora database clones created from snapshots. This enables ahassle-free data sync workflow that doesn't impact production.Features:- Create temporary RDS/Aurora clones from latest automated snapshots- Wait for clone availability with proper timeout handling- Trigger DBLab full refresh via API- Poll refresh status until completion- Clean up temporary clones automaticallyDeployment options:- AWS Lambda with SAM template and EventBridge scheduling- Standalone CLI binary for cron/manual execution- Docker container for containerized environmentsIncludes comprehensive documentation with IAM policy examplesand example configuration files.
Add a fully standalone version of the RDS/Aurora refresh tool that canbe built and deployed immediately without requiring a DBLab Enginerelease. This version has no dependencies on DBLab internal packages.The standalone component includes:- Self-contained Go module with minimal dependencies- Makefile for easy building (CLI, Lambda, Docker)- SAM template for AWS Lambda deployment- Dockerfile for container builds- Comprehensive documentationCan be used immediately by:1. go build -o rds-refresh .2. ./rds-refresh -config config.yaml
- Add UpdateSourceConfig method to update DBLab source connection before triggering refresh, ensuring DBLab knows the clone endpoint- Add source database credentials (dbName, username, password) to config for DBLab connection configuration- Remove Lambda-specific code, focusing on container/CLI deployment which better suits the long-running nature of refresh operations- Remove duplicate engine-integrated code (engine/internal/rdsrefresh and engine/cmd/rds-refresh) since standalone component is preferred- Update documentation with container deployment options (Docker, ECS Task, Kubernetes CronJob, CLI with cron)- Update workflow to include config update step before refresh trigger
DBLab expects a flat JSON structure matching ConfigProjection fields,not a nested structure. The API automatically reloads the config afterupdate (calls reloadFn internally), so no SIGHUP is needed.Changed from nested: {"retrieval": {"dbSource": {"host": "..."}}}To flat projection format: {"host": "...", "port": 5432, "dbname": "...", ...}NikolayS commentedDec 12, 2025
@cursor review |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
This PR is being reviewed by Cursor Bugbot
Details
You are on the Bugbot Free tier. On this plan, Bugbot will review limited PRs each billing cycle.
To receive Bugbot reviews on all of your PRs, visit theCursor dashboard to activate Pro and start your 14-day free trial.
Uh oh!
There was an error while loading.Please reload this page.
The function could return success prematurely if DBLab's status stillshowed StatusFinished from a previous refresh. Since /full-refreshtriggers the operation in a goroutine, polling immediately after couldsee stale status.Now tracks whether refresh has actually started before acceptingStatusFinished as success. This prevents premature clone deletion.
NikolayS commentedDec 13, 2025
@cursor review again |
Cursor Agent can help with this pull request. Just |
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
…tates- Add immediate status check before ticker loop in WaitForRefreshComplete to avoid unnecessary delay of up to pollInterval (30s) on first check- Align IsRefreshInProgress with WaitForRefreshComplete by including StatusPending and StatusRenewed as in-progress states, preventing potential duplicate refresh triggers
NikolayS commentedDec 13, 2025
@cursor check it again |
cursorbot commentedDec 13, 2025 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
I've reviewed the Review summary✅ Bug 1: Race condition (premature completion on stale status) - FIXEDThe ✅ Bug 2: Inconsistent refresh status check - FIXEDThe // IsRefreshInProgress checks if a refresh is currently in progress.// Considers all active states: refreshing, snapshotting, pending, renewed.func (c*DBLabClient)IsRefreshInProgress(ctx context.Context) (bool,error) {status,err:=c.GetStatus(ctx)iferr!=nil {returnfalse,err}switchstatus.Retrieving.Status {caseStatusRefreshing,StatusSnapshotting,StatusPending,StatusRenewed:returntrue,nildefault:returnfalse,nil}} ✅ Bug 3: Delayed first status check - FIXEDThe // immediate first checkifdone,err:=checkStatus();err!=nil {returnerr}elseifdone {returnnil}ticker:=time.NewTicker(pollInterval) No further issues found. The code looks ready for review from a correctness standpoint. Learn more about Cursor Agents |
NikolayS commentedDec 13, 2025
@agneum could you please have a look at this (of course, we need to bring this to gitlab and test thoroughly) |
- Add problem/solution section with ASCII diagram- Add prerequisites section- Restructure quick start with numbered steps- Add configuration reference table- Simplify scheduling examples (Docker cron, K8s, ECS)- Add network requirements diagram- Add cost estimates- Improve troubleshooting section
NikolayS commentedDec 13, 2025
@cursor review the instructions in README, and scrutinize it -- any potential issues? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
✅ Bugbot reviewed your changes and found no bugs!
- Add 'Why?' section explaining xmin horizon / bloat problem- Reduce README from ~400 to ~200 lines- Reduce CLI help from ~70 to ~40 lines- Use collapsible sections for K8s/ECS examples- Add troubleshooting table format- Focus on essential info only
- Use 'RDS clone' instead of 'clone' to avoid confusion with DBLab clones- Use 'RDS snapshot' instead of 'snapshot' to avoid confusion with DBLab snapshots- Update description: 'Perform full refresh from RDS/Aurora snapshots (logical mode)'
Uh oh!
There was an error while loading.Please reload this page.
Note
Introduces a standalone
rds-refreshtool that clones RDS/Aurora from snapshots, updates DBLab source, triggers full refresh, and cleans up—plus Docker/CLI, configs, and docs.rds-refreshmain.go,refresher.go).dblab.go).rds.go).config.go,config.example.yaml).Dockerfile,Makefile, module files (go.mod,go.sum).README.md).engine/configs/rds-refresh.example.yaml.engine/go.mod/go.sumwith AWS SDK (v2) indirect dependencies.Written byCursor Bugbot for commit0104395. Configurehere.