NotificationsYou must be signed in to change notification settings
Fork6
Star31

Commitad3fb04

committed

Fix race conditions in newly-added test.

Buildfarm has been failing sporadically on the new test. I was able toreproduce this by adding a random 0-10 s delay in the walreceiver, justbefore it connects to the primary. There's a race condition where node_3is promoted before it has fully caught up with node_1, leading to divergedtimelines. When node_1 is later reconfigured as standby following node_3,it fails to catch up:LOG: primary server contains no more WAL on requested timeline 1LOG: new timeline 2 forked off current database system timeline 1 before current recovery point 0/30000A0That's the situation where you'd need to use pg_rewind, but in this caseit happens already when we are just setting up the actual pg_rewindscenario we want to test, so change the test so that it waits untilnode_3 is connected and fully caught up before promoting it, so that youget a clean, controlled failover.Also rewrite some of the comments, for clarity. The existing commentsdetailed what each step in the test did, but didn't give a good overviewof the situation the steps were trying to create.For reasons I don't understand, the test setup had to be written slightlydifferently in 9.6 and 9.5 than in later versions. The 9.5/9.6 versionneeded node 1 to be reinitialized from backup, whereas in later versionsit could be shut down and reconfigured to be a standby. But even 9.5 shouldsupport "clean switchover", where primary makes sure that pending WAL isreplicated to standby on shutdown. It would be nice to figure out what'sgoing on there, but that's independent of pg_rewind and the scenario thatthis test tests.Discussion:https://www.postgresql.org/message-id/b0a3b95b-82d2-6089-6892-40570f8c5e60%40iki.fi

1 parentf3f033a commitad3fb04Copy full SHA for ad3fb04

File tree

1 file changed

+20

-13

lines changed

src/bin/pg_rewind/t
- 008_min_recovery_point.pl

1 file changed

+20

-13

lines changed

`‎src/bin/pg_rewind/t/008_min_recovery_point.pl`

Lines changed: 20 additions & 13 deletions

Original file line number	Diff line number	Diff line change
`@@ -50,54 +50,61 @@`
`50`	`50`	`$node_1->safe_psql('postgres','CREATE TABLE public.bar (t TEXT)');`
`51`	`51`	`$node_1->safe_psql('postgres',"INSERT INTO public.bar VALUES ('in both')");`
`52`	`52`
`53`		`-`
`54`		`-# Take backup`
	`53`	`+#`
	`54`	`+# Create node_2 and node_3 as standbys following node_1`
	`55`	`+#`
`55`	`56`	`my$backup_name ='my_backup';`
`56`	`57`	`$node_1->backup($backup_name);`
`57`	`58`
`58`		`-# Create streaming standby from backup`
`59`	`59`	`my$node_2 = get_new_node('node_2');`
`60`	`60`	`$node_2->init_from_backup($node_1,$backup_name,`
`61`	`61`	`has_streaming=> 1);`
`62`	`62`	`$node_2->start;`
`63`	`63`
`64`		`-# Create streaming standby from backup`
`65`	`64`	`my$node_3 = get_new_node('node_3');`
`66`	`65`	`$node_3->init_from_backup($node_1,$backup_name,`
`67`	`66`	`has_streaming=> 1);`
`68`	`67`	`$node_3->start;`
`69`	`68`
`70`		`-# Stop node_1`
	`69`	`+# Wait until node 3 has connected and caught up`
	`70`	`+my$lsn =$node_1->lsn('insert');`
	`71`	`+$node_1->wait_for_catchup('node_3','replay',$lsn);`
`71`	`72`
	`73`	`+#`
	`74`	`+# Swap the roles of node_1 and node_3, so that node_1 follows node_3.`
	`75`	`+#`
`72`	`76`	`$node_1->stop('fast');`
`73`		`-`
`74`		`-# Promote node_3`
`75`	`77`	`$node_3->promote;`
`76`	`78`
`77`		`-# node_1 rejoins node_3`
`78`		`-`
	`79`	`+# reconfigure node_1 as a standby following node_3`
`79`	`80`	`my$node_3_connstr =$node_3->connstr;`
`80`		`-`
`81`	`81`	`$node_1->append_conf('postgresql.conf',qq(`
`82`	`82`	`primary_conninfo='$node_3_connstr'`
`83`	`83`	`));`
`84`	`84`	`$node_1->set_standby_mode();`
`85`	`85`	`$node_1->start();`
`86`	`86`
`87`		`-# node_2 follows node_3`
`88`		`-`
	`87`	`+# also reconfigure node_2 to follow node_3`
`89`	`88`	`$node_2->append_conf('postgresql.conf',qq(`
`90`	`89`	`primary_conninfo='$node_3_connstr'`
`91`	`90`	`));`
`92`	`91`	`$node_2->restart();`
`93`	`92`
`94`		`-# Promote node_1`
	`93`	`+#`
	`94`	`+# Promote node_1, to create a split-brain scenario.`
	`95`	`+#`
	`96`	`+`
	`97`	`+# make sure node_1 is full caught up with node_3 first`
	`98`	`+$lsn =$node_3->lsn('insert');`
	`99`	`+$node_3->wait_for_catchup('node_1','replay',$lsn);`
`95`	`100`
`96`	`101`	`$node_1->promote;`
`97`	`102`
	`103`	`+#`
`98`	`104`	`# We now have a split-brain with two primaries. Insert a row on both to`
`99`	`105`	`# demonstratively create a split brain. After the rewind, we should only`
`100`	`106`	`# see the insert on 1, as the insert on node 3 is rewound away.`
	`107`	`+#`
`101`	`108`	`$node_1->safe_psql('postgres',"INSERT INTO public.foo (t) VALUES ('keep this')");`
`102`	`109`
`103`	`110`	`# Insert more rows in node 1, to bump up the XID counter. Otherwise, if`

0 commit comments

Comments

(0)

Movatterモバイル変換

Navigation Menu

Search code, repositories, users, issues, pull requests...

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Commitad3fb04

File tree

1 file changed

1 file changed

`‎src/bin/pg_rewind/t/008_min_recovery_point.pl`

0 commit comments