Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

Commit6716f2f

Browse files
committed
Prevent references to invalid relation pages after fresh promotion
If a standby crashes after promotion before having completed its firstpost-recovery checkpoint, then the minimal recovery point which marksthe LSN position where the cluster is able to reach consistency may beset to a position older than the first end-of-recovery checkpoint whileall the WAL available should be replayed. This leads to the instancethinking that it contains inconsistent pages, causing a PANIC and a hardinstance crash even if all the WAL available has not been replayed forcertain sets of records replayed. When in crash recovery,minRecoveryPoint is expected to always be set to InvalidXLogRecPtr,which forces the recovery to replay all the WAL available, so thiscommit makes sure that the local copy of minRecoveryPoint from thecontrol file is initialized properly and stays as it is while crashrecovery is performed. Once switching to archive recovery or if crashrecovery finishes, then the local copy minRecoveryPoint can be safelyupdated.Pavan Deolasee has reported and diagnosed the failure in the firstplace, and the base fix idea to rely on the local copy ofminRecoveryPoint comes from Kyotaro Horiguchi, which has been expandedinto a full-fledged patch by me. The test included in this commit hasbeen written by Álvaro Herrera and Pavan Deolasee, which I have modifiedto make it faster and more reliable with sleep phases.Backpatch down to all supported versions where the bug appears, aka 9.3which is where the end-of-recovery checkpoint is not run by the startupprocess anymore. The test gets easily supported down to 10, still ithas been tested on all branches.Reported-by: Pavan DeolaseeDiagnosed-by: Pavan DeolaseeReviewed-by: Pavan Deolasee, Kyotaro HoriguchiAuthor: Michael Paquier, Kyotaro Horiguchi, Pavan Deolasee, ÁlvaroHerreraDiscussion:https://postgr.es/m/CABOikdPOewjNL=05K5CbNMxnNtXnQjhTx2F--4p4ruorCjukbA@mail.gmail.com
1 parent0095809 commit6716f2f

File tree

2 files changed

+157
-31
lines changed

2 files changed

+157
-31
lines changed

‎src/backend/access/transam/xlog.c

Lines changed: 70 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -817,8 +817,14 @@ static XLogSource XLogReceiptSource = 0;/* XLOG_FROM_* code */
817817
staticXLogRecPtrReadRecPtr;/* start of last record read */
818818
staticXLogRecPtrEndRecPtr;/* end+1 of last record read */
819819

820-
staticXLogRecPtrminRecoveryPoint;/* local copy of
821-
* ControlFile->minRecoveryPoint */
820+
/*
821+
* Local copies of equivalent fields in the control file. When running
822+
* crash recovery, minRecoveryPoint is set to InvalidXLogRecPtr as we
823+
* expect to replay all the WAL available, and updateMinRecoveryPoint is
824+
* switched to false to prevent any updates while replaying records.
825+
* Those values are kept consistent as long as crash recovery runs.
826+
*/
827+
staticXLogRecPtrminRecoveryPoint;
822828
staticTimeLineIDminRecoveryPointTLI;
823829
staticboolupdateMinRecoveryPoint= true;
824830

@@ -2685,20 +2691,26 @@ UpdateMinRecoveryPoint(XLogRecPtr lsn, bool force)
26852691
if (!updateMinRecoveryPoint|| (!force&&lsn <=minRecoveryPoint))
26862692
return;
26872693

2694+
/*
2695+
* An invalid minRecoveryPoint means that we need to recover all the WAL,
2696+
* i.e., we're doing crash recovery. We never modify the control file's
2697+
* value in that case, so we can short-circuit future checks here too. The
2698+
* local values of minRecoveryPoint and minRecoveryPointTLI should not be
2699+
* updated until crash recovery finishes.
2700+
*/
2701+
if (XLogRecPtrIsInvalid(minRecoveryPoint))
2702+
{
2703+
updateMinRecoveryPoint= false;
2704+
return;
2705+
}
2706+
26882707
LWLockAcquire(ControlFileLock,LW_EXCLUSIVE);
26892708

26902709
/* update local copy */
26912710
minRecoveryPoint=ControlFile->minRecoveryPoint;
26922711
minRecoveryPointTLI=ControlFile->minRecoveryPointTLI;
26932712

2694-
/*
2695-
* An invalid minRecoveryPoint means that we need to recover all the WAL,
2696-
* i.e., we're doing crash recovery. We never modify the control file's
2697-
* value in that case, so we can short-circuit future checks here too.
2698-
*/
2699-
if (minRecoveryPoint==0)
2700-
updateMinRecoveryPoint= false;
2701-
elseif (force||minRecoveryPoint<lsn)
2713+
if (force||minRecoveryPoint<lsn)
27022714
{
27032715
XLogRecPtrnewMinRecoveryPoint;
27042716
TimeLineIDnewMinRecoveryPointTLI;
@@ -3083,7 +3095,16 @@ XLogNeedsFlush(XLogRecPtr record)
30833095
*/
30843096
if (RecoveryInProgress())
30853097
{
3086-
/* Quick exit if already known updated */
3098+
/*
3099+
* An invalid minRecoveryPoint means that we need to recover all the
3100+
* WAL, i.e., we're doing crash recovery. We never modify the control
3101+
* file's value in that case, so we can short-circuit future checks
3102+
* here too.
3103+
*/
3104+
if (XLogRecPtrIsInvalid(minRecoveryPoint))
3105+
updateMinRecoveryPoint= false;
3106+
3107+
/* Quick exit if already known to be updated or cannot be updated */
30873108
if (record <=minRecoveryPoint|| !updateMinRecoveryPoint)
30883109
return false;
30893110

@@ -3097,20 +3118,8 @@ XLogNeedsFlush(XLogRecPtr record)
30973118
minRecoveryPointTLI=ControlFile->minRecoveryPointTLI;
30983119
LWLockRelease(ControlFileLock);
30993120

3100-
/*
3101-
* An invalid minRecoveryPoint means that we need to recover all the
3102-
* WAL, i.e., we're doing crash recovery. We never modify the control
3103-
* file's value in that case, so we can short-circuit future checks
3104-
* here too.
3105-
*/
3106-
if (minRecoveryPoint==0)
3107-
updateMinRecoveryPoint= false;
3108-
31093121
/* check again */
3110-
if (record <=minRecoveryPoint|| !updateMinRecoveryPoint)
3111-
return false;
3112-
else
3113-
return true;
3122+
returnrecord>minRecoveryPoint;
31143123
}
31153124

31163125
/* Quick exit if already known flushed */
@@ -4258,6 +4267,12 @@ ReadRecord(XLogReaderState *xlogreader, XLogRecPtr RecPtr, int emode,
42584267
minRecoveryPoint=ControlFile->minRecoveryPoint;
42594268
minRecoveryPointTLI=ControlFile->minRecoveryPointTLI;
42604269

4270+
/*
4271+
* The startup process can update its local copy of
4272+
* minRecoveryPoint from this point.
4273+
*/
4274+
updateMinRecoveryPoint= true;
4275+
42614276
UpdateControlFile();
42624277
LWLockRelease(ControlFileLock);
42634278

@@ -6837,9 +6852,26 @@ StartupXLOG(void)
68376852
/* No need to hold ControlFileLock yet, we aren't up far enough */
68386853
UpdateControlFile();
68396854

6840-
/* initialize our local copy of minRecoveryPoint */
6841-
minRecoveryPoint=ControlFile->minRecoveryPoint;
6842-
minRecoveryPointTLI=ControlFile->minRecoveryPointTLI;
6855+
/*
6856+
* Initialize our local copy of minRecoveryPoint. When doing crash
6857+
* recovery we want to replay up to the end of WAL. Particularly, in
6858+
* the case of a promoted standby minRecoveryPoint value in the
6859+
* control file is only updated after the first checkpoint. However,
6860+
* if the instance crashes before the first post-recovery checkpoint
6861+
* is completed then recovery will use a stale location causing the
6862+
* startup process to think that there are still invalid page
6863+
* references when checking for data consistency.
6864+
*/
6865+
if (InArchiveRecovery)
6866+
{
6867+
minRecoveryPoint=ControlFile->minRecoveryPoint;
6868+
minRecoveryPointTLI=ControlFile->minRecoveryPointTLI;
6869+
}
6870+
else
6871+
{
6872+
minRecoveryPoint=InvalidXLogRecPtr;
6873+
minRecoveryPointTLI=0;
6874+
}
68436875

68446876
/*
68456877
* Reset pgstat data, because it may be invalid after recovery.
@@ -7806,6 +7838,8 @@ CheckRecoveryConsistency(void)
78067838
if (XLogRecPtrIsInvalid(minRecoveryPoint))
78077839
return;
78087840

7841+
Assert(InArchiveRecovery);
7842+
78097843
/*
78107844
* assume that we are called in the startup process, and hence don't need
78117845
* a lock to read lastReplayedEndRecPtr
@@ -9913,11 +9947,16 @@ xlog_redo(XLogReaderState *record)
99139947
* Update minRecoveryPoint to ensure that if recovery is aborted, we
99149948
* recover back up to this point before allowing hot standby again.
99159949
* This is important if the max_* settings are decreased, to ensure
9916-
* you don't run queries against the WAL preceding the change.
9950+
* you don't run queries against the WAL preceding the change. The
9951+
* local copies cannot be updated as long as crash recovery is
9952+
* happening and we expect all the WAL to be replayed.
99179953
*/
9918-
minRecoveryPoint=ControlFile->minRecoveryPoint;
9919-
minRecoveryPointTLI=ControlFile->minRecoveryPointTLI;
9920-
if (minRecoveryPoint!=0&&minRecoveryPoint<lsn)
9954+
if (InArchiveRecovery)
9955+
{
9956+
minRecoveryPoint=ControlFile->minRecoveryPoint;
9957+
minRecoveryPointTLI=ControlFile->minRecoveryPointTLI;
9958+
}
9959+
if (minRecoveryPoint!=InvalidXLogRecPtr&&minRecoveryPoint<lsn)
99219960
{
99229961
ControlFile->minRecoveryPoint=lsn;
99239962
ControlFile->minRecoveryPointTLI=ThisTimeLineID;
Lines changed: 87 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,87 @@
1+
# Test for promotion handling with WAL records generated post-promotion
2+
# before the first checkpoint is generated. This test case checks for
3+
# invalid page references at replay based on the minimum consistent
4+
# recovery point defined.
5+
use strict;
6+
use warnings;
7+
use PostgresNode;
8+
use TestLib;
9+
use Test::Moretests=> 1;
10+
11+
# Initialize primary node
12+
my$alpha = get_new_node('alpha');
13+
$alpha->init(allows_streaming=> 1);
14+
# Setting wal_log_hints to off is important to get invalid page
15+
# references.
16+
$alpha->append_conf("postgresql.conf",<<EOF);
17+
wal_log_hints = off
18+
EOF
19+
20+
# Start the primary
21+
$alpha->start;
22+
23+
# setup/start a standby
24+
$alpha->backup('bkp');
25+
my$bravo = get_new_node('bravo');
26+
$bravo->init_from_backup($alpha,'bkp',has_streaming=> 1);
27+
$bravo->append_conf('postgresql.conf',<<EOF);
28+
checkpoint_timeout=1h
29+
checkpoint_completion_target=0.9
30+
EOF
31+
$bravo->start;
32+
33+
# Dummy table for the upcoming tests.
34+
$alpha->safe_psql('postgres','create table test1 (a int)');
35+
$alpha->safe_psql('postgres','insert into test1 select generate_series(1, 10000)');
36+
37+
# take a checkpoint
38+
$alpha->safe_psql('postgres','checkpoint');
39+
40+
# The following vacuum will set visibility map bits and create
41+
# problematic WAL records.
42+
$alpha->safe_psql('postgres','vacuum verbose test1');
43+
# Wait for last record to have been replayed on the standby.
44+
$alpha->wait_for_catchup($bravo,'replay',
45+
$alpha->lsn('insert'));
46+
47+
# Now force a checkpoint on the standby. This seems unnecessary but for "some"
48+
# reason, the previous checkpoint on the primary does not reflect on the standby
49+
# and without an explicit checkpoint, it may start redo recovery from a much
50+
# older point, which includes even create table and initial page additions.
51+
$bravo->safe_psql('postgres','checkpoint');
52+
53+
# Now just use a dummy table and run some operations to move minRecoveryPoint
54+
# beyond the previous vacuum.
55+
$alpha->safe_psql('postgres','create table test2 (a int, b text)');
56+
$alpha->safe_psql('postgres','insert into test2 select generate_series(1,10000), md5(random()::text)');
57+
$alpha->safe_psql('postgres','truncate test2');
58+
59+
# Wait again for all records to be replayed.
60+
$alpha->wait_for_catchup($bravo,'replay',
61+
$alpha->lsn('insert'));
62+
63+
# Do the promotion, which reinitializes minRecoveryPoint in the control
64+
# file so as WAL is replayed up to the end.
65+
$bravo->promote;
66+
67+
# Truncate the table on the promoted standby, vacuum and extend it
68+
# again to create new page references. The first post-recovery checkpoint
69+
# has not happened yet.
70+
$bravo->safe_psql('postgres','truncate test1');
71+
$bravo->safe_psql('postgres','vacuum verbose test1');
72+
$bravo->safe_psql('postgres','insert into test1 select generate_series(1,1000)');
73+
74+
# Now crash-stop the promoted standby and restart. This makes sure that
75+
# replay does not see invalid page references because of an invalid
76+
# minimum consistent recovery point.
77+
$bravo->stop('immediate');
78+
$bravo->start;
79+
80+
# Check state of the table after full crash recovery. All its data should
81+
# be here.
82+
my$psql_out;
83+
$bravo->psql(
84+
'postgres',
85+
"SELECT count(*) FROM test1",
86+
stdout=> \$psql_out);
87+
is($psql_out,'1000',"Check that table state is correct");

0 commit comments

Comments
 (0)
[8]ページ先頭
©2009-2025 Movatter.jp