Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

Commit507069d

Browse files
committed
Add option to include WAL in base backup
When included, this makes the base backup a complete working"clone" of the initial database, ready to have a postmasterstarted against it without the need to set up any log archivingor similar.Magnus Hagander, reviewed by Fujii Masao and Heikki Linnakangas
1 parent5d5678d commit507069d

File tree

8 files changed

+197
-44
lines changed

8 files changed

+197
-44
lines changed

‎doc/src/sgml/protocol.sgml

Lines changed: 17 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1460,7 +1460,7 @@ The commands accepted in walsender mode are:
14601460
</varlistentry>
14611461

14621462
<varlistentry>
1463-
<term>BASE_BACKUP [<literal>LABEL</literal> <replaceable>'label'</replaceable>] [<literal>PROGRESS</literal>] [<literal>FAST</literal>]</term>
1463+
<term>BASE_BACKUP [<literal>LABEL</literal> <replaceable>'label'</replaceable>] [<literal>PROGRESS</literal>] [<literal>FAST</literal>] [<literal>WAL</literal>]</term>
14641464
<listitem>
14651465
<para>
14661466
Instructs the server to start streaming a base backup.
@@ -1505,6 +1505,18 @@ The commands accepted in walsender mode are:
15051505
</para>
15061506
</listitem>
15071507
</varlistentry>
1508+
1509+
<varlistentry>
1510+
<term><literal>WAL</literal></term>
1511+
<listitem>
1512+
<para>
1513+
Include the necessary WAL segments in the backup. This will include
1514+
all the files between start and stop backup in the
1515+
<filename>pg_xlog</filename> directory of the base directory tar
1516+
file.
1517+
</para>
1518+
</listitem>
1519+
</varlistentry>
15081520
</variablelist>
15091521
</para>
15101522
<para>
@@ -1561,7 +1573,10 @@ The commands accepted in walsender mode are:
15611573
</listitem>
15621574
<listitem>
15631575
<para>
1564-
<filename>pg_xlog</> (including subdirectories)
1576+
<filename>pg_xlog</>, including subdirectories. If the backup is run
1577+
with wal files included, a synthesized version of pg_xlog will be
1578+
included, but it will only contain the files necessary for the
1579+
backup to work, not the rest of the contents.
15651580
</para>
15661581
</listitem>
15671582
</itemizedlist>

‎doc/src/sgml/ref/pg_basebackup.sgml

Lines changed: 30 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -144,6 +144,31 @@ PostgreSQL documentation
144144
</listitem>
145145
</varlistentry>
146146

147+
<varlistentry>
148+
<term><option>-x</option></term>
149+
<term><option>--xlog</option></term>
150+
<listitem>
151+
<para>
152+
Includes the required transaction log files (WAL files) in the
153+
backup. This will include all transaction logs generated during
154+
the backup. If this option is specified, it is possible to start
155+
a postmaster directly in the extracted directory without the need
156+
to consult the log archive, thus making this a completely standalone
157+
backup.
158+
</para>
159+
<note>
160+
<para>
161+
The transaction log files are collected at the end of the backup.
162+
Therefore, it is necessary for the
163+
<xref linkend="guc-wal-keep-segments"> parameter to be set high
164+
enough that the log is not removed before the end of the backup.
165+
If the log has been rotated when it's time to transfer it, the
166+
backup will fail and be unusable.
167+
</para>
168+
</note>
169+
</listitem>
170+
</varlistentry>
171+
147172
<varlistentry>
148173
<term><option>-Z <replaceable class="parameter">level</replaceable></option></term>
149174
<term><option>--compress=<replaceable class="parameter">level</replaceable></option></term>
@@ -164,7 +189,7 @@ PostgreSQL documentation
164189
<variablelist>
165190
<varlistentry>
166191
<term><option>-c <replaceable class="parameter">fast|spread</replaceable></option></term>
167-
<term><option>--checkpoint<replaceable class="parameter">fast|spread</replaceable></option></term>
192+
<term><option>--checkpoint=<replaceable class="parameter">fast|spread</replaceable></option></term>
168193
<listitem>
169194
<para>
170195
Sets checkpoint mode to fast or spread (default).
@@ -191,7 +216,10 @@ PostgreSQL documentation
191216
Enables progress reporting. Turning this on will deliver an approximate
192217
progress report during the backup. Since the database may change during
193218
the backup, this is only an approximation and may not end at exactly
194-
<literal>100%</literal>.
219+
<literal>100%</literal>. In particular, when WAL log is included in the
220+
backup, the total amount of data cannot be estimated in advance, and
221+
in this case the progress report will only count towards the total
222+
amount of data without WAL.
195223
</para>
196224
<para>
197225
When this is enabled, the backup will start by enumerating the size of

‎src/backend/replication/basebackup.c

Lines changed: 118 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,7 @@ typedef struct
3737
constchar*label;
3838
boolprogress;
3939
boolfastcheckpoint;
40+
boolincludewal;
4041
}basebackup_options;
4142

4243

@@ -46,11 +47,17 @@ static void _tarWriteHeader(char *filename, char *linktarget,
4647
structstat*statbuf);
4748
staticvoidsend_int8_string(StringInfoData*buf,int64intval);
4849
staticvoidSendBackupHeader(List*tablespaces);
49-
staticvoidSendBackupDirectory(char*location,char*spcoid);
5050
staticvoidbase_backup_cleanup(intcode,Datumarg);
5151
staticvoidperform_base_backup(basebackup_options*opt,DIR*tblspcdir);
5252
staticvoidparse_basebackup_options(List*options,basebackup_options*opt);
5353

54+
/*
55+
* Size of each block sent into the tar stream for larger files.
56+
*
57+
* XLogSegSize *MUST* be evenly dividable by this
58+
*/
59+
#defineTAR_SEND_SIZE 32768
60+
5461
typedefstruct
5562
{
5663
char*oid;
@@ -78,7 +85,10 @@ base_backup_cleanup(int code, Datum arg)
7885
staticvoid
7986
perform_base_backup(basebackup_options*opt,DIR*tblspcdir)
8087
{
81-
do_pg_start_backup(opt->label,opt->fastcheckpoint);
88+
XLogRecPtrstartptr;
89+
XLogRecPtrendptr;
90+
91+
startptr=do_pg_start_backup(opt->label,opt->fastcheckpoint);
8292

8393
PG_ENSURE_ERROR_CLEANUP(base_backup_cleanup, (Datum)0);
8494
{
@@ -87,12 +97,6 @@ perform_base_backup(basebackup_options *opt, DIR *tblspcdir)
8797
structdirent*de;
8898
tablespaceinfo*ti;
8999

90-
91-
/* Add a node for the base directory */
92-
ti=palloc0(sizeof(tablespaceinfo));
93-
ti->size=opt->progress ?sendDir(".",1, true) :-1;
94-
tablespaces=lappend(tablespaces,ti);
95-
96100
/* Collect information about all tablespaces */
97101
while ((de=ReadDir(tblspcdir,"pg_tblspc"))!=NULL)
98102
{
@@ -120,6 +124,10 @@ perform_base_backup(basebackup_options *opt, DIR *tblspcdir)
120124
tablespaces=lappend(tablespaces,ti);
121125
}
122126

127+
/* Add a node for the base directory at the end */
128+
ti=palloc0(sizeof(tablespaceinfo));
129+
ti->size=opt->progress ?sendDir(".",1, true) :-1;
130+
tablespaces=lappend(tablespaces,ti);
123131

124132
/* Send tablespace header */
125133
SendBackupHeader(tablespaces);
@@ -128,13 +136,102 @@ perform_base_backup(basebackup_options *opt, DIR *tblspcdir)
128136
foreach(lc,tablespaces)
129137
{
130138
tablespaceinfo*ti= (tablespaceinfo*)lfirst(lc);
139+
StringInfoDatabuf;
140+
141+
/* Send CopyOutResponse message */
142+
pq_beginmessage(&buf,'H');
143+
pq_sendbyte(&buf,0);/* overall format */
144+
pq_sendint(&buf,0,2);/* natts */
145+
pq_endmessage(&buf);
146+
147+
sendDir(ti->path==NULL ?"." :ti->path,
148+
ti->path==NULL ?1 :strlen(ti->path),
149+
false);
131150

132-
SendBackupDirectory(ti->path,ti->oid);
151+
/*
152+
* If we're including WAL, and this is the main data directory we
153+
* don't terminate the tar stream here. Instead, we will append
154+
* the xlog files below and terminate it then. This is safe since
155+
* the main data directory is always sent *last*.
156+
*/
157+
if (opt->includewal&&ti->path==NULL)
158+
{
159+
Assert(lnext(lc)==NULL);
160+
}
161+
else
162+
pq_putemptymessage('c');/* CopyDone */
133163
}
134164
}
135165
PG_END_ENSURE_ERROR_CLEANUP(base_backup_cleanup, (Datum)0);
136166

137-
do_pg_stop_backup();
167+
endptr=do_pg_stop_backup();
168+
169+
if (opt->includewal)
170+
{
171+
/*
172+
* We've left the last tar file "open", so we can now append the
173+
* required WAL files to it.
174+
*/
175+
uint32logid,
176+
logseg;
177+
uint32endlogid,
178+
endlogseg;
179+
structstatstatbuf;
180+
181+
MemSet(&statbuf,0,sizeof(statbuf));
182+
statbuf.st_mode=S_IRUSR |S_IWUSR;
183+
#ifndefWIN32
184+
statbuf.st_uid=geteuid();
185+
statbuf.st_gid=getegid();
186+
#endif
187+
statbuf.st_size=XLogSegSize;
188+
statbuf.st_mtime=time(NULL);
189+
190+
XLByteToSeg(startptr,logid,logseg);
191+
XLByteToPrevSeg(endptr,endlogid,endlogseg);
192+
193+
while (true)
194+
{
195+
/* Send another xlog segment */
196+
charfn[MAXPGPATH];
197+
inti;
198+
199+
XLogFilePath(fn,ThisTimeLineID,logid,logseg);
200+
_tarWriteHeader(fn,NULL,&statbuf);
201+
202+
/* Send the actual WAL file contents, block-by-block */
203+
for (i=0;i<XLogSegSize /TAR_SEND_SIZE;i++)
204+
{
205+
charbuf[TAR_SEND_SIZE];
206+
XLogRecPtrptr;
207+
208+
ptr.xlogid=logid;
209+
ptr.xrecoff=logseg*XLogSegSize+TAR_SEND_SIZE*i;
210+
211+
XLogRead(buf,ptr,TAR_SEND_SIZE);
212+
if (pq_putmessage('d',buf,TAR_SEND_SIZE))
213+
ereport(ERROR,
214+
(errmsg("base backup could not send data, aborting backup")));
215+
}
216+
217+
/*
218+
* Files are always fixed size, and always end on a 512 byte
219+
* boundary, so padding is never necessary.
220+
*/
221+
222+
223+
/* Advance to the next WAL file */
224+
NextLogSeg(logid,logseg);
225+
226+
/* Have we reached our stop position yet? */
227+
if (logid>endlogid||
228+
(logid==endlogid&&logseg>endlogseg))
229+
break;
230+
}
231+
232+
/* Send CopyDone message for the last tar file */
233+
pq_putemptymessage('c');
234+
}
138235
}
139236

140237
/*
@@ -147,6 +244,7 @@ parse_basebackup_options(List *options, basebackup_options *opt)
147244
boolo_label= false;
148245
boolo_progress= false;
149246
boolo_fast= false;
247+
boolo_wal= false;
150248

151249
MemSet(opt,0,sizeof(*opt));
152250
foreach(lopt,options)
@@ -180,6 +278,15 @@ parse_basebackup_options(List *options, basebackup_options *opt)
180278
opt->fastcheckpoint= true;
181279
o_fast= true;
182280
}
281+
elseif (strcmp(defel->defname,"wal")==0)
282+
{
283+
if (o_wal)
284+
ereport(ERROR,
285+
(errcode(ERRCODE_SYNTAX_ERROR),
286+
errmsg("duplicate option \"%s\"",defel->defname)));
287+
opt->includewal= true;
288+
o_wal= true;
289+
}
183290
else
184291
elog(ERROR,"option \"%s\" not recognized",
185292
defel->defname);
@@ -316,26 +423,6 @@ SendBackupHeader(List *tablespaces)
316423
pq_puttextmessage('C',"SELECT");
317424
}
318425

319-
staticvoid
320-
SendBackupDirectory(char*location,char*spcoid)
321-
{
322-
StringInfoDatabuf;
323-
324-
/* Send CopyOutResponse message */
325-
pq_beginmessage(&buf,'H');
326-
pq_sendbyte(&buf,0);/* overall format */
327-
pq_sendint(&buf,0,2);/* natts */
328-
pq_endmessage(&buf);
329-
330-
/* tar up the data directory if NULL, otherwise the tablespace */
331-
sendDir(location==NULL ?"." :location,
332-
location==NULL ?1 :strlen(location),
333-
false);
334-
335-
/* Send CopyDone message */
336-
pq_putemptymessage('c');
337-
}
338-
339426

340427
staticint64
341428
sendDir(char*path,intbasepathlen,boolsizeonly)
@@ -506,7 +593,7 @@ static void
506593
sendFile(char*filename,intbasepathlen,structstat*statbuf)
507594
{
508595
FILE*fp;
509-
charbuf[32768];
596+
charbuf[TAR_SEND_SIZE];
510597
size_tcnt;
511598
pgoff_tlen=0;
512599
size_tpad;

‎src/backend/replication/repl_gram.y

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -71,6 +71,7 @@ Node *replication_parse_result;
7171
%tokenK_LABEL
7272
%tokenK_PROGRESS
7373
%tokenK_FAST
74+
%tokenK_WAL
7475
%tokenK_START_REPLICATION
7576

7677
%type<node>command
@@ -106,7 +107,7 @@ identify_system:
106107
;
107108

108109
/*
109-
* BASE_BACKUP [LABEL <label>] [PROGRESS] [FAST]
110+
* BASE_BACKUP [LABEL'<label>'] [PROGRESS] [FAST] [WAL]
110111
*/
111112
base_backup:
112113
K_BASE_BACKUPbase_backup_opt_list
@@ -136,7 +137,12 @@ base_backup_opt:
136137
$$ = makeDefElem("fast",
137138
(Node *)makeInteger(TRUE));
138139
}
139-
140+
|K_WAL
141+
{
142+
$$ = makeDefElem("wal",
143+
(Node *)makeInteger(TRUE));
144+
}
145+
;
140146

141147
/*
142148
* START_REPLICATION %X/%X

‎src/backend/replication/repl_scanner.l

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -61,6 +61,7 @@ FAST{ return K_FAST; }
6161
IDENTIFY_SYSTEM{return K_IDENTIFY_SYSTEM; }
6262
LABEL{return K_LABEL; }
6363
PROGRESS{return K_PROGRESS; }
64+
WAL{return K_WAL; }
6465
START_REPLICATION{return K_START_REPLICATION; }
6566
","{return','; }
6667
";"{return';'; }

‎src/backend/replication/walsender.c

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -105,7 +105,6 @@ static intWalSndLoop(void);
105105
staticvoidInitWalSnd(void);
106106
staticvoidWalSndHandshake(void);
107107
staticvoidWalSndKill(intcode,Datumarg);
108-
staticvoidXLogRead(char*buf,XLogRecPtrrecptr,Sizenbytes);
109108
staticboolXLogSend(char*msgbuf,bool*caughtup);
110109
staticvoidCheckClosedConnection(void);
111110
staticvoidIdentifySystem(void);
@@ -649,8 +648,13 @@ WalSndKill(int code, Datum arg)
649648
*
650649
* XXX probably this should be improved to suck data directly from the
651650
* WAL buffers when possible.
651+
*
652+
* Will open, and keep open, one WAL segment stored in the global file
653+
* descriptor sendFile. This means if XLogRead is used once, there will
654+
* always be one descriptor left open until the process ends, but never
655+
* more than one.
652656
*/
653-
staticvoid
657+
void
654658
XLogRead(char*buf,XLogRecPtrrecptr,Sizenbytes)
655659
{
656660
XLogRecPtrstartRecPtr=recptr;

0 commit comments

Comments
 (0)
[8]ページ先頭
©2009-2025 Movatter.jp