Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

Commit23a1c65

Browse files
committed
Introduce 'bbstreamer' abstraction to modularize pg_basebackup.
pg_basebackup knows how to do quite a few things with a backup that itgets from the server, like just write out the files, or compress themfirst, or even parse the tar format and inject a modifiedpostgresql.auto.conf file into the archive generated by the server.Unforatunely, this makes pg_basebackup.c a very large source file, andalso somewhat difficult to enhance, because for example the knowledgethat the server is sending us a 'tar' file rather than some other sortof archive is spread all over the place rather than centralized.In an effort to improve this situation, this commit invents a new'bbstreamer' abstraction. Each archive received from the server isfed to a bbstreamer which may choose to dispose of it or pass italong to some other bbstreamer. Chunks may also be "labelled"according to whether they are part of the payload data of a filein the archive or part of the archive metadata.So, for example, if we want to take a tar file, modify thepostgresql.auto.conf file it contains, and the gzip the resultand write it out, we can use a bbstreamer_tar_parser to parse thetar file received from the server, a bbstreamer_recovery_injectorto modify the contents of postgresql.auto.conf, abbstreamer_tar_archiver to replace the tar headers for the filemodified in the previous step with newly-built ones that arecorrect for the modified file, and a bbstreamer_gzip_writer togzip and write the resulting data. Only the objects with "tar"in the name know anything about the tar archive format, and intheory we could re-archive using some other format rather than"tar" if somebody wanted to write the code.These chances do add a substantial amount of code, but I think theresult is a lot more maintainable and extensible. pg_basebackup.citself shrinks by roughly a third, with a lot of the complexitypreviously contained there moving into the newly-added files.Patch by me. The larger patch series of which this is a part has beenreviewed and tested at various times by Andres Freund, SumantaMukherjee, Dilip Kumar, Suraj Kharage, Dipesh Pandit, Tushar Ahuja,Mark Dilger, Sergei Kornilov, and Jeevan Ladhe.Discussion:https://postgr.es/m/CA+TgmoZGwR=ZVWFeecncubEyPdwghnvfkkdBe9BLccLSiqdf9Q@mail.gmail.comDiscussion:https://postgr.es/m/CA+TgmoZvqk7UuzxsX1xjJRmMGkqoUGYTZLDCH8SmU1xTPr1Xig@mail.gmail.com
1 parent00a354a commit23a1c65

File tree

6 files changed

+1687
-727
lines changed

6 files changed

+1687
-727
lines changed

‎src/bin/pg_basebackup/Makefile

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -35,10 +35,16 @@ OBJS = \
3535
streamutil.o\
3636
walmethods.o
3737

38+
BBOBJS =\
39+
pg_basebackup.o\
40+
bbstreamer_file.o\
41+
bbstreamer_inject.o\
42+
bbstreamer_tar.o
43+
3844
all: pg_basebackup pg_receivewal pg_recvlogical
3945

40-
pg_basebackup:pg_basebackup.o$(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
41-
$(CC)$(CFLAGS)pg_basebackup.o$(OBJS)$(LDFLAGS)$(LDFLAGS_EX)$(LIBS) -o$@$(X)
46+
pg_basebackup:$(BBOBJS)$(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
47+
$(CC)$(CFLAGS)$(BBOBJS)$(OBJS)$(LDFLAGS)$(LDFLAGS_EX)$(LIBS) -o$@$(X)
4248

4349
pg_receivewal: pg_receivewal.o$(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
4450
$(CC)$(CFLAGS) pg_receivewal.o$(OBJS)$(LDFLAGS)$(LDFLAGS_EX)$(LIBS) -o$@$(X)
@@ -61,7 +67,7 @@ uninstall:
6167

6268
cleandistcleanmaintainer-clean:
6369
rm -f pg_basebackup$(X) pg_receivewal$(X) pg_recvlogical$(X)\
64-
pg_basebackup.o pg_receivewal.o pg_recvlogical.o\
70+
$(BBOBJS) pg_receivewal.o pg_recvlogical.o\
6571
$(OBJS)
6672
rm -rf tmp_check
6773

‎src/bin/pg_basebackup/bbstreamer.h

Lines changed: 217 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,217 @@
1+
/*-------------------------------------------------------------------------
2+
*
3+
* bbstreamer.h
4+
*
5+
* Each tar archive returned by the server is passed to one or more
6+
* bbstreamer objects for further processing. The bbstreamer may do
7+
* something simple, like write the archive to a file, perhaps after
8+
* compressing it, but it can also do more complicated things, like
9+
* annotating the byte stream to indicate which parts of the data
10+
* correspond to tar headers or trailing padding, vs. which parts are
11+
* payload data. A subsequent bbstreamer may use this information to
12+
* make further decisions about how to process the data; for example,
13+
* it might choose to modify the archive contents.
14+
*
15+
* Portions Copyright (c) 1996-2021, PostgreSQL Global Development Group
16+
*
17+
* IDENTIFICATION
18+
* src/bin/pg_basebackup/bbstreamer.h
19+
*-------------------------------------------------------------------------
20+
*/
21+
22+
#ifndefBBSTREAMER_H
23+
#defineBBSTREAMER_H
24+
25+
#include"lib/stringinfo.h"
26+
#include"pqexpbuffer.h"
27+
28+
structbbstreamer;
29+
structbbstreamer_ops;
30+
typedefstructbbstreamerbbstreamer;
31+
typedefstructbbstreamer_opsbbstreamer_ops;
32+
33+
/*
34+
* Each chunk of archive data passed to a bbstreamer is classified into one
35+
* of these categories. When data is first received from the remote server,
36+
* each chunk will be categorized as BBSTREAMER_UNKNOWN, and the chunks will
37+
* be of whatever size the remote server chose to send.
38+
*
39+
* If the archive is parsed (e.g. see bbstreamer_tar_parser_new()), then all
40+
* chunks should be labelled as one of the other types listed here. In
41+
* addition, there should be exactly one BBSTREAMER_MEMBER_HEADER chunk and
42+
* exactly one BBSTREAMER_MEMBER_TRAILER chunk per archive member, even if
43+
* that means a zero-length call. There can be any number of
44+
* BBSTREAMER_MEMBER_CONTENTS chunks in between those calls. There
45+
* should exactly BBSTREAMER_ARCHIVE_TRAILER chunk, and it should follow the
46+
* last BBSTREAMER_MEMBER_TRAILER chunk.
47+
*
48+
* In theory, we could need other classifications here, such as a way of
49+
* indicating an archive header, but the "tar" format doesn't need anything
50+
* else, so for the time being there's no point.
51+
*/
52+
typedefenum
53+
{
54+
BBSTREAMER_UNKNOWN,
55+
BBSTREAMER_MEMBER_HEADER,
56+
BBSTREAMER_MEMBER_CONTENTS,
57+
BBSTREAMER_MEMBER_TRAILER,
58+
BBSTREAMER_ARCHIVE_TRAILER
59+
}bbstreamer_archive_context;
60+
61+
/*
62+
* Each chunk of data that is classified as BBSTREAMER_MEMBER_HEADER,
63+
* BBSTREAMER_MEMBER_CONTENTS, or BBSTREAMER_MEMBER_TRAILER should also
64+
* pass a pointer to an instance of this struct. The details are expected
65+
* to be present in the archive header and used to fill the struct, after
66+
* which all subsequent calls for the same archive member are expected to
67+
* pass the same details.
68+
*/
69+
typedefstruct
70+
{
71+
charpathname[MAXPGPATH];
72+
pgoff_tsize;
73+
mode_tmode;
74+
uid_tuid;
75+
gid_tgid;
76+
boolis_directory;
77+
boolis_link;
78+
charlinktarget[MAXPGPATH];
79+
}bbstreamer_member;
80+
81+
/*
82+
* Generally, each type of bbstreamer will define its own struct, but the
83+
* first element should be 'bbstreamer base'. A bbstreamer that does not
84+
* require any additional private data could use this structure directly.
85+
*
86+
* bbs_ops is a pointer to the bbstreamer_ops object which contains the
87+
* function pointers appropriate to this type of bbstreamer.
88+
*
89+
* bbs_next is a pointer to the successor bbstreamer, for those types of
90+
* bbstreamer which forward data to a successor. It need not be used and
91+
* should be set to NULL when not relevant.
92+
*
93+
* bbs_buffer is a buffer for accumulating data for temporary storage. Each
94+
* type of bbstreamer makes its own decisions about whether and how to use
95+
* this buffer.
96+
*/
97+
structbbstreamer
98+
{
99+
constbbstreamer_ops*bbs_ops;
100+
bbstreamer*bbs_next;
101+
StringInfoDatabbs_buffer;
102+
};
103+
104+
/*
105+
* There are three callbacks for a bbstreamer. The 'content' callback is
106+
* called repeatedly, as described in the bbstreamer_archive_context comments.
107+
* Then, the 'finalize' callback is called once at the end, to give the
108+
* bbstreamer a chance to perform cleanup such as closing files. Finally,
109+
* because this code is running in a frontend environment where, as of this
110+
* writing, there are no memory contexts, the 'free' callback is called to
111+
* release memory. These callbacks should always be invoked using the static
112+
* inline functions defined below.
113+
*/
114+
structbbstreamer_ops
115+
{
116+
void(*content) (bbstreamer*streamer,bbstreamer_member*member,
117+
constchar*data,intlen,
118+
bbstreamer_archive_contextcontext);
119+
void(*finalize) (bbstreamer*streamer);
120+
void(*free) (bbstreamer*streamer);
121+
};
122+
123+
/* Send some content to a bbstreamer. */
124+
staticinlinevoid
125+
bbstreamer_content(bbstreamer*streamer,bbstreamer_member*member,
126+
constchar*data,intlen,
127+
bbstreamer_archive_contextcontext)
128+
{
129+
Assert(streamer!=NULL);
130+
streamer->bbs_ops->content(streamer,member,data,len,context);
131+
}
132+
133+
/* Finalize a bbstreamer. */
134+
staticinlinevoid
135+
bbstreamer_finalize(bbstreamer*streamer)
136+
{
137+
Assert(streamer!=NULL);
138+
streamer->bbs_ops->finalize(streamer);
139+
}
140+
141+
/* Free a bbstreamer. */
142+
staticinlinevoid
143+
bbstreamer_free(bbstreamer*streamer)
144+
{
145+
Assert(streamer!=NULL);
146+
streamer->bbs_ops->free(streamer);
147+
}
148+
149+
/*
150+
* This is a convenience method for use when implementing a bbstreamer; it is
151+
* not for use by outside callers. It adds the amount of data specified by
152+
* 'nbytes' to the bbstreamer's buffer and adjusts '*len' and '*data'
153+
* accordingly.
154+
*/
155+
staticinlinevoid
156+
bbstreamer_buffer_bytes(bbstreamer*streamer,constchar**data,int*len,
157+
intnbytes)
158+
{
159+
Assert(nbytes <=*len);
160+
161+
appendBinaryStringInfo(&streamer->bbs_buffer,*data,nbytes);
162+
*len-=nbytes;
163+
*data+=nbytes;
164+
}
165+
166+
/*
167+
* This is a convenence method for use when implementing a bbstreamer; it is
168+
* not for use by outsider callers. It attempts to add enough data to the
169+
* bbstreamer's buffer to reach a length of target_bytes and adjusts '*len'
170+
* and '*data' accordingly. It returns true if the target length has been
171+
* reached and false otherwise.
172+
*/
173+
staticinlinebool
174+
bbstreamer_buffer_until(bbstreamer*streamer,constchar**data,int*len,
175+
inttarget_bytes)
176+
{
177+
intbuflen=streamer->bbs_buffer.len;
178+
179+
if (buflen >=target_bytes)
180+
{
181+
/* Target length already reached; nothing to do. */
182+
return true;
183+
}
184+
185+
if (buflen+*len<target_bytes)
186+
{
187+
/* Not enough data to reach target length; buffer all of it. */
188+
bbstreamer_buffer_bytes(streamer,data,len,*len);
189+
return false;
190+
}
191+
192+
/* Buffer just enough to reach the target length. */
193+
bbstreamer_buffer_bytes(streamer,data,len,target_bytes-buflen);
194+
return true;
195+
}
196+
197+
/*
198+
* Functions for creating bbstreamer objects of various types. See the header
199+
* comments for each of these functions for details.
200+
*/
201+
externbbstreamer*bbstreamer_plain_writer_new(char*pathname,FILE*file);
202+
externbbstreamer*bbstreamer_gzip_writer_new(char*pathname,FILE*file,
203+
intcompresslevel);
204+
externbbstreamer*bbstreamer_extractor_new(constchar*basepath,
205+
constchar*(*link_map) (constchar*),
206+
void (*report_output_file) (constchar*));
207+
208+
externbbstreamer*bbstreamer_tar_parser_new(bbstreamer*next);
209+
externbbstreamer*bbstreamer_tar_archiver_new(bbstreamer*next);
210+
211+
externbbstreamer*bbstreamer_recovery_injector_new(bbstreamer*next,
212+
boolis_recovery_guc_supported,
213+
PQExpBufferrecoveryconfcontents);
214+
externvoidbbstreamer_inject_file(bbstreamer*streamer,char*pathname,
215+
char*data,intlen);
216+
217+
#endif

0 commit comments

Comments
 (0)
[8]ページ先頭
©2009-2025 Movatter.jp