Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

Commit643c27e

Browse files
committed
Increase distance between flush requests during bulk file copies.
copy_file() reads and writes data 64KB at a time (with default BLCKSZ),and historically has issued a pg_flush_data request after each write.This turns out to interact really badly with macOS's new APFS filesystem: a large file copy takes over 100X longer than it ought to onAPFS, as reported by Brent Dearth. While that's arguably a macOS bug,it's not clear whether Apple will do anything about it in the nearfuture, and in any case experimentation suggests that issuing flushesa bit less often can be helpful on other platforms too.Hence, rearrange the logic in copy_file() so that flush requests areissued once per N writes rather than every time through the loop.I set the FLUSH_DISTANCE to 32MB on macOS (any less than that stillresults in a noticeable speed degradation on APFS), but 1MB elsewhere.In limited testing on Linux and FreeBSD, this seems slightly fasterthan the previous code, and certainly no worse. It helps noticeablyon macOS even with the older HFS filesystem.A simpler change would have been to just increase the size of thecopy buffer without changing the loop logic, but that seems likelyto trash the processor cache without really helping much.Back-patch to 9.6 where we introduced msync() as an implementationoption for pg_flush_data(). The problem seems specific to APFS'smmap/msync support, so I don't think we need to go further back.Discussion:https://postgr.es/m/CADkxhTNv-j2jw2g8H57deMeAbfRgYBoLmVuXkC=YCFBXRuCOww@mail.gmail.com
1 parent8ec5429 commit643c27e

File tree

1 file changed

+30
-8
lines changed

1 file changed

+30
-8
lines changed

‎src/backend/storage/file/copydir.c

Lines changed: 30 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -139,10 +139,24 @@ copy_file(char *fromfile, char *tofile)
139139
intdstfd;
140140
intnbytes;
141141
off_toffset;
142+
off_tflush_offset;
142143

143-
/*Use palloc to ensure we get a maxaligned buffer */
144+
/*Size of copy buffer (read and write requests) */
144145
#defineCOPY_BUF_SIZE (8 * BLCKSZ)
145146

147+
/*
148+
* Size of data flush requests. It seems beneficial on most platforms to
149+
* do this every 1MB or so. But macOS, at least with early releases of
150+
* APFS, is really unfriendly to small mmap/msync requests, so there do it
151+
* only every 32MB.
152+
*/
153+
#if defined(__darwin__)
154+
#defineFLUSH_DISTANCE (32 * 1024 * 1024)
155+
#else
156+
#defineFLUSH_DISTANCE (1024 * 1024)
157+
#endif
158+
159+
/* Use palloc to ensure we get a maxaligned buffer */
146160
buffer=palloc(COPY_BUF_SIZE);
147161

148162
/*
@@ -163,11 +177,23 @@ copy_file(char *fromfile, char *tofile)
163177
/*
164178
* Do the data copying.
165179
*/
180+
flush_offset=0;
166181
for (offset=0;;offset+=nbytes)
167182
{
168183
/* If we got a cancel signal during the copy of the file, quit */
169184
CHECK_FOR_INTERRUPTS();
170185

186+
/*
187+
* We fsync the files later, but during the copy, flush them every so
188+
* often to avoid spamming the cache and hopefully get the kernel to
189+
* start writing them out before the fsync comes.
190+
*/
191+
if (offset-flush_offset >=FLUSH_DISTANCE)
192+
{
193+
pg_flush_data(dstfd,flush_offset,offset-flush_offset);
194+
flush_offset=offset;
195+
}
196+
171197
pgstat_report_wait_start(WAIT_EVENT_COPY_FILE_READ);
172198
nbytes=read(srcfd,buffer,COPY_BUF_SIZE);
173199
pgstat_report_wait_end();
@@ -190,15 +216,11 @@ copy_file(char *fromfile, char *tofile)
190216
errmsg("could not write to file \"%s\": %m",tofile)));
191217
}
192218
pgstat_report_wait_end();
193-
194-
/*
195-
* We fsync the files later but first flush them to avoid spamming the
196-
* cache and hopefully get the kernel to start writing them out before
197-
* the fsync comes.
198-
*/
199-
pg_flush_data(dstfd,offset,nbytes);
200219
}
201220

221+
if (offset>flush_offset)
222+
pg_flush_data(dstfd,flush_offset,offset-flush_offset);
223+
202224
if (CloseTransientFile(dstfd))
203225
ereport(ERROR,
204226
(errcode_for_file_access(),

0 commit comments

Comments
 (0)
[8]ページ先頭
©2009-2025 Movatter.jp