Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

Commitf83356c

Browse files
committed
Do a direct probe during postmaster startup to determine the maximum
number of openable files and the number already opened. This eliminatesdepending on sysconf(_SC_OPEN_MAX), and allows much saner behavior onplatforms where open-file slots are used up by semaphores.
1 parentb4c8dcb commitf83356c

File tree

5 files changed

+165
-109
lines changed

5 files changed

+165
-109
lines changed

‎doc/src/sgml/runtime.sgml

Lines changed: 9 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
<!--
2-
$PostgreSQL: pgsql/doc/src/sgml/runtime.sgml,v 1.241 2004/02/17 07:36:47 tgl Exp $
2+
$PostgreSQL: pgsql/doc/src/sgml/runtime.sgml,v 1.242 2004/02/23 20:45:58 tgl Exp $
33
-->
44

55
<Chapter Id="runtime">
@@ -932,19 +932,14 @@ SET ENABLE_SEQSCAN TO OFF;
932932
<listitem>
933933
<para>
934934
Sets the maximum number of simultaneously open files allowed to each
935-
server subprocess. The default is 1000. The limit actually used
936-
by the code is the smaller of this setting and the result of
937-
<literal>sysconf(_SC_OPEN_MAX)</literal>. Therefore, on systems
938-
where <function>sysconf</> returns a reasonable limit, you don't
939-
need to worry about this setting. But on some platforms
940-
(notably, most BSD systems), <function>sysconf</> returns a
941-
value that is much larger than the system can really support
942-
when a large number of processes all try to open that many
943-
files. If you find yourself seeing <quote>Too many open files</>
944-
failures, try reducing this setting. This option can only be set
945-
at server start or in the <filename>postgresql.conf</filename>
946-
configuration file; if changed in the configuration file, it
947-
only affects subsequently-started server subprocesses.
935+
server subprocess. The default is 1000. If the kernel is enforcing
936+
a safe per-process limit, you don't need to worry about this setting.
937+
But on some platforms (notably, most BSD systems), the kernel will
938+
allow individual processes to open many more files than the system
939+
can really support when a large number of processes all try to open
940+
that many files. If you find yourself seeing <quote>Too many open
941+
files</> failures, try reducing this setting.
942+
This option can only be set at server start.
948943
</para>
949944
</listitem>
950945
</varlistentry>

‎src/backend/postmaster/postmaster.c

Lines changed: 8 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,7 @@
3737
*
3838
*
3939
* IDENTIFICATION
40-
* $PostgreSQL: pgsql/src/backend/postmaster/postmaster.c,v 1.367 2004/02/17 03:54:56 momjian Exp $
40+
* $PostgreSQL: pgsql/src/backend/postmaster/postmaster.c,v 1.368 2004/02/23 20:45:59 tgl Exp $
4141
*
4242
* NOTES
4343
*
@@ -839,6 +839,12 @@ PostmasterMain(int argc, char *argv[])
839839
*/
840840
reset_shared(PostPortNumber);
841841

842+
/*
843+
* Estimate number of openable files. This must happen after setting up
844+
* semaphores, because on some platforms semaphores count as open files.
845+
*/
846+
set_max_safe_fds();
847+
842848
/*
843849
* Initialize the list of active backends.
844850
*/
@@ -848,13 +854,10 @@ PostmasterMain(int argc, char *argv[])
848854
/*
849855
* Initialize the child pid/HANDLE arrays
850856
*/
851-
/* FIXME: [fork/exec] Ideally, we would resize these arrays with changes
852-
* in MaxBackends, but this'll do as a first order solution.
853-
*/
854857
win32_childPIDArray= (pid_t*)malloc(NUM_BACKENDARRAY_ELEMS*sizeof(pid_t));
855858
win32_childHNDArray= (HANDLE*)malloc(NUM_BACKENDARRAY_ELEMS*sizeof(HANDLE));
856859
if (!win32_childPIDArray|| !win32_childHNDArray)
857-
ereport(LOG,
860+
ereport(FATAL,
858861
(errcode(ERRCODE_OUT_OF_MEMORY),
859862
errmsg("out of memory")));
860863
#endif

‎src/backend/storage/file/fd.c

Lines changed: 144 additions & 87 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@
77
* Portions Copyright (c) 1994, Regents of the University of California
88
*
99
* IDENTIFICATION
10-
* $PostgreSQL: pgsql/src/backend/storage/file/fd.c,v 1.106 2004/01/26 22:35:32 tgl Exp $
10+
* $PostgreSQL: pgsql/src/backend/storage/file/fd.c,v 1.107 2004/02/23 20:45:59 tgl Exp $
1111
*
1212
* NOTES:
1313
*
@@ -54,41 +54,50 @@
5454

5555

5656
/*
57-
* Problem: Postgres does a system(ld...) to do dynamic loading.
58-
* This will open several extra files in addition to those used by
59-
* Postgres. We need to guarantee that there are file descriptors free
60-
* for ld to use.
57+
* We must leave some file descriptors free for system(), the dynamic loader,
58+
* and other code that tries to open files without consulting fd.c. This
59+
* is the number left free. (While we can be pretty sure we won't get
60+
* EMFILE, there's never any guarantee that we won't get ENFILE due to
61+
* other processes chewing up FDs. So it's a bad idea to try to open files
62+
* without consulting fd.c. Nonetheless we cannot control all code.)
6163
*
62-
*The current solutionisto limit the number of file descriptors
63-
*that this code willallocate at one time: it leaves RESERVE_FOR_LD free.
64-
*
65-
*(Even though most dynamic loaders now use dlopen(3) or the
66-
*equivalent,theOS must stillopenseveralfilestoperform the
67-
*dynamic loading. And stdin/stdout/stderr count too. Keep this here.)
64+
*Because thisisjust a fixed setting, we are effectively assuming that
65+
*no such code willleave FDs open over the long term; otherwise the slop
66+
* is likely to be insufficient. Note in particular that we expect that
67+
*loading a shared library does not result in any permanent increase in
68+
* thenumber ofopen files. (This appearstobe true on most if not
69+
*all platforms as of Feb 2004.)
6870
*/
69-
#ifndefRESERVE_FOR_LD
70-
#defineRESERVE_FOR_LD10
71-
#endif
71+
#defineNUM_RESERVED_FDS10
7272

7373
/*
74-
* We need to ensure that we have at least some file descriptors
75-
* available to postgreSQL after we've reserved the ones for LD,
76-
* so we set that value here.
77-
*
78-
* I think 10 is an appropriate value so that's what it'll be
79-
* for now.
74+
* If we have fewer than this many usable FDs after allowing for the reserved
75+
* ones, choke.
8076
*/
81-
#ifndefFD_MINFREE
82-
#defineFD_MINFREE 10
83-
#endif
77+
#defineFD_MINFREE10
78+
8479

8580
/*
86-
* A number of platforms return values for sysconf(_SC_OPEN_MAX) that are
87-
* far beyond what they can really support. This GUC parameter limits what
88-
* we will believe.
81+
* A number of platforms allow individual processes to open many more files
82+
* than they can really support when *many* processes do the same thing.
83+
* This GUC parameter lets the DBA limit max_safe_fds to something less than
84+
* what the postmaster's initial probe suggests will work.
8985
*/
9086
intmax_files_per_process=1000;
9187

88+
/*
89+
* Maximum number of file descriptors to open for either VFD entries or
90+
* AllocateFile files. This is initialized to a conservative value, and
91+
* remains that way indefinitely in bootstrap or standalone-backend cases.
92+
* In normal postmaster operation, the postmaster calls set_max_safe_fds()
93+
* late in initialization to update the value, and that value is then
94+
* inherited by forked subprocesses.
95+
*
96+
* Note: the value of max_files_per_process is taken into account while
97+
* setting this variable, and so need not be tested separately.
98+
*/
99+
staticintmax_safe_fds=32;/* default if not changed */
100+
92101

93102
/* Debugging.... */
94103

@@ -199,7 +208,6 @@ static void FreeVfd(File file);
199208
staticintFileAccess(Filefile);
200209
staticFilefileNameOpenFile(FileNamefileName,intfileFlags,intfileMode);
201210
staticchar*filepath(constchar*filename);
202-
staticlongpg_nofile(void);
203211
staticvoidAtProcExit_Files(intcode,Datumarg);
204212
staticvoidCleanupTempFiles(boolisProcExit);
205213

@@ -236,6 +244,105 @@ pg_fdatasync(int fd)
236244
return0;
237245
}
238246

247+
/*
248+
* count_usable_fds --- count how many FDs the system will let us open,
249+
*and estimate how many are already open.
250+
*
251+
* We assume stdin (FD 0) is available for dup'ing
252+
*/
253+
staticvoid
254+
count_usable_fds(int*usable_fds,int*already_open)
255+
{
256+
int*fd;
257+
intsize;
258+
intused=0;
259+
inthighestfd=0;
260+
intj;
261+
262+
size=1024;
263+
fd= (int*)palloc(size*sizeof(int));
264+
265+
/* dup until failure ... */
266+
for (;;)
267+
{
268+
intthisfd;
269+
270+
thisfd=dup(0);
271+
if (thisfd<0)
272+
{
273+
/* Expect EMFILE or ENFILE, else it's fishy */
274+
if (errno!=EMFILE&&errno!=ENFILE)
275+
elog(WARNING,"dup(0) failed after %d successes: %m",used);
276+
break;
277+
}
278+
279+
if (used >=size)
280+
{
281+
size *=2;
282+
fd= (int*)repalloc(fd,size*sizeof(int));
283+
}
284+
fd[used++]=thisfd;
285+
286+
if (highestfd<thisfd)
287+
highestfd=thisfd;
288+
}
289+
290+
/* release the files we opened */
291+
for (j=0;j<used;j++)
292+
close(fd[j]);
293+
294+
pfree(fd);
295+
296+
/*
297+
* Return results. usable_fds is just the number of successful dups.
298+
* We assume that the system limit is highestfd+1 (remember 0 is a legal
299+
* FD number) and so already_open is highestfd+1 - usable_fds.
300+
*/
301+
*usable_fds=used;
302+
*already_open=highestfd+1-used;
303+
}
304+
305+
/*
306+
* set_max_safe_fds
307+
*Determine number of filedescriptors that fd.c is allowed to use
308+
*/
309+
void
310+
set_max_safe_fds(void)
311+
{
312+
intusable_fds;
313+
intalready_open;
314+
315+
/*
316+
* We want to set max_safe_fds to
317+
*MIN(usable_fds, max_files_per_process - already_open)
318+
* less the slop factor for files that are opened without consulting
319+
* fd.c. This ensures that we won't exceed either max_files_per_process
320+
* or the experimentally-determined EMFILE limit.
321+
*/
322+
count_usable_fds(&usable_fds,&already_open);
323+
324+
max_safe_fds=Min(usable_fds,max_files_per_process-already_open);
325+
326+
/*
327+
* Take off the FDs reserved for system() etc.
328+
*/
329+
max_safe_fds-=NUM_RESERVED_FDS;
330+
331+
/*
332+
* Make sure we still have enough to get by.
333+
*/
334+
if (max_safe_fds<FD_MINFREE)
335+
ereport(FATAL,
336+
(errcode(ERRCODE_INSUFFICIENT_RESOURCES),
337+
errmsg("insufficient file descriptors available to start server process"),
338+
errdetail("System allows %d, we need at least %d.",
339+
max_safe_fds+NUM_RESERVED_FDS,
340+
FD_MINFREE+NUM_RESERVED_FDS)));
341+
342+
elog(DEBUG2,"max_safe_fds = %d, usable_fds = %d, already_open = %d",
343+
max_safe_fds,usable_fds,already_open);
344+
}
345+
239346
/*
240347
* BasicOpenFile --- same as open(2) except can free other FDs if needed
241348
*
@@ -279,63 +386,6 @@ BasicOpenFile(FileName fileName, int fileFlags, int fileMode)
279386
return-1;/* failure */
280387
}
281388

282-
/*
283-
* pg_nofile: determine number of filedescriptors that fd.c is allowed to use
284-
*/
285-
staticlong
286-
pg_nofile(void)
287-
{
288-
staticlongno_files=0;
289-
290-
/* need do this calculation only once */
291-
if (no_files==0)
292-
{
293-
/*
294-
* Ask the system what its files-per-process limit is.
295-
*/
296-
#ifdefHAVE_SYSCONF
297-
no_files=sysconf(_SC_OPEN_MAX);
298-
if (no_files <=0)
299-
{
300-
#ifdefNOFILE
301-
no_files= (long)NOFILE;
302-
#else
303-
no_files= (long)max_files_per_process;
304-
#endif
305-
elog(LOG,"sysconf(_SC_OPEN_MAX) failed; using %ld",
306-
no_files);
307-
}
308-
#else/* !HAVE_SYSCONF */
309-
#ifdefNOFILE
310-
no_files= (long)NOFILE;
311-
#else
312-
no_files= (long)max_files_per_process;
313-
#endif
314-
#endif/* HAVE_SYSCONF */
315-
316-
/*
317-
* Some platforms return hopelessly optimistic values.Apply a
318-
* configurable upper limit.
319-
*/
320-
if (no_files> (long)max_files_per_process)
321-
no_files= (long)max_files_per_process;
322-
323-
/*
324-
* Make sure we have enough to get by after reserving some for LD.
325-
*/
326-
if ((no_files-RESERVE_FOR_LD)<FD_MINFREE)
327-
ereport(FATAL,
328-
(errcode(ERRCODE_INSUFFICIENT_RESOURCES),
329-
errmsg("insufficient file descriptors available to start server process"),
330-
errdetail("System allows %ld, we need at least %d.",
331-
no_files,RESERVE_FOR_LD+FD_MINFREE)));
332-
333-
no_files-=RESERVE_FOR_LD;
334-
}
335-
336-
returnno_files;
337-
}
338-
339389
#if defined(FDDEBUG)
340390

341391
staticvoid
@@ -439,7 +489,7 @@ LruInsert(File file)
439489

440490
if (FileIsNotOpen(file))
441491
{
442-
while (nfile+numAllocatedFiles >=pg_nofile())
492+
while (nfile+numAllocatedFiles >=max_safe_fds)
443493
{
444494
if (!ReleaseLruFile())
445495
break;
@@ -698,7 +748,7 @@ fileNameOpenFile(FileName fileName,
698748
file=AllocateVfd();
699749
vfdP=&VfdCache[file];
700750

701-
while (nfile+numAllocatedFiles >=pg_nofile())
751+
while (nfile+numAllocatedFiles >=max_safe_fds)
702752
{
703753
if (!ReleaseLruFile())
704754
break;
@@ -1042,7 +1092,14 @@ AllocateFile(char *name, char *mode)
10421092

10431093
DO_DB(elog(LOG,"AllocateFile: Allocated %d",numAllocatedFiles));
10441094

1045-
if (numAllocatedFiles >=MAX_ALLOCATED_FILES)
1095+
/*
1096+
* The test against MAX_ALLOCATED_FILES prevents us from overflowing
1097+
* allocatedFiles[]; the test against max_safe_fds prevents AllocateFile
1098+
* from hogging every one of the available FDs, which'd lead to infinite
1099+
* looping.
1100+
*/
1101+
if (numAllocatedFiles >=MAX_ALLOCATED_FILES||
1102+
numAllocatedFiles >=max_safe_fds-1)
10461103
elog(ERROR,"too many private FDs demanded");
10471104

10481105
TryAgain:

‎src/backend/utils/misc/guc.c

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@
1010
* Written by Peter Eisentraut <peter_e@gmx.net>.
1111
*
1212
* IDENTIFICATION
13-
* $PostgreSQL: pgsql/src/backend/utils/misc/guc.c,v 1.187 2004/02/17 03:54:57 momjian Exp $
13+
* $PostgreSQL: pgsql/src/backend/utils/misc/guc.c,v 1.188 2004/02/23 20:45:59 tgl Exp $
1414
*
1515
*--------------------------------------------------------------------
1616
*/
@@ -1102,7 +1102,7 @@ static struct config_int ConfigureNamesInt[] =
11021102
},
11031103

11041104
{
1105-
{"max_files_per_process",PGC_BACKEND,RESOURCES_KERNEL,
1105+
{"max_files_per_process",PGC_POSTMASTER,RESOURCES_KERNEL,
11061106
gettext_noop("Sets the maximum number of simultaneously open files for each server process."),
11071107
NULL
11081108
},

0 commit comments

Comments
 (0)
[8]ページ先頭
©2009-2025 Movatter.jp