Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

Commit53edc94

Browse files
committed
Fix extreme skew detection in Parallel Hash Join.
After repartitioning the inner side of a hash join that would haveexceeded the allowed size, we check if all the tuples from a parentpartition moved to one child partition. That is evidence that itcontains duplicate keys and later attempts to repartition will alsofail, so we should give up trying to limit memory (for lack of a betterfallback strategy).A thinko prevented the check from working correctly in partition 0 (theone that is partially loaded into memory already). Afterrepartitioning, we should check for extreme skew if the *parent*partition's space_exhausted flag was set, not the child partition's.The consequence was repeated futile repartitioning until per-partitiondata exceeded various limits including "ERROR: invalid DSA memory allocrequest size 1811939328", OS allocation failure, or temporary disk spaceerrors. (We could also do something about some of those symptoms, butthat's material for separate patches.)This problem only became likely when PostgreSQL 16 introduced supportfor Parallel Hash Right/Full Join, allowing NULL keys into the hashtable. Repartitioning always leaves NULL in partition 0, no matter howmany times you do it, because the hash value is all zero bits. That'sunlikely for other hashed values, but they might still have causedwasted extra effort before giving up.Back-patch to all supported releases.Reported-by: Craig Milhiser <craig@milhiser.com>Reviewed-by: Andrei Lepikhov <lepihov@gmail.com>Discussion:https://postgr.es/m/CA%2BwnhO1OfgXbmXgC4fv_uu%3DOxcDQuHvfoQ4k0DFeB0Qqd-X-rQ%40mail.gmail.com
1 parenta342979 commit53edc94

File tree

1 file changed

+12
-5
lines changed

1 file changed

+12
-5
lines changed

‎src/backend/executor/nodeHash.c

Lines changed: 12 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1250,32 +1250,39 @@ ExecParallelHashIncreaseNumBatches(HashJoinTable hashtable)
12501250
if (BarrierArriveAndWait(&pstate->grow_batches_barrier,
12511251
WAIT_EVENT_HASH_GROW_BATCHES_DECIDE))
12521252
{
1253+
ParallelHashJoinBatch*old_batches;
12531254
boolspace_exhausted= false;
12541255
boolextreme_skew_detected= false;
12551256

12561257
/* Make sure that we have the current dimensions and buckets. */
12571258
ExecParallelHashEnsureBatchAccessors(hashtable);
12581259
ExecParallelHashTableSetCurrentBatch(hashtable,0);
12591260

1261+
old_batches=dsa_get_address(hashtable->area,pstate->old_batches);
1262+
12601263
/* Are any of the new generation of batches exhausted? */
12611264
for (inti=0;i<hashtable->nbatch;++i)
12621265
{
1263-
ParallelHashJoinBatch*batch=hashtable->batches[i].shared;
1266+
ParallelHashJoinBatch*batch;
1267+
ParallelHashJoinBatch*old_batch;
1268+
intparent;
12641269

1270+
batch=hashtable->batches[i].shared;
12651271
if (batch->space_exhausted||
12661272
batch->estimated_size>pstate->space_allowed)
1267-
{
1268-
intparent;
1269-
12701273
space_exhausted= true;
12711274

1275+
parent=i %pstate->old_nbatch;
1276+
old_batch=NthParallelHashJoinBatch(old_batches,parent);
1277+
if (old_batch->space_exhausted||
1278+
batch->estimated_size>pstate->space_allowed)
1279+
{
12721280
/*
12731281
* Did this batch receive ALL of the tuples from its
12741282
* parent batch? That would indicate that further
12751283
* repartitioning isn't going to help (the hash values
12761284
* are probably all the same).
12771285
*/
1278-
parent=i %pstate->old_nbatch;
12791286
if (batch->ntuples==hashtable->batches[parent].shared->old_ntuples)
12801287
extreme_skew_detected= true;
12811288
}

0 commit comments

Comments
 (0)
[8]ページ先頭
©2009-2025 Movatter.jp