forked frompostgres/postgres
- Notifications
You must be signed in to change notification settings - Fork6
Commit0d5f05c
committed
Allow multi-inserts during COPY into a partitioned table
CopyFrom allows multi-inserts to be used for non-partitioned tables, butthis was disabled for partitioned tables. The reason for this appearedto be that the tuple may not belong to the same partition as theprevious tuple did. Not allowing multi-inserts here greatly slowed downimports into partitioned tables. These could take twice as long as acopy to an equivalent non-partitioned table. It seems wise to dosomething about this, so this change allows the multi-inserts byflushing the so-far inserted tuples to the partition when the next tupledoes not belong to the same partition, or when the buffer fills. Thisimproves performance when the next tuple in the stream commonly belongsto the same partition as the previous tuple.In cases where the target partition changes on every tuple, usingmulti-inserts slightly slows the performance. To get around this wetrack the average size of the batches that have been inserted andadaptively enable or disable multi-inserts based on the size of thebatch. Some testing was done and the regression only seems to existwhen the average size of the insert batch is close to 1, so let's justenable multi-inserts when the average size is at least 1.3. Moreperformance testing might reveal a better number for, this, but sincethe slowdown was only 1-2% it does not seem critical enough to spend toomuch time calculating it. In any case it may depend on other factorsrather than just the size of the batch.Allowing multi-inserts for partitions required a bit of work around theper-tuple memory contexts as we must flush the tuples when the nexttuple does not belong the same partition. In which case there is nogood time to reset the per-tuple context, as we've already built the newtuple by this time. In order to work around this we maintain twoper-tuple contexts and just switch between them every time the partitionchanges and reset the old one. This does mean that the first of eachbatch of tuples is not allocated in the same memory context as theothers, but that does not matter since we only reset the context oncethe previous batch has been inserted.Author: David Rowley <david.rowley@2ndquadrant.com>Reviewed-by: Melanie Plageman <melanieplageman@gmail.com>1 parentb6d6488 commit0d5f05c
File tree
6 files changed
+330
-88
lines changed- src
- backend
- commands
- executor
- include/executor
- test/regress
- input
- output
6 files changed
+330
-88
lines changed0 commit comments
Comments
(0)