forked frompostgres/postgres
- Notifications
You must be signed in to change notification settings - Fork6
Commit548e509
committed
Improve parallel scheduling logic in pg_dump/pg_restore.
Previously, the way this worked was that a parallel pg_dump wouldre-order the TABLE_DATA items in the dump's TOC into decreasing sizeorder, and separately re-order (some of) the INDEX items into decreasingsize order. Then pg_dump would dump the items in that order. Later,parallel pg_restore just followed the TOC order. This method had lotsof deficiencies:* TOC ordering randomly differed between parallel and non-paralleldumps, and was hard to predict in the former case, causing problemsfor building stable pg_dump test cases.* Parallel restore only followed a well-chosen order if the dump hadbeen done in parallel; in particular, this never happened for restorefrom custom-format dumps.* The best order for restore isn't necessarily the same as for dump,and it's not really static either because of locking considerations.* TABLE_DATA and INDEX items aren't the only things that might take a lotof work during restore. Scheduling was particularly stupid for the BLOBSitem, which might require lots of work during dump as well as restore,but was left to the end in either case.This patch removes the logic that changed the TOC order, fixing thetest instability problem. Instead, we sort the parallelizable itemsjust before processing them during a parallel dump. Independentlyof that, parallel restore prioritizes the ready-to-execute tasksbased on the size of the underlying table. In the case of dependenttasks such as index, constraint, or foreign key creation, the largestrelevant table is used as the metric for estimating the task length.(This is pretty crude, but it should be enough to avoid the case wewant to avoid, which is ending the run with just a few large taskssuch that we can't make use of all N workers.)Patch by me, responding to a complaint from Peter Eisentraut,who also reviewed the patch.Discussion:https://postgr.es/m/5137fe12-d0a2-4971-61b6-eb4e7e8875f8@2ndquadrant.com1 parent20bef2c commit548e509
File tree
8 files changed
+473
-229
lines changed- src/bin/pg_dump
8 files changed
+473
-229
lines changedLines changed: 0 additions & 12 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
252 | 252 | | |
253 | 253 | | |
254 | 254 | | |
255 | | - | |
256 | | - | |
257 | | - | |
258 | | - | |
259 | | - | |
260 | | - | |
261 | | - | |
262 | | - | |
263 | | - | |
264 | | - | |
265 | | - | |
266 | | - | |
267 | 255 | | |
268 | 256 | | |
269 | 257 | | |
| |||
0 commit comments
Comments
(0)