- Notifications
You must be signed in to change notification settings - Fork5.2k
Commit92fe23d
committed
Add nbtree skip scan optimization.
Teach nbtree multi-column index scans to opportunistically skip overirrelevant sections of the index given a query with no "=" conditions onone or more prefix index columns. When nbtree is passed input scan keysderived from a predicate "WHERE b = 5", new nbtree preprocessing stepsoutput "WHERE a = ANY(<every possible 'a' value>) AND b = 5" scan keys.That is, preprocessing generates a "skip array" (and an output scan key)for the omitted prefix column "a", which makes it safe to mark the scankey on "b" as required to continue the scan. The scan is therefore ableto repeatedly reposition itself by applying both the "a" and "b" keys.A skip array has "elements" that are generated procedurally and ondemand, but otherwise works just like a regular ScalarArrayOp array.Preprocessing can freely add a skip array before or after any inputScalarArrayOp arrays. Index scans with a skip array decide when andwhere to reposition the scan using the same approach as any other scanwith array keys. This design builds on the design for array advancementand primitive scan scheduling added to Postgres 17 by commit5bf748b.Testing has shown that skip scans of an index with a low cardinalityskipped prefix column can be multiple orders of magnitude faster than anequivalent full index scan (or sequential scan). In general, thecardinality of the scan's skipped column(s) limits the number of leafpages that can be skipped over.The core B-Tree operator classes on most discrete types generate theirarray elements with the help of their own custom skip support routine.This infrastructure gives nbtree a way to generate the next requiredarray element by incrementing (or decrementing) the current array value.It can reduce the number of index descents in cases where the nextpossible indexable value frequently turns out to be the next valuestored in the index. Opclasses that lack a skip support routine fallback on having nbtree "increment" (or "decrement") a skip array'scurrent element by setting the NEXT (or PRIOR) scan key flag, withoutdirectly changing the scan key's sk_argument. These sentinel valuesbehave just like any other value from an array -- though they can neverlocate equal index tuples (they can only locate the next group of indextuples containing the next set of non-sentinel values that the scan'sarrays need to advance to).A skip array's range is constrained by "contradictory" inequality keys.For example, a skip array on "x" will only generate the values 1 and 2given a qual such as "WHERE x BETWEEN 1 AND 2 AND y = 66". Such a skiparray qual usually has near-identical performance characteristics to acomparable SAOP qual "WHERE x = ANY('{1, 2}') AND y = 66". However,improved performance isn't guaranteed. Much depends on physical indexcharacteristics.B-Tree preprocessing is optimistic about skipping working out: itapplies static, generic rules when determining where to generate skiparrays, which assumes that the runtime overhead of maintaining skiparrays will pay for itself -- or lead to only a modest performance loss.As things stand, these assumptions are much too optimistic: skip arraymaintenance will lead to unacceptable regressions with unsympatheticqueries (queries whose scan can't skip over many irrelevant leaf pages).An upcoming commit will address the problems in this area by enhancing_bt_readpage's approach to saving cycles on scan key evaluation, makingit work in a way that directly considers the needs of = array keys(particularly = skip array keys).Author: Peter Geoghegan <pg@bowt.ie>Reviewed-By: Masahiro Ikeda <masahiro.ikeda@nttdata.com>Reviewed-By: Heikki Linnakangas <heikki.linnakangas@iki.fi>Reviewed-By: Matthias van de Meent <boekewurm+postgres@gmail.com>Reviewed-By: Tomas Vondra <tomas@vondra.me>Reviewed-By: Aleksander Alekseev <aleksander@timescale.com>Reviewed-By: Alena Rybakina <a.rybakina@postgrespro.ru>Discussion:https://postgr.es/m/CAH2-Wzmn1YsLzOGgjAQZdn1STSG_y8qP__vggTaPAYXJP+G4bw@mail.gmail.com1 parent3ba2cda commit92fe23d
File tree
35 files changed
+3024
-381
lines changed- doc/src/sgml
- src
- backend
- access
- index
- nbtree
- commands
- utils/adt
- include
- access
- catalog
- utils
- test/regress
- expected
- sql
- tools/pgindent
35 files changed
+3024
-381
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
207 | 207 | | |
208 | 208 | | |
209 | 209 | | |
210 | | - | |
| 210 | + | |
211 | 211 | | |
212 | 212 | | |
213 | 213 | | |
| |||
583 | 583 | | |
584 | 584 | | |
585 | 585 | | |
| 586 | + | |
| 587 | + | |
| 588 | + | |
| 589 | + | |
| 590 | + | |
| 591 | + | |
| 592 | + | |
| 593 | + | |
| 594 | + | |
| 595 | + | |
| 596 | + | |
| 597 | + | |
| 598 | + | |
| 599 | + | |
| 600 | + | |
| 601 | + | |
| 602 | + | |
| 603 | + | |
| 604 | + | |
| 605 | + | |
| 606 | + | |
| 607 | + | |
| 608 | + | |
| 609 | + | |
| 610 | + | |
| 611 | + | |
| 612 | + | |
| 613 | + | |
| 614 | + | |
586 | 615 | | |
587 | 616 | | |
588 | 617 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
835 | 835 | | |
836 | 836 | | |
837 | 837 | | |
838 | | - | |
| 838 | + | |
| 839 | + | |
839 | 840 | | |
840 | 841 | | |
841 | 842 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
460 | 460 | | |
461 | 461 | | |
462 | 462 | | |
463 | | - | |
| 463 | + | |
464 | 464 | | |
465 | | - | |
466 | | - | |
| 465 | + | |
| 466 | + | |
| 467 | + | |
| 468 | + | |
| 469 | + | |
| 470 | + | |
| 471 | + | |
| 472 | + | |
| 473 | + | |
| 474 | + | |
| 475 | + | |
| 476 | + | |
| 477 | + | |
| 478 | + | |
| 479 | + | |
| 480 | + | |
| 481 | + | |
| 482 | + | |
| 483 | + | |
| 484 | + | |
| 485 | + | |
| 486 | + | |
| 487 | + | |
| 488 | + | |
| 489 | + | |
| 490 | + | |
| 491 | + | |
| 492 | + | |
| 493 | + | |
| 494 | + | |
| 495 | + | |
| 496 | + | |
| 497 | + | |
| 498 | + | |
467 | 499 | | |
468 | 500 | | |
469 | | - | |
470 | | - | |
471 | | - | |
472 | | - | |
473 | | - | |
474 | | - | |
475 | | - | |
476 | | - | |
| 501 | + | |
| 502 | + | |
| 503 | + | |
| 504 | + | |
| 505 | + | |
| 506 | + | |
| 507 | + | |
| 508 | + | |
| 509 | + | |
| 510 | + | |
| 511 | + | |
| 512 | + | |
477 | 513 | | |
478 | 514 | | |
479 | 515 | | |
| |||
669 | 705 | | |
670 | 706 | | |
671 | 707 | | |
672 | | - | |
673 | | - | |
674 | | - | |
| 708 | + | |
| 709 | + | |
| 710 | + | |
| 711 | + | |
| 712 | + | |
| 713 | + | |
| 714 | + | |
675 | 715 | | |
676 | 716 | | |
677 | 717 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
4263 | 4263 | | |
4264 | 4264 | | |
4265 | 4265 | | |
4266 | | - | |
| 4266 | + | |
| 4267 | + | |
| 4268 | + | |
| 4269 | + | |
4267 | 4270 | | |
4268 | 4271 | | |
4269 | 4272 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
860 | 860 | | |
861 | 861 | | |
862 | 862 | | |
| 863 | + | |
| 864 | + | |
| 865 | + | |
| 866 | + | |
| 867 | + | |
| 868 | + | |
| 869 | + | |
| 870 | + | |
| 871 | + | |
| 872 | + | |
| 873 | + | |
| 874 | + | |
| 875 | + | |
| 876 | + | |
| 877 | + | |
| 878 | + | |
| 879 | + | |
| 880 | + | |
| 881 | + | |
| 882 | + | |
| 883 | + | |
| 884 | + | |
| 885 | + | |
| 886 | + | |
| 887 | + | |
| 888 | + | |
| 889 | + | |
| 890 | + | |
| 891 | + | |
| 892 | + | |
| 893 | + | |
| 894 | + | |
863 | 895 | | |
864 | 896 | | |
865 | 897 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
461 | 461 | | |
462 | 462 | | |
463 | 463 | | |
| 464 | + | |
| 465 | + | |
| 466 | + | |
| 467 | + | |
| 468 | + | |
| 469 | + | |
| 470 | + | |
464 | 471 | | |
465 | 472 | | |
466 | 473 | | |
| |||
1062 | 1069 | | |
1063 | 1070 | | |
1064 | 1071 | | |
1065 | | - | |
| 1072 | + | |
| 1073 | + | |
1066 | 1074 | | |
1067 | 1075 | | |
1068 | 1076 | | |
| |||
1075 | 1083 | | |
1076 | 1084 | | |
1077 | 1085 | | |
1078 | | - | |
| 1086 | + | |
| 1087 | + | |
1079 | 1088 | | |
1080 | 1089 | | |
1081 | 1090 | | |
| |||
1088 | 1097 | | |
1089 | 1098 | | |
1090 | 1099 | | |
1091 | | - | |
| 1100 | + | |
| 1101 | + | |
1092 | 1102 | | |
1093 | 1103 | | |
1094 | 1104 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
489 | 489 | | |
490 | 490 | | |
491 | 491 | | |
492 | | - | |
| 492 | + | |
| 493 | + | |
493 | 494 | | |
494 | 495 | | |
495 | 496 | | |
| |||
0 commit comments
Comments
(0)