NotificationsYou must be signed in to change notification settings
Fork6
Star31

Commit667e65a

committed

Use TidStore for dead tuple TIDs storage during lazy vacuum.

Previously, we used a simple array for storing dead tuple IDs duringlazy vacuum, which had a number of problems:* The array used a single allocation and so was limited to 1GB.* The allocation was pessimistically sized according to table size.* Lookup with binary search was slow because of poor CPU cache and branch prediction behavior.This commit replaces that array with the TID store from commit30e1442.Since the backing radix tree makes small allocations as needed, the1GB limit is now gone. Further, the total memory used is now oftensmaller by an order of magnitude or more, depending on thedistribution of blocks and offsets. These two features should makemultiple rounds of heap scanning and index cleanup an extremely rareevent. TID lookup during index cleanup is also several times faster,even more so when index order is correlated with heap tuple order.Since there is no longer a predictable relationship between the numberof dead tuples vacuumed and the space taken up by their TIDs, thenumber of tuples no longer provides any meaningful insights for users,nor is the maximum number predictable. For that reason this commitalso changes to byte-based progress reporting, with the relevantcolumns of pg_stat_progress_vacuum renamed accordingly tomax_dead_tuple_bytes and dead_tuple_bytes.For parallel vacuum, both the TID store and supplemental informationspecific to vacuum are shared among the parallel vacuum workers. Aswith the previous array, we don't take any locks on TidStore duringparallel vacuum since writes are still only done by the leaderprocess.Bump catalog version.Reviewed-by: John Naylor, (in an earlier version) Dilip KumarDiscussion:https://postgr.es/m/CAD21AoAfOZvmfR0j8VmZorZjL7RhTiQdVttNuC4W-Shdc2a-AA%40mail.gmail.com

1 parentd5d2205 commit667e65aCopy full SHA for 667e65a

File tree

14 files changed

+226

-289

lines changed

doc/src/sgml
- config.sgml
- monitoring.sgml
src
- backend
  - access/heap
    - vacuumlazy.c
  - catalog
    - system_views.sql
  - commands
    - vacuum.c
    - vacuumparallel.c
  - storage/lmgr
    - lwlock.c
  - utils/activity
    - wait_event_names.txt
- include
  - catalog
    - catversion.h
  - commands
    - progress.h
    - vacuum.h
  - storage
    - lwlock.h
- test/regress/expected
  - rules.out
- tools/pgindent
  - typedefs.list

14 files changed

+226

-289

lines changed

`‎doc/src/sgml/config.sgml`

Lines changed: 0 additions & 12 deletions

Original file line number	Diff line number	Diff line change
`@@ -1919,11 +1919,6 @@ include_dir 'conf.d'`
`1919`	`1919`	`too high. It may be useful to control for this by separately`
`1920`	`1920`	`setting <xref linkend="guc-autovacuum-work-mem"/>.`
`1921`	`1921`	`</para>`
`1922`		`- <para>`
`1923`		`- Note that for the collection of dead tuple identifiers,`
`1924`		`- <command>VACUUM</command> is only able to utilize up to a maximum of`
`1925`		`- <literal>1GB</literal> of memory.`
`1926`		`- </para>`
`1927`	`1922`	`</listitem>`
`1928`	`1923`	`</varlistentry>`
`1929`	`1924`
`@@ -1946,13 +1941,6 @@ include_dir 'conf.d'`
`1946`	`1941`	`<filename>postgresql.conf</filename> file or on the server command`
`1947`	`1942`	`line.`
`1948`	`1943`	`</para>`
`1949`		`- <para>`
`1950`		`- For the collection of dead tuple identifiers, autovacuum is only able`
`1951`		`- to utilize up to a maximum of <literal>1GB</literal> of memory, so`
`1952`		`- setting <varname>autovacuum_work_mem</varname> to a value higher than`
`1953`		`- that has no effect on the number of dead tuples that autovacuum can`
`1954`		`- collect while scanning a table.`
`1955`		`- </para>`
`1956`	`1944`	`</listitem>`
`1957`	`1945`	`</varlistentry>`
`1958`	`1946`

`‎doc/src/sgml/monitoring.sgml`

Lines changed: 4 additions & 4 deletions

Original file line number	Diff line number	Diff line change
`@@ -6237,21 +6237,21 @@ FROM pg_stat_get_backend_idset() AS backendid;`
`6237`	`6237`
`6238`	`6238`	`<row>`
`6239`	`6239`	`<entry role="catalog_table_entry"><para role="column_definition">`
`6240`		`- <structfield>max_dead_tuples</structfield> <type>bigint</type>`
	`6240`	`+ <structfield>max_dead_tuple_bytes</structfield> <type>bigint</type>`
`6241`	`6241`	`</para>`
`6242`	`6242`	`<para>`
`6243`		`-Number of deadtuples that we can store before needing to perform`
	`6243`	`+Amount of deadtuple data that we can store before needing to perform`
`6244`	`6244`	`an index vacuum cycle, based on`
`6245`	`6245`	`<xref linkend="guc-maintenance-work-mem"/>.`
`6246`	`6246`	`</para></entry>`
`6247`	`6247`	`</row>`
`6248`	`6248`
`6249`	`6249`	`<row>`
`6250`	`6250`	`<entry role="catalog_table_entry"><para role="column_definition">`
`6251`		`- <structfield>num_dead_tuples</structfield> <type>bigint</type>`
	`6251`	`+ <structfield>dead_tuple_bytes</structfield> <type>bigint</type>`
`6252`	`6252`	`</para>`
`6253`	`6253`	`<para>`
`6254`		`-Number of deadtuples collected since the last index vacuum cycle.`
	`6254`	`+Amount of deadtuple data collected since the last index vacuum cycle.`
`6255`	`6255`	`</para></entry>`
`6256`	`6256`	`</row>`
`6257`	`6257`

0 commit comments

Comments

(0)

Movatterモバイル変換

Navigation Menu

Search code, repositories, users, issues, pull requests...

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Commit667e65a

File tree

14 files changed

14 files changed

`‎doc/src/sgml/config.sgml`

`‎doc/src/sgml/monitoring.sgml`

0 commit comments