Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

Commit52585f8

Browse files
committed
Create common infrastructure for cross-version upgrade testing.
To test pg_upgrade across major PG versions, we have to be able tomodify or drop any old objects with no-longer-supported properties,and we have to be able to deal with cosmetic changes in pg_dump output.Up to now, the buildfarm and pg_upgrade's own test infrastructure hadseparate implementations of the former, and we had nothing but veryad-hoc rules for the latter (including an arbitrary threshold on howmany lines of unchecked diff were okay!). This patch creates a Perlmodule that can be shared by both those use-cases, and adds logicthat deals with pg_dump output diffs in a much more tightly definedfashion.This largely supersedes previous efforts in commits0df9641,9814ff5, and62be9e4, which developed a SQL-script-based solutionfor the task of dropping old objects. There was nothing fundamentallywrong with that work in itself, but it had no basis for solving theoutput-formatting problem. The most plausible way to deal withformatting is to build a Perl module that can perform editing on thedump files; and once we commit to that, it makes more sense for thesame module to also embed the knowledge of what has to be done fordropping old objects.Back-patch versions of the helper module as far as 9.2, tosupport buildfarm animals that still test that far back.It's also necessary to back-patch PostgreSQL/Version.pm,because the new code depends on that. I fixed up pg_upgrade's002_pg_upgrade.pl in v15, but did not look into back-patchingit further than that.Tom Lane and Andrew DunstanDiscussion:https://postgr.es/m/891521.1673657296@sss.pgh.pa.us
1 parent980ae17 commit52585f8

File tree

4 files changed

+586
-190
lines changed

4 files changed

+586
-190
lines changed

‎src/bin/pg_upgrade/TESTING

Lines changed: 27 additions & 34 deletions
Original file line numberDiff line numberDiff line change
@@ -10,31 +10,14 @@ This will run the TAP tests to run pg_upgrade, performing an upgrade
1010
from the version in this source tree to a new instance of the same
1111
version.
1212

13-
Testing an upgrade from a different version requires a dump to set up
14-
the contents of this instance, with its set of binaries. The following
15-
variables are available to control the test (see DETAILS below about
16-
the creation of the dump):
13+
Testing an upgrade from a different PG version is also possible, and
14+
provides a more thorough test that pg_upgrade does what it's meant for.
15+
This requires both a source tree and an installed tree for the old
16+
version, as well as a dump file to set up the instance to be upgraded.
17+
The following environment variables must be set to enable this testing:
1718
export olddump=...somewhere/dump.sql(old version's dump)
1819
export oldinstall=...otherversion/(old version's install base path)
19-
20-
"filter_rules" is a variable that can be used to specify a file with custom
21-
filtering rules applied before comparing the dumps of the PostgreSQL
22-
instances near the end of the tests, in the shape of regular expressions
23-
valid for perl. This is useful to enforce certain validation cases where
24-
pg_dump could create inconsistent outputs across major versions.
25-
For example:
26-
27-
# Remove all CREATE POLICY statements
28-
s/^CREATE\sPOLICY.*//mgx
29-
# Replace REFRESH with DROP for materialized views
30-
s/^REFRESH\s(MATERIALIZED\sVIEW)/DROP $1/mgx
31-
32-
Lines beginning with '#' and empty lines are ignored. One rule can be
33-
defined per line.
34-
35-
Finally, the tests can be done by running
36-
37-
make check
20+
See DETAILS below for more information about creation of the dump.
3821

3922
You can also test the different transfer modes (--copy, --link,
4023
--clone) by setting the environment variable PG_TEST_PG_UPGRADE_MODE
@@ -52,22 +35,32 @@ The most effective way to test pg_upgrade, aside from testing on user
5235
data, is by upgrading the PostgreSQL regression database.
5336

5437
This testing process first requires the creation of a valid regression
55-
database dump that canbethen used for $olddump. Such files contain
38+
database dump that can then be used for $olddump. Such files contain
5639
most database features and are specific to each major version of Postgres.
5740

5841
Here are the steps needed to create a dump file:
5942

6043
1) Create and populate the regression database in the old cluster.
6144
This database can be created by running 'make installcheck' from
62-
src/test/regressusing its source code tree.
45+
src/test/regressin the old version's source code tree.
6346

64-
2) Use pg_dumpall to dump out the contents of the instance, including the
65-
regression database, in the shape of a SQL file. This requires the *old*
66-
cluster's pg_dumpall so as the dump creatediscompatible with the
67-
version of the cluster it is dumped into.
47+
If you like, you can also populate regression databases for one or
48+
more contrib modules by running 'make installcheck USE_MODULE_DB=1'
49+
in their directories. (USE_MODULE_DBisessential so that the
50+
pg_upgrade test script will understand which database is which.)
6851

69-
Once the dump is created, it can be repeatedly used with $olddump and
70-
`make check`, that automates the dump of the old database, its upgrade,
71-
the dump out of the new database and the comparison of the dumps between
72-
the old and new databases. The contents of the dumps can also be manually
73-
compared.
52+
2) Use pg_dumpall to dump out the contents of the instance, including the
53+
regression database(s), into a SQL file. Use the *old* version's
54+
pg_dumpall so that the dump created is compatible with that version.
55+
56+
Once the dump file is created, it can be used repeatedly. Set $olddump
57+
to point to the dump file and run 'make check' or 'make installcheck'
58+
in the new version's src/bin/pg_upgrade directory. (If you included any
59+
contrib databases in the old dump, you must use 'make installcheck' and
60+
ensure that the corresponding contrib modules have been installed in
61+
the new version's installation tree.) This will build a temporary cluster
62+
using the old installation's executables, populate it from the dump file,
63+
and then try to pg_upgrade it to the new version. Success is reported
64+
if pg_dumpall output matches between the pre-upgrade and post-upgrade
65+
databases. In case of trouble, manually comparing those dump files may
66+
help to isolate the problem.

‎src/bin/pg_upgrade/t/002_pg_upgrade.pl

Lines changed: 35 additions & 41 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@
1010

1111
use PostgreSQL::Test::Cluster;
1212
use PostgreSQL::Test::Utils;
13+
use PostgreSQL::Test::AdjustUpgrade;
1314
use Test::More;
1415

1516
# Can be changed to test the other modes.
@@ -37,37 +38,16 @@ sub generate_db
3738
# This returns the path to the filtered dump.
3839
subfilter_dump
3940
{
40-
my ($node,$dump_file) =@_;
41+
my ($is_old,$old_version,$dump_file) =@_;
4142
my$dump_contents = slurp_file($dump_file);
4243

43-
# Remove the comments.
44-
$dump_contents =~s/^\-\-.*//mgx;
45-
# Remove empty lines.
46-
$dump_contents =~s/^\n//mgx;
47-
48-
# Apply custom filtering rules, if any.
49-
if (defined($ENV{filter_rules}))
44+
if ($is_old)
5045
{
51-
my$filter_file =$ENV{filter_rules};
52-
die"no file with custom filter rules found!"unless-e$filter_file;
53-
54-
openmy$filter_handle,'<',$filter_file
55-
ordie"could not open$filter_file";
56-
while (<$filter_handle>)
57-
{
58-
my$filter_line =$_;
59-
60-
# Skip comments and empty lines
61-
nextif ($filter_line =~/^#/);
62-
nextif ($filter_line =~/^\s*$/);
63-
64-
# Apply lines with filters.
65-
note"Applying custom rule$filter_line to$dump_file";
66-
my$filter ="\$dump_contents =~$filter_line";
67-
## no critic (ProhibitStringyEval)
68-
eval$filter;
69-
}
70-
close$filter_handle;
46+
$dump_contents = adjust_old_dumpfile($old_version,$dump_contents);
47+
}
48+
else
49+
{
50+
$dump_contents = adjust_new_dumpfile($old_version,$dump_contents);
7151
}
7252

7353
my$dump_file_filtered ="${dump_file}_filtered";
@@ -83,7 +63,7 @@ sub filter_dump
8363
# that gets upgraded. Before running the upgrade, a logical dump of the
8464
# old cluster is taken, and a second logical dump of the new one is taken
8565
# after the upgrade. The upgrade test passes if there are no differences
86-
# in these two dumps.
66+
#(after filtering)in these two dumps.
8767

8868
# Testing upgrades with an older version of PostgreSQL requires setting up
8969
# two environment variables, as of:
@@ -198,15 +178,29 @@ sub filter_dump
198178
# only if different major versions are used for the dump.
199179
if (defined($ENV{oldinstall}))
200180
{
201-
# Note that upgrade_adapt.sql and psql from the new version are used,
202-
# to cope with an upgrade to this version.
203-
$newnode->command_ok(
204-
[
205-
'psql','-X',
206-
'-f',"$srcdir/src/bin/pg_upgrade/upgrade_adapt.sql",
207-
'-d',$oldnode->connstr('regression'),
208-
],
209-
'ran adapt script');
181+
# Consult AdjustUpgrade to find out what we need to do.
182+
my$dbnames =
183+
$oldnode->safe_psql('postgres',qq(SELECT datname FROM pg_database));
184+
my%dbnames;
185+
do {$dbnames{$_} = 1; }
186+
foreachsplit /\s+/s,$dbnames;
187+
my$adjust_cmds =
188+
adjust_database_contents($oldnode->pg_version,%dbnames);
189+
190+
foreachmy$updb (keys%$adjust_cmds)
191+
{
192+
my$upcmds =join(";\n", @{$adjust_cmds->{$updb} });
193+
194+
# For simplicity, use the newer version's psql to issue the commands.
195+
$newnode->command_ok(
196+
[
197+
'psql','-X',
198+
'-v','ON_ERROR_STOP=1',
199+
'-c',$upcmds,
200+
'-d',$oldnode->connstr($updb),
201+
],
202+
"ran version adaptation commands for database$updb");
203+
}
210204
}
211205

212206
# Take a dump before performing the upgrade as a base comparison. Note
@@ -359,8 +353,8 @@ sub filter_dump
359353
my$dump2_filtered =$dump2_file;
360354
if ($oldnode->pg_version !=$newnode->pg_version)
361355
{
362-
$dump1_filtered = filter_dump($oldnode,$dump1_file);
363-
$dump2_filtered = filter_dump($newnode,$dump2_file);
356+
$dump1_filtered = filter_dump(1,$oldnode->pg_version,$dump1_file);
357+
$dump2_filtered = filter_dump(0,$oldnode->pg_version,$dump2_file);
364358
}
365359

366360
# Compare the two dumps, there should be no differences.
@@ -371,7 +365,7 @@ sub filter_dump
371365
if ($compare_res != 0)
372366
{
373367
my ($stdout,$stderr) =
374-
run_command(['diff',$dump1_filtered,$dump2_filtered ]);
368+
run_command(['diff','-u',$dump1_filtered,$dump2_filtered ]);
375369
print"=== diff of$dump1_filtered and$dump2_filtered\n";
376370
print"=== stdout ===\n";
377371
print$stdout;

‎src/bin/pg_upgrade/upgrade_adapt.sql

Lines changed: 0 additions & 115 deletions
This file was deleted.

0 commit comments

Comments
 (0)
[8]ページ先頭
©2009-2025 Movatter.jp