Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

Commit4ad0896

Browse files
committed
Create common infrastructure for cross-version upgrade testing.
To test pg_upgrade across major PG versions, we have to be able tomodify or drop any old objects with no-longer-supported properties,and we have to be able to deal with cosmetic changes in pg_dump output.Up to now, the buildfarm and pg_upgrade's own test infrastructure hadseparate implementations of the former, and we had nothing but veryad-hoc rules for the latter (including an arbitrary threshold on howmany lines of unchecked diff were okay!). This patch creates a Perlmodule that can be shared by both those use-cases, and adds logicthat deals with pg_dump output diffs in a much more tightly definedfashion.This largely supersedes previous efforts in commits0df9641,9814ff5, and62be9e4, which developed a SQL-script-based solutionfor the task of dropping old objects. There was nothing fundamentallywrong with that work in itself, but it had no basis for solving theoutput-formatting problem. The most plausible way to deal withformatting is to build a Perl module that can perform editing on thedump files; and once we commit to that, it makes more sense for thesame module to also embed the knowledge of what has to be done fordropping old objects.Back-patch versions of the helper module as far as 9.2, tosupport buildfarm animals that still test that far back.It's also necessary to back-patch PostgreSQL/Version.pm,because the new code depends on that. I fixed up pg_upgrade's002_pg_upgrade.pl in v15, but did not look into back-patchingit further than that.Tom Lane and Andrew DunstanDiscussion:https://postgr.es/m/891521.1673657296@sss.pgh.pa.us
1 parentac01fa6 commit4ad0896

File tree

4 files changed

+622
-147
lines changed

4 files changed

+622
-147
lines changed

‎src/bin/pg_upgrade/TESTING‎

Lines changed: 28 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -3,21 +3,21 @@ THE SHORT VERSION
33

44
On non-Windows machines, you can execute the testing process
55
described below by running the following command in this directory:
6+
67
make check
78

89
This will run the TAP tests to run pg_upgrade, performing an upgrade
910
from the version in this source tree to a new instance of the same
1011
version.
1112

12-
Testing an upgrade from a different version requires a dump to set up
13-
the contents of this instance, with its set of binaries. The following
14-
variables are available to control the test (see DETAILS below about
15-
the creation of the dump):
13+
Testing an upgrade from a different PG version is also possible, and
14+
provides a more thorough test that pg_upgrade does what it's meant for.
15+
This requires both a source tree and an installed tree for the old
16+
version, as well as a dump file to set up the instance to be upgraded.
17+
The following environment variables must be set to enable this testing:
1618
export olddump=...somewhere/dump.sql(old version's dump)
1719
export oldinstall=...otherversion/(old version's install base path)
18-
19-
Finally, the tests can be done by running
20-
make check
20+
See DETAILS below for more information about creation of the dump.
2121

2222
DETAILS
2323
-------
@@ -26,22 +26,32 @@ The most effective way to test pg_upgrade, aside from testing on user
2626
data, is by upgrading the PostgreSQL regression database.
2727

2828
This testing process first requires the creation of a valid regression
29-
database dump that canbethen used for $olddump. Such files contain
29+
database dump that can then be used for $olddump. Such files contain
3030
most database features and are specific to each major version of Postgres.
3131

3232
Here are the steps needed to create a dump file:
3333

3434
1) Create and populate the regression database in the old cluster.
3535
This database can be created by running 'make installcheck' from
36-
src/test/regress using its source code tree.
36+
src/test/regress in the old version's source code tree.
37+
38+
If you like, you can also populate regression databases for one or
39+
more contrib modules by running 'make installcheck USE_MODULE_DB=1'
40+
in their directories. (USE_MODULE_DB is essential so that the
41+
pg_upgrade test script will understand which database is which.)
3742

3843
2) Use pg_dumpall to dump out the contents of the instance, including the
39-
regression database, in the shape of a SQL file. This requires the *old*
40-
cluster's pg_dumpall so as the dump created is compatible with the
41-
version of the cluster it is dumped into.
42-
43-
Once the dump is created, it can be repeatedly used with $olddump and
44-
`make check`, that automates the dump of the old database, its upgrade,
45-
the dump out of the new database and the comparison of the dumps between
46-
the old and new databases. The contents of the dumps can also be manually
47-
compared.
44+
regression database(s), into a SQL file. Use the *old* version's
45+
pg_dumpall so that the dump created is compatible with that version.
46+
47+
Once the dump file is created, it can be used repeatedly. Set $olddump
48+
to point to the dump file and run 'make check' or 'make installcheck'
49+
in the new version's src/bin/pg_upgrade directory. (If you included any
50+
contrib databases in the old dump, you must use 'make installcheck' and
51+
ensure that the corresponding contrib modules have been installed in
52+
the new version's installation tree.) This will build a temporary cluster
53+
using the old installation's executables, populate it from the dump file,
54+
and then try to pg_upgrade it to the new version. Success is reported
55+
if pg_dumpall output matches between the pre-upgrade and post-upgrade
56+
databases. In case of trouble, manually comparing those dump files may
57+
help to isolate the problem.

‎src/bin/pg_upgrade/t/002_pg_upgrade.pl‎

Lines changed: 94 additions & 38 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@
1010

1111
use PostgreSQL::Test::Cluster;
1212
use PostgreSQL::Test::Utils;
13+
use PostgreSQL::Test::AdjustUpgrade;
1314
use Test::More;
1415

1516
# Generate a database with a name made of a range of ASCII characters.
@@ -30,11 +31,36 @@ sub generate_db
3031
"created database with ASCII characters from$from_char to$to_char");
3132
}
3233

34+
# Filter the contents of a dump before its use in a content comparison.
35+
# This returns the path to the filtered dump.
36+
subfilter_dump
37+
{
38+
my ($is_old,$old_version,$dump_file) =@_;
39+
my$dump_contents = slurp_file($dump_file);
40+
41+
if ($is_old)
42+
{
43+
$dump_contents = adjust_old_dumpfile($old_version,$dump_contents);
44+
}
45+
else
46+
{
47+
$dump_contents = adjust_new_dumpfile($old_version,$dump_contents);
48+
}
49+
50+
my$dump_file_filtered ="${dump_file}_filtered";
51+
open(my$dh,'>',$dump_file_filtered)
52+
||die"opening$dump_file_filtered";
53+
print$dh$dump_contents;
54+
close($dh);
55+
56+
return$dump_file_filtered;
57+
}
58+
3359
# The test of pg_upgrade requires two clusters, an old one and a new one
3460
# that gets upgraded. Before running the upgrade, a logical dump of the
3561
# old cluster is taken, and a second logical dump of the new one is taken
3662
# after the upgrade. The upgrade test passes if there are no differences
37-
# in these two dumps.
63+
#(after filtering)in these two dumps.
3864

3965
# Testing upgrades with an older version of PostgreSQL requires setting up
4066
# two environment variables, as of:
@@ -49,8 +75,10 @@ sub generate_db
4975
die"olddump or oldinstall is undefined";
5076
}
5177

52-
# Temporary location for the dumps taken
53-
my$tempdir = PostgreSQL::Test::Utils::tempdir;
78+
# Paths to the dumps taken during the tests.
79+
my$tempdir = PostgreSQL::Test::Utils::tempdir;
80+
my$dump1_file ="$tempdir/dump1.sql";
81+
my$dump2_file ="$tempdir/dump2.sql";
5482

5583
# Initialize node to upgrade
5684
my$oldnode =
@@ -60,7 +88,10 @@ sub generate_db
6088
# To increase coverage of non-standard segment size and group access without
6189
# increasing test runtime, run these tests with a custom setting.
6290
# --allow-group-access and --wal-segsize have been added in v11.
63-
$oldnode->init(extra=> ['--wal-segsize','1','--allow-group-access' ]);
91+
my%node_params = ();
92+
$node_params{extra} = ['--wal-segsize','1','--allow-group-access' ]
93+
if$oldnode->pg_version >= 11;
94+
$oldnode->init(%node_params);
6495
$oldnode->start;
6596

6697
# The default location of the source code is the root of this directory.
@@ -129,37 +160,52 @@ sub generate_db
129160
is($rc, 0,'regression tests pass');
130161
}
131162

163+
# Initialize a new node for the upgrade.
164+
my$newnode = PostgreSQL::Test::Cluster->new('new_node');
165+
$newnode->init(%node_params);
166+
167+
my$newbindir =$newnode->config_data('--bindir');
168+
my$oldbindir =$oldnode->config_data('--bindir');
169+
132170
# Before dumping, get rid of objects not existing or not supported in later
133171
# versions. This depends on the version of the old server used, and matters
134172
# only if different major versions are used for the dump.
135173
if (defined($ENV{oldinstall}))
136174
{
137-
# Note that upgrade_adapt.sql from the new version is used, to
138-
# cope with an upgrade to this version.
139-
$oldnode->command_ok(
140-
[
141-
'psql','-X',
142-
'-f',"$srcdir/src/bin/pg_upgrade/upgrade_adapt.sql",
143-
'regression'
144-
],
145-
'ran adapt script');
175+
# Consult AdjustUpgrade to find out what we need to do.
176+
my$dbnames =
177+
$oldnode->safe_psql('postgres',qq(SELECT datname FROM pg_database));
178+
my%dbnames;
179+
do {$dbnames{$_} = 1; }
180+
foreachsplit /\s+/s,$dbnames;
181+
my$adjust_cmds =
182+
adjust_database_contents($oldnode->pg_version,%dbnames);
183+
184+
foreachmy$updb (keys%$adjust_cmds)
185+
{
186+
my$upcmds =join(";\n", @{$adjust_cmds->{$updb} });
187+
188+
# For simplicity, use the newer version's psql to issue the commands.
189+
$newnode->command_ok(
190+
[
191+
'psql','-X',
192+
'-v','ON_ERROR_STOP=1',
193+
'-c',$upcmds,
194+
'-d',$oldnode->connstr($updb),
195+
],
196+
"ran version adaptation commands for database$updb");
197+
}
146198
}
147199

148-
# Initialize a new node for the upgrade.
149-
my$newnode = PostgreSQL::Test::Cluster->new('new_node');
150-
$newnode->init(extra=> ['--wal-segsize','1','--allow-group-access' ]);
151-
my$newbindir =$newnode->config_data('--bindir');
152-
my$oldbindir =$oldnode->config_data('--bindir');
153-
154200
# Take a dump before performing the upgrade as a base comparison. Note
155201
# that we need to use pg_dumpall from the new node here.
156-
$newnode->command_ok(
157-
[
158-
'pg_dumpall','--no-sync',
159-
'-d',$oldnode->connstr('postgres'),
160-
'-f',"$tempdir/dump1.sql"
161-
],
162-
'dump before running pg_upgrade');
202+
my@dump_command =(
203+
'pg_dumpall','--no-sync','-d',$oldnode->connstr('postgres'),
204+
'-f',$dump1_file);
205+
# --extra-float-digits is needed when upgrading from a version older than 11.
206+
push(@dump_command,'--extra-float-digits','0')
207+
if ($oldnode->pg_version < 12);
208+
$newnode->command_ok(\@dump_command,'dump before running pg_upgrade');
163209

164210
# After dumping, update references to the old source tree's regress.so
165211
# to point to the new tree.
@@ -173,7 +219,7 @@ sub generate_db
173219
chomp($output);
174220
my@libpaths =split("\n",$output);
175221

176-
my$dump_data = slurp_file("$tempdir/dump1.sql");
222+
my$dump_data = slurp_file($dump1_file);
177223

178224
my$newregresssrc ="$srcdir/src/test/regress";
179225
foreach (@libpaths)
@@ -183,7 +229,7 @@ sub generate_db
183229
$dump_data =~s/$libpath/$newregresssrc/g;
184230
}
185231

186-
openmy$fh,">","$tempdir/dump1.sql"ordie"could not open dump file";
232+
openmy$fh,">",$dump1_fileordie"could not open dump file";
187233
print$fh$dump_data;
188234
close$fh;
189235

@@ -284,24 +330,34 @@ sub generate_db
284330
}
285331

286332
# Second dump from the upgraded instance.
287-
$newnode->command_ok(
288-
[
289-
'pg_dumpall','--no-sync',
290-
'-d',$newnode->connstr('postgres'),
291-
'-f',"$tempdir/dump2.sql"
292-
],
293-
'dump after running pg_upgrade');
333+
@dump_command = (
334+
'pg_dumpall','--no-sync','-d',$newnode->connstr('postgres'),
335+
'-f',$dump2_file);
336+
# --extra-float-digits is needed when upgrading from a version older than 11.
337+
push(@dump_command,'--extra-float-digits','0')
338+
if ($oldnode->pg_version < 12);
339+
$newnode->command_ok(\@dump_command,'dump after running pg_upgrade');
340+
341+
# No need to apply filters on the dumps if working on the same version
342+
# for the old and new nodes.
343+
my$dump1_filtered =$dump1_file;
344+
my$dump2_filtered =$dump2_file;
345+
if ($oldnode->pg_version !=$newnode->pg_version)
346+
{
347+
$dump1_filtered = filter_dump(1,$oldnode->pg_version,$dump1_file);
348+
$dump2_filtered = filter_dump(0,$oldnode->pg_version,$dump2_file);
349+
}
294350

295351
# Compare the two dumps, there should be no differences.
296-
my$compare_res = compare("$tempdir/dump1.sql","$tempdir/dump2.sql");
352+
my$compare_res = compare($dump1_filtered,$dump2_filtered);
297353
is($compare_res, 0,'old and new dumps match after pg_upgrade');
298354

299355
# Provide more context if the dumps do not match.
300356
if ($compare_res != 0)
301357
{
302358
my ($stdout,$stderr) =
303-
run_command(['diff',"$tempdir/dump1.sql","$tempdir/dump2.sql" ]);
304-
print"=== diff of$tempdir/dump1.sql and$tempdir/dump2.sql\n";
359+
run_command(['diff','-u',$dump1_filtered,$dump2_filtered ]);
360+
print"=== diff of$dump1_filtered and$dump2_filtered\n";
305361
print"=== stdout ===\n";
306362
print$stdout;
307363
print"=== stderr ===\n";

‎src/bin/pg_upgrade/upgrade_adapt.sql‎

Lines changed: 0 additions & 91 deletions
This file was deleted.

0 commit comments

Comments
 (0)
[8]ページ先頭
©2009-2025 Movatter.jp