Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

Commit62017cb

Browse files
committed
ci: macos: use cached macports install
A significant chunk of the time on the macos CI task is spent installingpackages using homebrew. The downloads of the packages are cached, but theinstallation needs to happen every time. We can't cache the whole homebrewinstallation, because it is too large due to pre-installed packages.Speed this up by installing packages using macports and caching theinstallation as .dmg. That's a lot faster than unpacking a tarball.In addition, don't install llvm - it wasn't enabled when building, so it'sjust a waste of time/space.This substantially speeds up the mac CI time, both in the cold cache and inthe warm cache case (the latter from ~1m20s to ~5s).It doesn't seem great to have diverging sources of packages for CI betweenbranches, so backpatch to 15 (where CI was added).Discussion:https://postgr.es/m/20230805202539.r3umyamsnctysdc7@awork3.anarazel.deBackpatch: 15-, where CI was added
1 parentd23e60e commit62017cb

File tree

2 files changed

+124
-40
lines changed

2 files changed

+124
-40
lines changed

‎.cirrus.yml

Lines changed: 27 additions & 40 deletions
Original file line numberDiff line numberDiff line change
@@ -233,8 +233,7 @@ task:
233233

234234
CIRRUS_WORKING_DIR:${HOME}/pgsql/
235235
CCACHE_DIR:${HOME}/ccache
236-
HOMEBREW_CACHE:${HOME}/homebrew-cache
237-
PERL5LIB:${HOME}/perl5/lib/perl5
236+
MACPORTS_CACHE:${HOME}/macports-cache
238237

239238
only_if:$CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*(macos|darwin|osx).*'
240239

@@ -251,51 +250,39 @@ task:
251250
-mkdir ${HOME}/cores
252251
-sudo sysctl kern.corefile="${HOME}/cores/core.%P"
253252

254-
perl_cache:
255-
folder:~/perl5
256-
cpan_install_script:
257-
-perl -mIPC::Run -e 1 || cpan -T IPC::Run
258-
-perl -mIO::Pty -e 1 || cpan -T IO::Pty
259-
upload_caches:perl
260-
261-
262-
# XXX: Could we instead install homebrew into a cached directory? The
263-
# homebrew installation takes a good bit of time every time, even if the
264-
# packages do not need to be downloaded.
265-
homebrew_cache:
266-
folder:$HOMEBREW_CACHE
253+
# Use macports, even though homebrew is installed. The installation
254+
# of the additional packages we need would take quite a while with
255+
# homebrew, even if we cache the downloads. We can't cache all of
256+
# homebrew, because it's already large. So we use macports. To cache
257+
# the installation we create a .dmg file that we mount if it already
258+
# exists.
259+
# XXX: The reason for the direct p5.34* references is that we'd need
260+
# the large macport tree around to figure out that p5-io-tty is
261+
# actually p5.34-io-tty. Using the unversioned name works, but
262+
# updates macports every time.
263+
macports_cache:
264+
folder:${MACPORTS_CACHE}
267265
setup_additional_packages_script:|
268-
brew install \
266+
sh src/tools/ci/ci_macports_packages.sh \
269267
ccache \
270-
icu4c \
271-
krb5 \
272-
llvm \
268+
gmake \
269+
icu \
270+
kerberos5 \
273271
lz4 \
274-
make \
275272
openldap \
276273
openssl \
277-
python \
278-
tcl-tk \
274+
p5.34-io-tty \
275+
p5.34-ipc-run \
276+
tcl \
279277
zstd
280-
281-
brew cleanup -s # to reduce cache size
282-
upload_caches:homebrew
278+
# Make macports install visible for subsequent steps
279+
echo PATH=/opt/local/sbin/:/opt/local/bin/:$PATH >> $CIRRUS_ENV
280+
upload_caches:macports
283281

284282
ccache_cache:
285283
folder:$CCACHE_DIR
286284
configure_script:|
287-
brewpath="/opt/homebrew"
288-
INCLUDES="${brewpath}/include:${INCLUDES}"
289-
LIBS="${brewpath}/lib:${LIBS}"
290-
291-
for pkg in icu4c krb5 openldap openssl zstd ; do
292-
pkgpath="${brewpath}/opt/${pkg}"
293-
INCLUDES="${pkgpath}/include:${INCLUDES}"
294-
LIBS="${pkgpath}/lib:${LIBS}"
295-
PKG_CONFIG_PATH="${pkgpath}/lib/pkgconfig:${PKG_CONFIG_PATH}"
296-
done
297-
298-
export PKG_CONFIG_PATH
285+
export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/"
299286
300287
./configure \
301288
--enable-cassert --enable-debug --enable-tap-tests \
@@ -311,13 +298,13 @@ task:
311298
--with-perl \
312299
--with-python \
313300
--with-ssl=openssl \
314-
--with-tcl --with-tclconfig=${brewpath}/opt/tcl-tk/lib/ \
301+
--with-tcl --with-tclconfig=/opt/local/lib \
315302
--with-uuid=e2fs \
316303
--with-zstd \
317304
\
318305
--prefix=${HOME}/install \
319-
--with-includes="${INCLUDES}" \
320-
--with-libs="${LIBS}" \
306+
--with-includes=/opt/local/include \
307+
--with-libs=/opt/local/lib \
321308
\
322309
CC="ccache cc" \
323310
CXX="ccache c++" \

‎src/tools/ci/ci_macports_packages.sh

Lines changed: 97 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,97 @@
1+
#!/bin/sh
2+
3+
# Installs the passed in packages via macports. To make it fast enough
4+
# for CI, cache the installation as a .dmg file. To avoid
5+
# unnecessarily updating the cache, the cached image is only modified
6+
# when packages are installed or removed. Any package this script is
7+
# not instructed to install, will be removed again.
8+
#
9+
# This currently expects to be run in a macos cirrus-ci environment.
10+
11+
set -e
12+
# set -x
13+
14+
packages="$@"
15+
16+
macports_url="https://github.com/macports/macports-base/releases/download/v2.8.1/MacPorts-2.8.1-13-Ventura.pkg"
17+
cache_dmg="macports.hfs.dmg"
18+
19+
if ["$CIRRUS_CI"!="true" ];then
20+
echo"expect to be called within cirrus-ci"1>2
21+
exit 1
22+
fi
23+
24+
sudo mkdir -p /opt/local
25+
mkdir -p${MACPORTS_CACHE}/
26+
27+
# If we are starting from clean cache, perform a fresh macports
28+
# install. Otherwise decompress the .dmg we created previously.
29+
#
30+
# After this we have a working macports installation, with an unknown set of
31+
# packages installed.
32+
new_install=0
33+
update_cached_image=0
34+
if [-e${MACPORTS_CACHE}/${cache_dmg}.zstd ];then
35+
time zstd -T0 -d${MACPORTS_CACHE}/${cache_dmg}.zstd -o${cache_dmg}
36+
time sudo hdiutil attach -kernel${cache_dmg} -owners on -shadow${cache_dmg}.shadow -mountpoint /opt/local
37+
else
38+
new_install=1
39+
curl -fsSL -o macports.pkg"$macports_url"
40+
time sudo installer -pkg macports.pkg -target /
41+
# this is a throwaway environment, and it'd be a few lines to gin
42+
# up a correct user / group when using the cache.
43+
echo macportsuser root| sudo tee -a /opt/local/etc/macports/macports.conf
44+
fi
45+
export PATH=/opt/local/sbin/:/opt/local/bin/:$PATH
46+
47+
# mark all installed packages unrequested, that allows us to detect
48+
# packages that aren't needed anymore
49+
if [-n"$(port -q installed installed)" ];then
50+
sudo port unsetrequested installed
51+
fi
52+
53+
# if setting all the required packages as requested fails, we need
54+
# to install at least one of them
55+
if! sudo port setrequested$packages> /dev/null2>&1;then
56+
echo not all required packages installed, doing so now
57+
update_cached_image=1
58+
# to keep the image small, we deleted the ports tree from the image...
59+
sudo port selfupdate
60+
# XXX likely we'll need some other way to force an upgrade at some
61+
# point...
62+
sudo port upgrade outdated
63+
sudo port install -N$packages
64+
sudo port setrequested$packages
65+
fi
66+
67+
# check if any ports should be uninstalled
68+
if [-n"$(port -q installed rleaves)" ];then
69+
echo superflous packages installed
70+
update_cached_image=1
71+
sudo port uninstall --follow-dependencies rleaves
72+
73+
# remove prior cache contents, don't want to increase size
74+
rm -f${MACPORTS_CACHE}/*
75+
fi
76+
77+
# Shrink installation if we created / modified it
78+
if ["$new_install"-eq 1-o"$update_cached_image"-eq 1 ];then
79+
sudo /opt/local/bin/port clean --all installed
80+
sudo rm -rf /opt/local/var/macports/{software,sources}/*
81+
fi
82+
83+
# If we're starting from a clean cache, start a new image. If we have
84+
# an image, but the contents changed, update the image in the cache
85+
# location.
86+
if ["$new_install"-eq 1 ];then
87+
# use a generous size, so additional software can be installed later
88+
time sudo hdiutil create -fs HFS+ -format UDRO -size 10g -layout NONE -srcfolder /opt/local/${cache_dmg}
89+
time zstd -T -10 -z${cache_dmg} -o${MACPORTS_CACHE}/${cache_dmg}.zstd
90+
elif ["$update_cached_image"-eq 1 ];then
91+
sudo hdiutil detach /opt/local/
92+
time hdiutil convert -format UDRO${cache_dmg} -shadow${cache_dmg}.shadow -o updated.hfs.dmg
93+
rm${cache_dmg}.shadow
94+
mv updated.hfs.dmg${cache_dmg}
95+
time zstd --force -T -10 -z${cache_dmg} -o${MACPORTS_CACHE}/${cache_dmg}.zstd
96+
time sudo hdiutil attach -kernel${cache_dmg} -owners on -shadow${cache_dmg}.shadow -mountpoint /opt/local
97+
fi

0 commit comments

Comments
 (0)
[8]ページ先頭
©2009-2025 Movatter.jp