Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

Commita2a6249

Browse files
committed
ci: macos: use cached macports install
A significant chunk of the time on the macos CI task is spent installingpackages using homebrew. The downloads of the packages are cached, but theinstallation needs to happen every time. We can't cache the whole homebrewinstallation, because it is too large due to pre-installed packages.Speed this up by installing packages using macports and caching theinstallation as .dmg. That's a lot faster than unpacking a tarball.In addition, don't install llvm - it wasn't enabled when building, so it'sjust a waste of time/space.This substantially speeds up the mac CI time, both in the cold cache and inthe warm cache case (the latter from ~1m20s to ~5s).It doesn't seem great to have diverging sources of packages for CI betweenbranches, so backpatch to 15 (where CI was added).Discussion:https://postgr.es/m/20230805202539.r3umyamsnctysdc7@awork3.anarazel.deBackpatch: 15-, where CI was added
1 parent881cd9e commita2a6249

File tree

2 files changed

+122
-38
lines changed

2 files changed

+122
-38
lines changed

‎.cirrus.yml

Lines changed: 25 additions & 38 deletions
Original file line numberDiff line numberDiff line change
@@ -435,8 +435,7 @@ task:
435435

436436
CIRRUS_WORKING_DIR:${HOME}/pgsql/
437437
CCACHE_DIR:${HOME}/ccache
438-
HOMEBREW_CACHE:${HOME}/homebrew-cache
439-
PERL5LIB:${HOME}/perl5/lib/perl5
438+
MACPORTS_CACHE:${HOME}/macports-cache
440439

441440
CC:ccache cc
442441
CXX:ccache c++
@@ -459,55 +458,43 @@ task:
459458
-mkdir ${HOME}/cores
460459
-sudo sysctl kern.corefile="${HOME}/cores/core.%P"
461460

462-
perl_cache:
463-
folder:~/perl5
464-
cpan_install_script:
465-
-perl -mIPC::Run -e 1 || cpan -T IPC::Run
466-
-perl -mIO::Pty -e 1 || cpan -T IO::Pty
467-
upload_caches:perl
468-
469-
470-
# XXX: Could we instead install homebrew into a cached directory? The
471-
# homebrew installation takes a good bit of time every time, even if the
472-
# packages do not need to be downloaded.
473-
homebrew_cache:
474-
folder:$HOMEBREW_CACHE
461+
# Use macports, even though homebrew is installed. The installation
462+
# of the additional packages we need would take quite a while with
463+
# homebrew, even if we cache the downloads. We can't cache all of
464+
# homebrew, because it's already large. So we use macports. To cache
465+
# the installation we create a .dmg file that we mount if it already
466+
# exists.
467+
# XXX: The reason for the direct p5.34* references is that we'd need
468+
# the large macport tree around to figure out that p5-io-tty is
469+
# actually p5.34-io-tty. Using the unversioned name works, but
470+
# updates macports every time.
471+
macports_cache:
472+
folder:${MACPORTS_CACHE}
475473
setup_additional_packages_script:|
476-
brew install \
474+
sh src/tools/ci/ci_macports_packages.sh \
477475
ccache \
478-
icu4c \
479-
krb5 \
480-
llvm \
476+
icu \
477+
kerberos5 \
481478
lz4 \
482-
make \
483479
meson \
484480
openldap \
485481
openssl \
486-
python \
487-
tcl-tk \
482+
p5.34-io-tty \
483+
p5.34-ipc-run \
484+
tcl \
488485
zstd
489-
490-
brew cleanup -s # to reduce cache size
491-
upload_caches:homebrew
486+
# Make macports install visible for subsequent steps
487+
echo PATH=/opt/local/sbin/:/opt/local/bin/:$PATH >> $CIRRUS_ENV
488+
upload_caches:macports
492489

493490
ccache_cache:
494491
folder:$CCACHE_DIR
495492
configure_script:|
496-
brewpath="/opt/homebrew"
497-
PKG_CONFIG_PATH="${brewpath}/lib/pkgconfig:${PKG_CONFIG_PATH}"
498-
499-
for pkg in icu4c krb5 openldap openssl zstd ; do
500-
pkgpath="${brewpath}/opt/${pkg}"
501-
PKG_CONFIG_PATH="${pkgpath}/lib/pkgconfig:${PKG_CONFIG_PATH}"
502-
PATH="${pkgpath}/bin:${pkgpath}/sbin:$PATH"
503-
done
504-
505-
export PKG_CONFIG_PATH PATH
506-
493+
export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/"
507494
meson setup \
508495
--buildtype=debug \
509-
-Dextra_include_dirs=${brewpath}/include \
510-
-Dextra_lib_dirs=${brewpath}/lib \
496+
-Dextra_include_dirs=/opt/local/include \
497+
-Dextra_lib_dirs=/opt/local/lib \
511498
-Dcassert=true \
512499
-Duuid=e2fs -Ddtrace=auto \
513500
-DPG_TEST_EXTRA="$PG_TEST_EXTRA" \

‎src/tools/ci/ci_macports_packages.sh

Lines changed: 97 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,97 @@
1+
#!/bin/sh
2+
3+
# Installs the passed in packages via macports. To make it fast enough
4+
# for CI, cache the installation as a .dmg file. To avoid
5+
# unnecessarily updating the cache, the cached image is only modified
6+
# when packages are installed or removed. Any package this script is
7+
# not instructed to install, will be removed again.
8+
#
9+
# This currently expects to be run in a macos cirrus-ci environment.
10+
11+
set -e
12+
# set -x
13+
14+
packages="$@"
15+
16+
macports_url="https://github.com/macports/macports-base/releases/download/v2.8.1/MacPorts-2.8.1-13-Ventura.pkg"
17+
cache_dmg="macports.hfs.dmg"
18+
19+
if ["$CIRRUS_CI"!="true" ];then
20+
echo"expect to be called within cirrus-ci"1>2
21+
exit 1
22+
fi
23+
24+
sudo mkdir -p /opt/local
25+
mkdir -p${MACPORTS_CACHE}/
26+
27+
# If we are starting from clean cache, perform a fresh macports
28+
# install. Otherwise decompress the .dmg we created previously.
29+
#
30+
# After this we have a working macports installation, with an unknown set of
31+
# packages installed.
32+
new_install=0
33+
update_cached_image=0
34+
if [-e${MACPORTS_CACHE}/${cache_dmg}.zstd ];then
35+
time zstd -T0 -d${MACPORTS_CACHE}/${cache_dmg}.zstd -o${cache_dmg}
36+
time sudo hdiutil attach -kernel${cache_dmg} -owners on -shadow${cache_dmg}.shadow -mountpoint /opt/local
37+
else
38+
new_install=1
39+
curl -fsSL -o macports.pkg"$macports_url"
40+
time sudo installer -pkg macports.pkg -target /
41+
# this is a throwaway environment, and it'd be a few lines to gin
42+
# up a correct user / group when using the cache.
43+
echo macportsuser root| sudo tee -a /opt/local/etc/macports/macports.conf
44+
fi
45+
export PATH=/opt/local/sbin/:/opt/local/bin/:$PATH
46+
47+
# mark all installed packages unrequested, that allows us to detect
48+
# packages that aren't needed anymore
49+
if [-n"$(port -q installed installed)" ];then
50+
sudo port unsetrequested installed
51+
fi
52+
53+
# if setting all the required packages as requested fails, we need
54+
# to install at least one of them
55+
if! sudo port setrequested$packages> /dev/null2>&1;then
56+
echo not all required packages installed, doing so now
57+
update_cached_image=1
58+
# to keep the image small, we deleted the ports tree from the image...
59+
sudo port selfupdate
60+
# XXX likely we'll need some other way to force an upgrade at some
61+
# point...
62+
sudo port upgrade outdated
63+
sudo port install -N$packages
64+
sudo port setrequested$packages
65+
fi
66+
67+
# check if any ports should be uninstalled
68+
if [-n"$(port -q installed rleaves)" ];then
69+
echo superflous packages installed
70+
update_cached_image=1
71+
sudo port uninstall --follow-dependencies rleaves
72+
73+
# remove prior cache contents, don't want to increase size
74+
rm -f${MACPORTS_CACHE}/*
75+
fi
76+
77+
# Shrink installation if we created / modified it
78+
if ["$new_install"-eq 1-o"$update_cached_image"-eq 1 ];then
79+
sudo /opt/local/bin/port clean --all installed
80+
sudo rm -rf /opt/local/var/macports/{software,sources}/*
81+
fi
82+
83+
# If we're starting from a clean cache, start a new image. If we have
84+
# an image, but the contents changed, update the image in the cache
85+
# location.
86+
if ["$new_install"-eq 1 ];then
87+
# use a generous size, so additional software can be installed later
88+
time sudo hdiutil create -fs HFS+ -format UDRO -size 10g -layout NONE -srcfolder /opt/local/${cache_dmg}
89+
time zstd -T -10 -z${cache_dmg} -o${MACPORTS_CACHE}/${cache_dmg}.zstd
90+
elif ["$update_cached_image"-eq 1 ];then
91+
sudo hdiutil detach /opt/local/
92+
time hdiutil convert -format UDRO${cache_dmg} -shadow${cache_dmg}.shadow -o updated.hfs.dmg
93+
rm${cache_dmg}.shadow
94+
mv updated.hfs.dmg${cache_dmg}
95+
time zstd --force -T -10 -z${cache_dmg} -o${MACPORTS_CACHE}/${cache_dmg}.zstd
96+
time sudo hdiutil attach -kernel${cache_dmg} -owners on -shadow${cache_dmg}.shadow -mountpoint /opt/local
97+
fi

0 commit comments

Comments
 (0)
[8]ページ先頭
©2009-2025 Movatter.jp