- Notifications
You must be signed in to change notification settings - Fork20
Kiln helps you maintain product tiles for VMware Tanzu Operations Manager.
License
pivotal-cf/kiln
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
Kiln bakes tiles
Kiln helps tile developers build products forVMware Tanzu Operations Manager. It providesan opinionated folder structure and templating capabilities. It is designed to be usedboth in CI environments and in command-line to produce a tile.
More information for those just getting started can be found in theOps Manager Tile Developer Guide .Looking at anexample kiln tile may also be helpful
To install thekiln CLI
install with Homebrew
brew tap pivotal-cf/kiln https://github.com/pivotal-cf/kilnbrew install kiln
download from thereleases page
export KILN_VERSIONKILN_VERSION="$(curl -H"Accept: application/vnd.github.v3+json"'https://api.github.com/repos/pivotal-cf/kiln/releases?per_page=1'| jq -r'.[0].name')"curl -L -o kiln"https://github.com/pivotal-cf/kiln/releases/download/${KILN_VERSION}/kiln-darwin-${KILN_VERSION}"# check the checksumcp kiln"$(go env GOPATH)/bin"kiln version
build from source
git clone git@github.com:pivotal-cf/kiln.gitcd kiln./scripts/install.shcopy from a Docker image (to another image)
docker pull pivotalcfreleng/kiln:latest
FROM pivotalcfreleng/kiln:latest as kilnFROM ubuntuCOPY --from=kiln /kiln /usr/bin/kilnCMD /usr/bin/bash
Each tile must have a Kilnfile and Kilnfile.lock. Both are YAML files. Kiln won't generate them for you.
The code for parsing the Kilnfile and Kilnfile.lock exists in this package:"github.com/pivotal-cf/kiln/pkg/cargo".Although the package interface is not yet stable, we have found importing it directly to be useful in CI or one-off scripts.
The Specification(source)
This file contains a range of configuration in support of kiln commands.
This field must be a string.
This field should be populated with the TanzuNet product slug where this tile is published.
It is used by kiln publish.
This field must be a list of strings.
This should be populated with the names of TanzuNet User Groups to attach to a pre-release on TanzuNet.
It is used by kiln publish.
This field must be a list of objects with keys fromReleaseSourceConfig.All elements must have atype field.
The values for thetype (string) field are"bosh.io","s3","github", or"artifactory"
Seefetch documentation for more details.
Each release you want to add to your tile must have an element in this array.The "name" field with the BOSH Release Name as the value must be set here.The BOSH Release Name must be identical to the name of the release.
You must set a"name" field with the BOSH Release Name as the value must be set here.The BOSH Release Name must be identical to the name of the release.
You may set a"version" field. The value must match theconstraints specification for this library.
You may set a"github_repository" field. This should be where the BOSH Release source is maintained. It is used for generating Release Notes for your tile.
You may set a"float_always" field. When you set this,kiln glaze will not lock this release's version.
You may set a"maintenance_version_bump_policy" field. When you runkiln deglaze this field specifies how to reset the version constraint afterkiln glaze locked it.
- "LockNone": Given a glazed version value "1.2.3", this setting resets the version constraint to
"*" - "LockPatch": Given a glazed version value "1.2.3", this setting resets the version constraint to
"1.2.3" - "LockMinor": Given a glazed version value "1.2.3", this setting resets the version constraint to
"1.2.*" - "LockMajor": (default) Given a glazed version value "1.2.3", this setting resets the version constraint to
"1.*"
You may set a"slack" field. Kiln does not use this field. It can be useful for product tile Authors to know who to reach out to when something goes wrong.
You may add a list ofkiln bake flags in the Kilnfile to keep a record of how your tile was baked and to keep CI scripts simpler.
If you set add more than one element to the bake_configurations list, you need to select one by adding akiln bake --variables=tile_name=big-footprint-topology flag corresponding to a bake configuration with a- tile_name: big-footprint-topology element
These are the mappings from bake flag to each field in a bake_configurations element:
| bake_configurations element field | bake flag | documentation |
|---|---|---|
"tile_name" | --variables=tile_name= | This field is used when selecting a configuration from a list of bake_configurations. |
"metadata_filepath" | --metadata= | This should be the path to the product template entrypoint. Usually calledbase.yml |
"icon_filepath" | --icon= | This may be a path to a png file. |
"forms_directories" | --forms-directory= | This may be a list of directories. |
"instance_groups_directories" | --instance-groups-directory= | This may be a list of directories. |
"jobs_directories" | --jobs-directory= | This may be a list of directories. |
"migrations_directories" | --migrations-directory= | This may be a list of directories. |
"properties_directories" | --properties-directory= | This may be a list of directories. |
"runtime_configurations_directories" | --runtime-configs-directory= | This may be a list of directories. |
"bosh_variables_directories" | --bosh-variables-directory= | This may be a list of directories. |
"embed_files" | --embed= | This may be a list of filepaths. |
"variable_files" | --variables-file= | This may be a list of filepaths. |
The Lock File(source)
This file specifies the exact BOSH Release tarballs to package in a tile.
This is an array ofBOSH Release locks.Elements will be modified by runningkiln update-release.Each element in the releases array in the Kilnfile will have a corresponding element in the Kilnfile.lock releases array.
The release name, release version, sha1 checksum, remote_source, remote_path are fields on each element.
Usage: kiln [options] <command> [<args>] --help, -h bool prints this usage information (default: false) --version, -v bool prints the kiln release version (default: false)Commands: bake bakes a tile cache-compiled-releases Cache compiled releases fetch fetches releases find-release-version prints a json string of a remote release satisfying the Kilnfile version and stemcell constraints find-stemcell-version prints the latest stemcell version from Pivnet using the stemcell type listed in the Kilnfile generate-osm-manifest Print an OSM-format manifest. glaze Pin versions in Kilnfile to match lock. help prints this usage information publish publish tile on Pivnet re-bake re-bake constructs a tile from a bake record release-notes generates release notes from bosh-release release notes sync-with-local update the Kilnfile.lock based on local releases test Test manifest for a product update-release bumps a release to a new version update-stemcell updates stemcell and release information in Kilnfile.lock upload-release uploads a BOSH release to an s3 release_source validate validate Kilnfile and Kilnfile.lock version prints the kiln release versionIt takes release and stemcell tarballs, metadata YAML, and JavaScript migrationsas inputs and produces an OpsMan-compatible tile as its output.
The produce a tile, you simply need to be within a tile directory and execute the following command:
$ kiln bakeThis will ensure that you have the necessary releases by first callingkiln fetch.
Refer to theexample-tile for a complete example showing thedifferent features kiln supports.
Additional bake options
The--allow-only-publishable-releases flag should be used for development onlyand allows additional releases other than those specified in the kilnfile.lock tobe included in the tile
The--bosh-variables-directory flag can be used to include CredHub variabledeclarations. You should prefer the use of variables rather than Ops Managersecrets. Each.yml file in the directory should define a top-levelvariableskey.
This flag can be specified multiple times if you have organized yourvariables into subdirectories for development convenience.
Examplevariables directory.
The--download-threads flag is for those using S3 as a BOSH release source.This flag sets the number of parallel threads to download parts from S3
The--embed flag is for embedding any extra files or directories into thetile. There are very few reasons a tile developer should want to do this, but ifyou do, you can include these extra files here. The flag can be specifiedmultiple times to embed multiple files or directories.
The--final flag is to bake a final release tile. When passing the --final flag,Kiln creates a baked record file with metadata like source revision SHA, tile version, kiln version andfile checksums. This bake record file will be created under bake_records folder. Thisbake record file can later be used to re-bake the tile.
The--forms-directory flag takes a path to a directory that contains oneor more forms. The flag can be specified more than once.
To reference a form file in the directory you can use theformtemplate helper:
$ cat /path/to/metadata---form_types:- $( form "first" )Exampleforms directory.
The--icon flag takes a path to an icon file.
To include the base64'd representation of the icon you can use theicontemplate helper:
$ cat /path/to/metadata---icon_image: $( icon )The--instance-groups-directory flag takes a path to a directory that contains oneor more instance groups. The flag can be specified more than once.
To reference an instance group in the directory you can use theinstance_grouptemplate helper:
$ cat /path/to/metadata---job_types:- $( instance_group "my-instance-group" )Exampleinstance-groups directory.
The--jobs-directory flag takes a path to a directory that contains oneor more jobs. The flag can be specified more than once.
To reference a job file in the directory you can use thejobtemplate helper:
$ cat /path/to/instance-group---templates:- $( job "my-job" )- $( job "my-aliased-job" )- $( job "my-errand" )Examplejobs directory.
You may find that you want to define different job files for the same BOSH jobwith different properties. To do this you add analias key to the job whichwill be used in preference to the job name when resolving job references:
$ cat /path/to/my-aliased-job---name: my-jobalias: my-aliased-jobThe--kilnfile flag is required with kiln version v0.84.0 and laterThe flag expects filepath to a Kilnfile (default: Kilnfile). Thisfile contain links to all the bosh sources used to build a tile
See theKilnfile section for more information on Kilnfile formatting
Tile authors will also need to include a Kilnfile.lock in the same directoryas the Kilnfile.
See theKilnfile.lock section for more information on Kilnfile.lock formatting
Specify a file path to a tile metadata file for the--metadata flag. Thismetadata file will contain the contents of your tile configuration as specifiedin the OpsManager tile development documentation.
The--metadata-only flag outputs the generated metadata to stdout.This flag cannot be used with--output-file.
If your tile has JavaScript migrations, then you will need to include the--migrations-directory flag. This flag can be specified multiple times if youhave organized your migrations into subdirectories for development convenience.
Theno-confirm flag will delete extra releases in releases directory without prompting.This flag defaults totrue
The--output-file flag takes a path to the location on the filesystem whereyour tile will be created. The flag expects a full file name liketiles/my-tile-1.2.3-build.4.pivotal.
Cannot be used with--metadata-only.
The--properties-directory flag takes a path to a directory that contains oneor more blueprint property files. The flag can be specified more than once.
To reference a properties file in the directory you can use thepropertytemplate helper:
$ cat /path/to/metadata---property_blueprints:- $( property "rep_password" )Exampleproperties directory.
The--releases-directory flag takes a path to a directory that contains one ormany release tarballs. The flag can be specified more than once. This isuseful if you consume bosh releases as Concourse resources. Each release willlikely show up in the task as a separate directory. For example:
$ tree /path/to/releases|-- first| |-- cflinuxfs2-release-1.166.0.tgz| `-- consul-release-190.tgz`-- second `-- nats-release-22.tgzTo reference a release you can use therelease template helper:
$ cat /path/to/metadata---releases:- $( release "cflinuxfs2" )- $( release "consul" )- $( release "nats" )Example kiln command line:
$ kiln bake \ --version 2.0.0 \ --metadata /path/to/metadata.yml \ --releases-directory /path/to/releases/first \ --releases-directory /path/to/releases/second \ --stemcells-directory /path/to/stemcells/first \ --stemcells-directory /path/to/stemcells/second \ --output-file /path/to/cf-2.0.0-build.4.pivotalThe--runtime-configs-directory flag takes a path to a directory thatcontains one or more runtime config files. The flag can be specifiedmore than once.
To reference a runtime config in the directory you can use theruntime_configtemplate helper:
$ cat /path/to/metadata---runtime_configs:- $( runtime_config "first-runtime-config" )Exampleruntime-configs directory.
The--sha256 flag calculates the sha256 checksum of the output file
The--skip-fetch-directories skips the automatic release fetching ofthe specified release directories
Warning:--stemcell-tarball will be removed in a future version of kiln.Use--stemcells-directory in the future.
The--stemcell-tarball flag takes a path to a stemcell.
To include information about the stemcell in your metadata you can use thestemcell template helper:
$ cat /path/to/metadata---stemcell_criteria: $( stemcell )The--stemcells-directory flag takes a path to a directory containing oneor more stemcells.
To include information about the stemcell in your metadata you can use thestemcell template helper. It takes a single argument that specifies whichstemcell os.
Thestemcell helper does not support multiple versions of the same operatingsystem currently.
$ cat /path/to/metadata---stemcell_criteria: $( stemcell "ubuntu-xenial" )additional_stemcells_criteria:- $( stemcell "windows" )For tile developers looking to get some quick feedback about their tilemetadata, the--stub-releases flag will skip including the release tarballsinto the built tile output. This should result in a much smaller file thatshould upload much more quickly to OpsManager.
The--variable flag takes akey=value argument that allows you to specifyarbitrary variables for use in your metadata. The flag can be specifiedmore than once.
To reference a variable you can use thevariable template helper:
$ cat /path/to/metadata---$( variable "some-variable" )The--variables-file flag takes a path to a YAML file that contains arbitraryvariables for use in your metadata. The flag can be specified more than once.
To reference a variable you can use thevariable template helper:
$ cat /path/to/metadata---$( variable "some-variable" )Examplevariables file.
The--version flag takes the version number you want your tile to become.
To reference the version you use theversion template helper:
$ cat /path/to/metadata---product_version: $( version )provides_product_versions:- name: example version: $( version )It constructs a tile from a given bake record file.
To run the command, you simply need to be within a tile directory and execute the following command:
$ kiln re-bake --output-file tile.pivotal bake_records/1.0.0.jsonAny variables that Kilnfile needs for the kiln re-bake command should be set in~/.kiln/credentials.yml file
Thetest command exercises to ginkgo tests under the/<tile>/test/manifest and/<tile>/migrations paths of thepivotal/tas repos (where<tile> is tas, ist, or tasw).
Running these tests require a docker daemon and ssh-agent to be running.
If you run into this docker errorcould not execute "test": failed to connect to Docker daemon: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running,then create a symlinksudo ln -s ~/.docker/run/docker.sock /var/run/docker.sock
If no ssh identity is added (check withssh-add -l) , thenkiln testwill add a ssh key in the following order, prompting for a passphrase if required:
~/.ssh/id_rs~/.ssh/id_dsa~/.ssh/d_ecdsa~/.ssh/d_ed25519~/.ssh/dentityThe identity must have access to github.com/pivotal/ops-manager.
Here are command line examples:
$ cd ~/workspace/tas/ist$ kiln testcd ~$ kiln test --verbose -tp ~/workspace/tas/ist --ginkgo-manifest-flags "-p -nodes 8 -v"Additional test options
The--ginkgo-manifest-flags flag can be used to pass through Ginkgo test flags. The defaults being passed through are-r -p -slowSpecThreshold 15. Passhelp as a flag to retrieve the available options for the embeded version of ginkgo.
The--manifest-only flag can be used to run only Manifest tests. If not passed,kiln test will run both Manifest and Migration tests by default.
The--migrations-only flag can be used to run only Migration tests. If not passed,kiln test will run both Manifest and Migration tests by default.
The--tile-path (-tp) flag can be set the path the directory you wish to test. It defaults to the current working directory. For example
$ kiln test -tp ~/workspace/tas/istThe--verbose (-v) flag will log additional debugging info.
Thefetch command downloads bosh release tarballs specified in the Kilnfile andKilnfile.lock files to a local directory specified by the--releases-directory flag.
Kiln verifies that the checksum (SHA1) of the downloaded release matcheschecksum specified for the release in the Kilnfile.lock file. If the checksums donot match, then the releases that don't match will be deleted from disk.SinceBOSH releases from different directors with the same packages result in compliedreleases with different hashes this may result in some problems where if youdownload a release that was compiled with a different director those releaseswill be deleted.
Kiln will not download releases if an existing release exists with the correctrelease version and checksum.
A Kilnfile contains information about the bosh releases and stemcell used bya particular tile
Example Kilnfile:
---slug:some-slug#optional but if included should match network.pivotal.iorelease_sources:-type:bosh.ioreleases:-name:bpmversion:'*'stemcell_criteria:os:ubuntu-xenialversion:"~621"
release_sources:-type:bosh.io
release_sources: -type:s3id:unique-namepublishable:true# if this bucket contains releases that are suitable to ship to customersbucket:some-bucket-in-s3region:us-east-1# must be the region of the above bucketaccess_key_id:$(variable "s3_access_key_id")# Must have at least read permissions to bucketsecret_access_key:$(variable "s3_secret_access_key")path_template:bosh-releases/compiled/{{.Name}}-{{.Version}}-{{.StemcellOS}}-{{.StemcellVersion}}.tgz# See Templating
-type:githubid:optional-unique-name-defaults-to-github-org-nameorg:the-github-orgendpoint:$(variable "github_host")github_token:$(variable "github_token")
-type:artifactoryid:unique-nameartifactory_host:https://build-artifactory.your-artifactory-url.comrepo:some-artifactory-repopublishable:true# if this repo contains releases that are suitable to ship to customersusername:$(variable "artifactory_username")password:$(variable "artifactory_password")path_template:shared-releases/{{.Name}}-{{.Version}}-{{.StemcellOS}}-{{.StemcellVersion}}.tgz# See Templating
Kilnfile files support the following templating options:
{{.Name}}for release name{{.Version}}for release version{{.StemcellOS}}for stemcell OS{{.StemcellVersion}}for stemcell versionThere's also access to a
trimSuffixhelper (e.g.{{trimSuffix .Name "-release"}})
Theselect function allows you to pluck values for nested fields from atemplate helper.
For instance, this section in our example tile:
my_release_version:$( release "my-release" | select "version" )
Results in:
my_release_version:1.2.3
release_sources: -type:s3compiled:truebucket:compiled-releasesregion:us-west-1access_key_id:$(variable "aws_access_key_id")secret_access_key:$(variable "aws_secret_access_key")path_template:2.6/{{trimSuffix .Name "-release"}}/{{.Name}}-{{.Version}}-{{.StemcellOS}}-{{.StemcellVersion}}.tgz
Credentials like S3 keys are not stored in git repos. To support separatingthat information from non-sensitive configuration, you can reference variableslike you do in tile config.
---aws_access_key_id:SOME_REALLY_SECRET_IDaws_secret_access_key:SOME_REALLY_SECRET_KEY
Interpolating this file in kiln would look something like this.
- Get your credentials from Lastpass by running:
lpass show --notes 'pas-releng-fetch-releases' > vars.yml
kiln bake --kilnfile random-Kilnfile --variables-file vars.yml
The Kilnfile.lock file name is expected to be a file in the same directory as theKilnfile withlock as as the filename extension.
This file contains the full list of specific versions of all releases, shas, and sources forbosh releases that will go into the tile as well as the target stemcell.
The file has two top level membersreleases andstemcell_criteria.
Thereleases member is an array of members with each element having the following members.
name: bosh release namesha1: checksum of the tarballversion: semantic version of the releaseremote_source: the resource-type for bosh.io or the id for the other typesremote_path: the path that where the bosh release is stored
Thestemcell_criteria member is defines the stemcell used to generate the tile
os: the stemcell os used (e.g. ubuntu-xenial)version: semantic version of the stemcell
Example Kilnfile.lock :
releases:-name:bpmsha1:86675f90d66f7018c57f4ae0312f1b3834dd58c9version:1.1.18remote_source:bosh.ioremote_path:https://bosh.io/d/github.com/cloudfoundry/bpm-release?v=1.1.18-name:backup-and-restore-sdksha1:0f48faa2f85297043e5201e2200567c2fe5a9f9aversion:1.18.84remote_source:unique-name# this could be artifactory or s3remote_path:bosh-releases/compiled/backup-and-restore-sdk-1.18.84-ubuntu-jammy-1.179.tgz-name:hello-releasesha1:d7de88ab98d7d61d0a4e660c8fff76727817c059version:0.4.0remote_source:the-github-orgremote_path:https://github.com/releen/hello-release/releases/download/0.4.0/hello-release-0.4.0.tgzstemcell_criteria:os:ubuntu-xenialversion:"621.0"
About
Kiln helps you maintain product tiles for VMware Tanzu Operations Manager.