Cookbooks
automation made delicious
gerrit	operations/cookbooks
installed on	cumin1003.eqiad.wmnet, cumin2002.codfw.wmnet

To list or run a Cookbook you must be logged into one of the cumin hosts (cumin1003.eqiad.wmnet, cumin2002.codfw.wmnet).

To show a tree of the available cookbooks run:

 $ cookbook -l

To include a brief one-line description of each cookbook in the tree use:

 $ cookbook -lv

The generic way of running a cookbook is:

 $ sudo cookbook [-d] [-v] COOKBOOK [COOKBOOK_ARGS]

The full dotted name as shown in the tree of the available cookbooks (e.g.sre.hosts.downtime) must be used.To see the global options of the cookbook executable:

 $ cookbook -h

To see the specific options of a given cookbook:

 $ sudo cookbook sre.hosts.downtime -h

To run the cookbook in dry-run mode (it will not!log and will not execute any RW operation in the infrastructure):

 $ sudo cookbook -d sre.hosts.downtime [COOKBOOK_ARGS]

And finally, to actually run the cookbook:

 $ sudo cookbook sre.hosts.downtime [COOKBOOK_ARGS]

How the cookbook store their logs is described in theSpicerack documentation.

See theCookbooks API section of the Spicerack documentation andSpicerack#Explore Spicerack.

Make sure to set the ownership field of the cookbook if not set at the parent package level. The accepted values are the SRE teams from Puppet'sWmflib::Team define. They are currently listed also in the cookbooks repository tests so that CI can ensure only valid values are used.

Gerrit patches can be tested on the cumin hosts (cumin1003.eqiad.wmnet, cumin2002.codfw.wmnet) before actually merging them using thetest-cookbook binary. It allows to checkout the patch from Gerrit and run spicerack using that checkout of the cookbooks. It manages automatically the checkout and allows to clean it up too. Here's the current (June 2023) help message for the script:

$test-cookbook-husage: test-cookbook [-c CHANGE] [--ps PS] [--delete] [--no-sal-logging] [-h]Cookbook testing scriptIt allows testing a Gerrit patch on the cookbook repository checking out the change locallyon the cumin host where the command is run and running the cookbook with the change code insteadof the officially deployed one. It also allows to specify a given patch set (PS) to use.The cookbook binary will be invoked with the -c/--config argument already set to point to thecustom configuration pointing to the testing checkout.After a first checkout it is possible to modify the files in place and test it again. The scriptwill automatically detect that there are local changes and ask the operator what to do, as longas the local modifications are on the same branch (hence the same PS) requested.It should be used with caution but at least provides a standard way to test cookbook changes.All the extra parameters are passed to the cookbook binary.Example usage:    #UsethelatestPSofGerritchange12345tomakeaDRY-RUNofthedowntimecookbook    test-cookbook -c 12345 --dry-run sre.hosts.downtime -h    #UseaspecificPSofGerritchange12345tomakeaREALrunofthedowntimecookbook    test-cookbook -c 12345 --ps 3 sre.hosts.downtime -h    #Cleanupapreviouslytestedchange    test-cookbook --delete -c 12345The generated files structure is as follows:    ~/cookbooks_testing/  # Parent directory of all modified files    ~/cookbooks_testing/config.yaml  # The configuration file that is passed to the cookbook binary    ~/cookbooks_testing/cookbooks  # Symlink that points to the currently tested change    ~/cookbooks_testing/cookbooks-$CHANGE_ID  # The git checkout of a given the change ID    ~/cookbook_testing/logs  # The log directory where all cookbooks will log intooptional arguments:  -c CHANGE, --change CHANGE                        The Gerrit change ID to fetch.  --ps PS               The optional patch set to fetch. If not specified the last one will be fetched.  --delete              Delete any existing environment for the given change.  --no-sal-logging      Do not log to SAL. Use with CAUTION!  -h, --help            show this help message and exit

Once a patch is +2 on Gerrit, CI will take care of the actual merge and at the next Puppet run the cookbook will be updated automatically on the cluster management hosts (cumin1003.eqiad.wmnet, cumin2002.codfw.wmnet). Forcing a Puppet run after the merge will update the cookbooks immediately.

Due to the current way of deploying cookbooks, if a cookbook gets renamed or deleted, its compiled files (.pyc), if present, are not automatically deleted and need a manual deletion.

Find the existing files with this command, assuming the cookbook being renamed/deleted was calledsre.somedir.example-action, and then proceed to delete them, for example by appending-delete to the find command.

sudocumin'A:cumin''find /srv/deployment/spicerack/cookbooks/sre/somedir/ -name "example-action*.pyc"'

• tox

• Clone the required repositories from Gerrit under~/wmf/gerrit for simplicity

$ git clone ssh://$GERRIT_USER@gerrit.wikimedia.org:29418/operations/software/spicerack$ git clone ssh://$GERRIT_USER@gerrit.wikimedia.org:29418/operations/cookbooks

• Change to thespicerack directory and runtox. It will create the proper virtualenv for us with all the dependencies required:

$ tox -e py3N-unit  # where N is picked from any of environment listed with `tox -av | grep unit` for which the related Python version is available on the system

• Note: We could just run 'tox' at this point, it will be just a bit slower because it will run all the tests with all the compatible python versions that are installed in the system and configured in the tox.ini file to run the tests
• There should be a.tox/py3N-tests/ directory with the virtualenv created, we can source the activate script

$ source .tox/py3N-tests/bin/activate

• Create a configuration file for spicerack calledconfig.yaml:

cookbooks_base_dirs:  - ~/wmf/gerrit/cookbookslogs_base_dir: ~/wmf/gerrit/spicerack/logs

• The cookbook entry point script should be available, and you should be able to see the list of available cookbooks

$ cookbook -c config.yaml -l

• Clone the additional repository for WMCS-specific cookbooks:

$ git clone ssh://$GERRIT_USER@gerrit.wikimedia.org:29418/cloud/wmcs-cookbooks

• Add the new checked out repository to the list of available cookbooks in theconfig.yaml file:

cookbooks_base_dirs:  - ~/wmf/gerrit/cookbooks  - ~/wmf/gerrit/wmcs-cookbookslogs_base_dir: ~/wmf/gerrit/spicerack/logs

• Runbook

• Deployment
• SRE Infrastructure Foundations