Cloud VPS administrators have to routinely perform some tasks like adding or removing users to a project, adding or removing a node to a cluster, etc. We have created a set ofcookbooks to automate those tasks, and make sure they are performed in a way that is consistent, repeatable and traceable. Our cookbooks are implemented using the WikimediaSpicerack library and are maintained in thewmcs-cookbook repository.

In WMCS, we use the wordcookbook to describe the automation scripts described in this wiki page, implemented using the Spicerack library. We use instead the wordRunbooks to describe manual procedures documented in Wikitech. A runbook might require you to run a cookbook to perform one of the runbook steps, or it might not involve any cookbook at all.

You can run a cookbook from a cloudcumin host, or from your laptop.

Running from a cloudcumin host is recommended because:

• It ensures you are using the tip of the "main" branch
• Logs will be stored in the cloudcumin host, where other admins can see them if they need to
• If the cookbook takes a long time to complete, you can usescreen ortmux to let the cookbook continue to run even if you lose your internet connection, or if you turn your computer off.

Some reasons why you might want to run a cookbook from your laptop instead:

• The cookbook does not work correctly when run from a cloudcumin host. If this happens, please open a bug in Phabricator as ideally all cookbooks should work fine from a cloudcumin host.
• You don't have "sudo" privileges in cloudcumin hosts. This is something we would like to solve in the future, and is tracked inphab:T343330.
• You are developing a new cookbook and need a quicker feedback than pushing to the repo and fetching.
• You are developing a patch for spicerack or any dependent library and need to test it/adapt cookbooks with it.

We have two cloudcumin hosts,cloudcumin1001.eqiad.wmnet andcloudcumin2001.codfw.wmnet.

They are almost identical, but1001 is configured to target theeqiad1 OpenStack cluster, and2001 is configured to target thecodfw1dev OpenStack cluster (see/etc/cumin/config.yaml).

Both hosts are Ganeti virtual machines, managed by theInfra Foundation team.

At the moment, some cookbooks are failing when run from cloudcumins. These issues are tracked as subtasks ofphab:T343330.

fnegri@cloudcumin1001:~$sudocookbook-l

fnegri@cloudcumin1001:~$sudocookbookwmcs.cookbook.name-h

fnegri@cloudcumin1001:~$sudocookbookwmcs.cookbook.name--projectPROJECT_NAME--task-idPHAB_TASK_ID

If you are running a cookbook that you expect will take a long time to complete, you should run it inside a Screen or tmux session, so you can detach from the session and let the cookbook continue to run.

Screen example:

fnegri@cloudcumin1001:~$screenfnegri@cloudcumin1001:~$sudocookbook...  #Press"Ctrl+a d"todetachwhilethecookbookisrunningfnegri@cloudcumin1001:~$screen-x# To reattachfnegri@cloudcumin1001:~$ exit #Toterminatethesession

Tmux example:

fnegri@cloudcumin1001:~$tmuxfnegri@cloudcumin1001:~$sudocookbook...  #Press"Ctrl+b d"todetachwhilethecookbookisrunningfnegri@cloudcumin1001:~$tmuxattach# To reattachfnegri@cloudcumin1001:~$ exit #Toterminatethesession

To run locally the cookbooks, you can follow the docs herehttps://gerrit.wikimedia.org/g/cloud/wmcs-cookbooks

All WMCS cookbooks have some common options:

• --task-id to indicate a related Phabricator task. This ID will be included inSAL messages, and SAL messages will be displayed in Phab comments.
• --project to indicate the related Cloud VPS project. This will make sure thatSAL messages are displayed under the correct project insal.toolforge.org. If the option is not included, it defaults toadmin.
• --no-dologmsg to disable sending messages toSAL. Note: this only affects messages sent with the SALLogger class and not the ones sent with sal_logger. For more information, readphab:T343528.

Some additional common options are provided by Spicerack itself and are documented atSpicerack/Cookbooks#Cookbook_Operations.

The cloudcumin hosts are also capable of runningCumin commands. Example:

user@cloudcumin1001:~ $sudocumin'O{project:toolsbeta name:toolsbeta-test-k8s-*}''apt-get update'

WMCS admins with Global Root permissions can also run the SRE cookbooks (the ones with a name starting withsre.). SRE cookbooks (from theoperations/cookbooks repo) are not installed in cloudcumin hosts, but only in production cumin hosts (cuminXXXX).

TODO: Create "shared" cookbooks that are installed in both production and cloud cumin hosts. (phab:T343894)

These are some of the enhancements we would like to make to WMCS Cookbooks. We don't have a timeline for those but please leave a comment in the Phab tasks if a feature is particularly useful/relevant to you.

• phab:T343330 Provide shared hosts for people without root privileges
• phab:T343528 Use a single bot for SAL logging

• Wikimedia Cloud Services team/EnhancementProposals/Operational Automation