Repository files navigation

This Terraform example configurations uses theVMware vSphere provider to provision virtual machines on VMwareandTerraForm Module ICP Deploy to prepare VMs and deployIBM Cloud Private on them. This Terraform template automates best practices learned from installing ICP on VMware at numerous client sites in production.

This template provisions anHA cluster with ICP 3.1 enterprise edition. There is a smaller and simpler non-HA cluster with ICP 3.1 community edition terraform template available in this repository in thece-template folder.

• Working copy ofTerraform
• The example assumes the VMs are provisioned from a template that has ssh public keys loaded in${HOME}/.ssh/authorized_keys. After VM creation, terraform will SSH into the VM to prepare and start installation of ICP using the SSH private key provided. If your VM template uses a different user from root, update thessh_user section invariables.tf
• The template is tested on VM templates based onUbuntu 16.04

1. Create a VM image (RHEL or Ubuntu 16.04).

   • The automation will create an additional block device at unit number 1 (i.e./dev/sdb) for local docker container images and attempt to configure Docker in direct-lvm or overlay2 mode, depending on your operating system. RHEL 7.4 and older will default to devicemapper, RHEL 7.5 and newer will default to Overlay2. All supported Ubuntu versions will default to Overlay2. You may pre-install docker, but it must be configured in direct-lvm or overlay2 mode. For direct-lvm the block device must be set as the second disk. The simplest way of getting this to work is to create a template with a single OS disk without installing docker and let the automation configure direct-lvm or overlay2 mode as appropriate.

   • Additional block devices are also mounted at various directories that hold ICP data depending on the node role. For instance, an additional block device at unit number 4 (i.e./dev/sdd) with the possibility to use a different datastore than the other disks is mounted at/var/lib/etcd. The reason for this is to allow the allocation of a different tier datastore with potentially better performance for I/O disk intensive ICP components such as ETCD.

2. Ensure that thessh_user can callsudo without password.

3. Install any other needed packages:

   1. Shared storage clients, e.g. if the shared storage is NFS for the registry and audit directories, install NFS clients.
   2. In environments with restricted outbound internet, the terraform automation requires and attempts to installPyYAML. You can pre-install it into the template:
      • yum install PyYAML on RHEL
      • apt install python-yaml on Ubuntu
   3. Install VMware Tools oropen-vm-tools
   4. For docker installation, install any additional pre-requisites:
      • yum install device-mapper libseccomp libtool-ltdl libcgroup iptables on RHEL
      • apt install libtdl7 thin-provisioning-tools lvm2 on Ubuntu
   5. For RHEL templates, VMware guest customization requiresperl.
   6. For RHEL templates, either disable the firewall (systemctl disable firewalld), or addfirewall_enabled: true toicp-config.yaml.
   7. For RHEL templates, VMware guest you most also installyum install open-vm-tools

4. (optional) If you are not providingimage_location, and have pre-installed docker, you can also pre-load the docker images from the ICP package you wish to install.

   tar xf ibm-cloud-private-x86_64-3.1.tar.gz -O| sudo docker load

5. Shutdown the VM Convert to a template, make note the name of the template.

The automation requires an HTTP or NFS server to hold the ICP binaries and docker packages. NFS storage is recommended as it can host these binaries and also the shared storage location forregistry_mount_src,audit_mount_src,docker_package_location andimage_location.

1. git clone or download the templates

2. Export your vSphere username and password into environment variables on the system you will run Terraform from. This can be done via the following commands:

   • export VSPHERE_USER={myusername} replacing{myusername} with your vSphere username
   • export VSPHERE_PASSWORD={mypassword} replacing{mypassword} with your vSphere password

3. Create aterraform.tfvars file to reflect your environment. Please seevariables.tf and below tables for variable names and descriptions. Here is an exampleterraform.tfvars file:

   ############################################ vSphere Access Credentials #############################################vsphere_server = "10.25.0.20"# Set username/password as environment variables VSPHERE_USER and VSPHERE_PASSWORD################################################### vSphere deployment specifications ##################################################### Following resources must exist in vSpherevsphere_datacenter = "DC1"vsphere_cluster = "Cluster1"vsphere_resource_pool = "ICP31_pool/terraform_icp_31"network_label = "LabPrivate"datastore = "LabDatastore"datastore_etcd = "Tier0_LabDatastore"template = "ubuntu_1604_base_template"# Folder to provision the new VMs in, does not need to exist in vSpherefolder = "terraform_icp_31"####################################### ICP deployment details ############################################ ICP instance name ###### MUST consist of only lower case alphanumeric characters and '-'instance_name = "user1-icp-31"##### Network #####staticipblock = "10.30.0.0/24"staticipblock_offset = 2gateway = "10.0.0.1"netmask = "16"dns_servers = [ "10.0.0.11", "10.0.0.12" ]# Cluster accesscluster_vip = "10.30.0.1"cluster_vip_iface = "ens160"proxy_vip = "10.0.0.2"proxy_vip_iface = "ens160"##### Local Terraform connectivity details #####ssh_user = "virtuser"ssh_password = "SuperPa88w0rd"##### ICP installation method #####icp_inception_image = "ibmcom/icp-inception:3.1.0-ee"private_registry    = "registry.example.com"registry_username   = "myUsername"registry_password   = "myPassword"##### ICP admin user password ###### Non default admin user password 'admin' recommendedicppassword = "SuperPa88w0rd"##### ICP Cluster Components #####master = {    nodes = "3"    vcpu = "8"    memory = "16384"    docker_disk_size = "250"    thin_provisioned = "true"    thin_provisioned_etcd = "false"}proxy = {    nodes = "3"    vcpu = "4"    memory = "8192"    thin_provisioned = "true"}worker = {    nodes = "3"    vcpu = "8"    memory = "16384"    thin_provisioned = "true"}management = {    nodes = "3"    vcpu = "8"    memory = "16384"    thin_provisioned = "true"}va = {    nodes = "2"    vcpu = "8"    memory = "16384"    thin_provisioned = "true"}##### NFS Server #####registry_mount_src = "10.0.0.5:/storage/user1-icp-31/registry"audit_mount_src = "10.0.0.5:/storage/user1-icp-31/audit"

4. Runterraform init to download dependencies (modules and plugins)

5. Runterraform plan to investigate deployment plan

6. Runterraform apply to start deployment

Below you can find the different paths to install ICP on VMware.

1. Install from ICP binary package. In order to install ICP from it's binary package, we need to specify theimage_location of the binary in theterraform.tfvars file:

   ##### ICP installation method #####icp_inception_image = "ibmcom/icp-inception:3.1.0-ee"image_location = "nfs:<nfs_server_ip_address>:<path_within_your_nfs_server>/ibm-cloud-private-x86_64-3.1.0.tar.gz"

2. Install from a private Docker registry which does not require authentication. In order to install ICP from a previously configured (with ICP images loaded into) private Docker registry which does not require authentication, we need to specify theprivate_registry in theterraform.tfvars file:

   ##### ICP installation method #####icp_inception_image = "ibmcom/icp-inception:3.1.0-ee"private_registry    = "registry.example.com"

3. Install from a private Docker registry which requires authentication. In order to install ICP from a previously configured (with ICP images loaded into) private Docker registry which requires authentication, we need to specify theprivate_registry and its credentials,registry_username andregistry_password, in theterraform.tfvars file:

   ##### ICP installation method #####icp_inception_image = "ibmcom/icp-inception:3.1.0-ee"private_registry    = "registry.example.com"registry_username   = "myUsername"registry_password   = "myPassword"

Although the template defaults configure the cluster using DHCP addresses, it is strongly recommended to configure values for each of these variables so that the cluster is configured using static IP addresses.

The ICP configuration can further be customized by editing theicp-config.yaml file and by following theproduct documentation. The Terraform templates here expose the key variables that have infrastructure dependencies.