./requirements.sh mac

Command line tool

To install a cluster, default one is a CDP 7 - 10 nodes with Kerberos and TLS set:

N.B. : This assumes that a passwordless connection is present from here to all your cluster nodes, however provide a password with--node-password or a private key file with--node-key

Configuration

Many more configurations are available, see them all with:

NEW: Script to create Instances and launch creation

It is only working with AWS as of now.

Note: This requires you to install Terraform and AWS CLI on the machine where you will launch this script.

It will require also that you create in advance a key pair in AWS, get your private key locally, and also get AWS access key and secret access key:

All parameters above must be let like this, as they are appropriate to AWS machines.After these parameters, you can add all other parameters that worked with script: setup-cluster.sh.

The script, will use terraform to provide your machines, setup connectivity and then launch setup-cluster.sh with pre-configured parameters to create the wanted cluster.

Examples

CM & Ambari

At the end, CM or Ambari depending on your installation should be available at the first node URL with appropriate http or https and port(depending on tls parameters for HDP which is false by default and tls for CDP which is true by default).

During the installation, you can also follow the installation from CM or Ambari by connecting to it.

N.B.: It is recommended to not interfer with the cluster during ansible installation until it is done

Users and Data

At the end of the installation, if it completed successfully, users are created on machines, their keytabs too and are retrieved in your local computer under/tmp/,krb5.conf is also retrieved.

Moreover, it is also possible to launch some random data generation into various systems.

All default passwords are Cloudera1234

Architecture

Once you gathered all previous requirements, a launch could be made, it will mainly consist of 5 steps:

• Prepare your machines

• Launch the installation from the first node of your cluster using appropriate ansible playbook and files

• Do post-install configuration (mainly for CDP)

• Create users on your cluster

• Load some data into your cluster

Each step could be skipped (see command line help).

Scripts

This group of scripts, coordinated by main script:setup-cluster.sh has the goal to configure machines provided and launch a CDP (or HDP, CDH) installation with ansible.Finally, some extra configurations steps and random data could be generated into different services.

All this, is only made from your machine.

This script relies on ansible scripts that must be accessible from your machine (if they are not, please setup an internal webserver and provide its url through command line).

Ansible CDP/CDH repository

Ansible HDP repository

Ansible script relies also on Cloudera repository to access CDP, CM, HDP, Ambari etc…​ (if they are not accessible, please setup an internal webserver and provide its url through command line).

Cloudera repository

This script relies also on github repository to load data. (if they are not accessible, please setup an internal webserver and provide its url through command line).

Data Load repository

Setup Machines

This step usesPlaybook hosts_setup.

If you did not set parameter--setup to false, it will prepare all machines by setting ssh-passwordless, pushing required files to them.

N.B.: This step can be done only one time and then bypass if you reuse same machines

Ansible Installation

This step usesPlaybook ansible_install_preparation and then launch commands directly on the host to launch ansible installation there.

The first playbook used can be skipped setting parameter--install to false, which is true by default.

It cleans up the first node, creates a directory~/deployment/ansible-repo/, get ansible repository as a zip in it and add files for your installation in it.

Then, the proper ansible command corresponding to the installation is lauched directly on the first node.

Post Installation

This step usesPlaybook post_install.

If you install a CDP cluster and let parameter--post-install to true, it will do some extra-steps, such as setting no unlogin on CM, fix various potential bugs.

User Creation

This step usesPlaybook user_creation.

If you did not set explicitly parameter--user-creation to false, and installation completed succesfully, some users are created defined inextra_vars of user_creation.

They are present on all nodes with their/home directory containing their keytabs.

Their keytabs are also fetch in your/tmp directory along with thekrb5.conf allowing you to kinit directly from your computer.

Data Loading

This step usesPlaybook data_load.

If you let parameter--data-load to true, a data loading step will start (only on CDP, HDP 2 and CDH 5 currently) to generate data into existing services of the paltform: HDFS, HBase, Hive etc…​

It is based onrandom-datagen project

Note that this step is completely extensible as you can add new files to specify how data should be generated in folderplaybooks/data_load/generate_data/models

N.B.: This step will also create Ranger required policies, and these are also extensible by adding policies inplaybooks/data_load/ranger_policies/push_policies/policies

Cluster Definition

To provide a quick new definition of a cluster:

1. Copy-Paste directory ansible-cdp and name it for example: ansible-cdp-configured

2. Make all your modifications in files of your copied directory

3. Launch script with argument:--cluster-type=ansible-cdp-configured (It will automatically take files under ansible-cdp-configured/ directory)

User Creation & Data Loading

Those steps can be launched indepently and you can configure it to create more users or load different and more data.

Look inside playbooks folder toextra_vars.yml to get more about possibilities.

Private Cloud

Private Cloud setup (on ECS or OC) can also be launched independently on a running cluster.

Configuration of private cloud cluster can also be launched independently. (Use--install-pvc=false but--pvc=true to configure but not re-install your pvc).

Inextra_vars.yml you can provide CDWs, CDEs, CMLs that will be provisionned for you and also rights that you expect on your users.