Custom network topologies¶

Sahara accesses instances at several stages of cluster spawning throughSSH and HTTP. Floating IPs and network namespaces(seeNetworking configuration) will be automatically used foraccess when present. When floating IPs are not assigned to instances andnamespaces are not being used, sahara will need an alternative method toreach them.

Theproxy_command parameter of the configuration file can be used togive sahara a command to access instances. This command is run on thesahara host and must open a netcat socket to the instance destinationport. The{host} and{port} keywords should be used to describe thedestination, they will be substituted at runtime. Other keywords thatcan be used are:{tenant_id},{network_id} and{router_id}.

For example, the following parameter in the sahara configuration filewould be used if instances are accessed through a relay machine:

[DEFAULT]proxy_command='ssh relay-machine-{tenant_id} nc {host} {port}'

Whereas the following shows an example of accessing instances thougha custom network namespace:

[DEFAULT]proxy_command='ip netns exec ns_for_{network_id} nc {host} {port}'

DNS Hostname Resolution¶

Sahara can resolve hostnames of cluster instances by using DNS. For this Saharauses Designate. With this feature, for each instance of the cluster Sahara willcreate twoA records (for internal and external ips) under one hostnameand onePTR record. Also all links in the Sahara dashboard will bedisplayed as hostnames instead of just ip addresses.

You should configure DNS server with Designate. Designate service should beproperly installed and registered in Keystone catalog. The detailedinstructions about Designate configuration can be found here:Designate manualinstallation and here:Configuring OpenStack Networking with Designate.Also if you use devstack you can just enable Designate plugin:Designate devstack.

When Designate is configured you should create domain(s) for hostnameresolution. This can be done by using the Designate dashboard or by CLI. Alsoyou have to createin-addr.arpa. domain for reverse hostname resolutionbecause some plugins (e.g.HDP) determine hostname by ip.

Sahara also should be properly configured. Insahara.conf you must specifytwo config properties:

[DEFAULT]# Use Designate for internal and external hostnames resolution:use_designate=true# IP addresses of Designate nameservers:nameservers=1.1.1.1,2.2.2.2

An OpenStack operator should properly configure the network. It must enableDHCP and specify DNS server ip addresses (e.g. 1.1.1.1 and 2.2.2.2) inDNSNameServers field in theSubnetDetails. If the subnet alreadyexists and changing it or creating new one is impossible then Sahara willmanually change/etc/resolv.conf file on every instance of the cluster (ifnameservers list have been specified insahara.conf). In this case,though, Sahara cannot guarantee that these changes will not be overwritten byDHCP or other services of the existing network. Sahara has a health check fortrack this situation (and if it occurs the health status will be red).

In order to resolve hostnames from your local machine you should properlychange your/etc/resolv.conf file by adding appropriate ip addresses ofDNS servers (e.g. 1.1.1.1 and 2.2.2.2). Also the VMs with DNS servers shouldbe available from your local machine.

Data-locality configuration¶

Hadoop provides the data-locality feature to enable task tracker anddata nodes the capability of spawning on the same rack, Compute node,or virtual machine. Sahara exposes this functionality to the userthrough a few configuration parameters and user defined topology files.

To enable data-locality, set theenable_data_locality parameter totrue in the sahara configuration file

[DEFAULT]enable_data_locality=true

With data locality enabled, you must now specify the topology filesfor the Compute and Object Storage services. These files arespecified in the sahara configuration file as follows:

[DEFAULT]compute_topology_file=/etc/sahara/compute.topologyswift_topology_file=/etc/sahara/swift.topology

Thecompute_topology_file should contain mappings between Computenodes and racks in the following format:

compute1 /rack1compute2 /rack2compute3 /rack2

Note that the Compute node names must be exactly the same as configured inOpenStack (host column in admin list for instances).

Theswift_topology_file should contain mappings between Object Storagenodes and racks in the following format:

node1 /rack1node2 /rack2node3 /rack2

Note that the Object Storage node names must be exactly the same asconfigured in the object ring. Also, you should ensure that instanceswith the task tracker process have direct access to the Object Storagenodes.

Hadoop versions after 1.2.0 support four-layer topology (for more detailplease seeHADOOP-8468 JIRA issue). To enable this feature set theenable_hypervisor_awareness parameter totrue in the configurationfile. In this case sahara will add the Compute node ID as a second level oftopology for virtual machines.

Distributed mode configuration¶

Sahara can be configured to run in a distributed mode that creates aseparation between the API and engine processes. This allows the APIprocess to remain relatively free to handle requests while offloadingintensive tasks to the engine processes.

Thesahara-api application works as a front-end and serves userrequests. It offloads ‘heavy’ tasks to thesahara-engine processvia RPC mechanisms. While thesahara-engine process could be loadedwith tasks,sahara-api stays free and hence may quickly respond touser queries.

If sahara runs on several hosts, the API requests could bebalanced between severalsahara-api hosts using a load balancer.It is not required to balance load between differentsahara-enginehosts as this will be automatically done via the message broker.

If a single host becomes unavailable, other hosts will continueserving user requests. Hence, a better scalability is achieved and somefault tolerance as well. Note that distributed mode is not a truehigh availability. While the failure of a single host does notaffect the work of the others, all of the operations running onthe failed host will stop. For example, if a cluster scaling isinterrupted, the cluster will be stuck in a half-scaled state. Thecluster might continue working, but it will be impossible to scale itfurther or run jobs on it via EDP.

To run sahara in distributed mode pick several hosts on whichyou want to run sahara services and follow these steps:

• On each host install and configure sahara using theinstallation guideexcept:

  • Do not runsahara-db-manage or launch sahara withsahara-all
  • Ensure that each configuration file provides a database connectionstring to a single database for all hosts.

• Runsahara-db-manage as described in the installation guide,but only on a single (arbitrarily picked) host.

• Thesahara-api andsahara-engine processes use oslo.messaging tocommunicate with each other. You will need to configure it properly oneach host (see below).

• Runsahara-api andsahara-engine on the desired hosts. You mayrun both processes on the same or separate hosts as long as they areconfigured to use the same message broker and database.

To configure oslo.messaging, first you will need to choose a messagebroker driver. Currently there are two drivers provided: RabbitMQor ZeroMQ. For the RabbitMQ drivers please see theNotifications configuration documentation for an explanation ofcommon configuration options.

For an expanded view of all the options provided by each message brokerdriver in oslo.messaging please refer to the options available in therespective source trees:

• For Rabbit MQ see
  • rabbit_opts variable inimpl_rabbit.py
  • amqp_opts variable inamqp.py
• For Zmq see
  • zmq_opts variable inimpl_zmq.py
  • matchmaker_opts variable inmatchmaker.py
  • matchmaker_redis_opts variable inmatchmaker_redis.py
  • matchmaker_opts variable inmatchmaker_ring.py

These options will also be present in the generated sample configurationfile. For instructions on creating the configuration file please see theSahara Configuration Guide.

Distributed periodic tasks configuration¶

If sahara is configured to run in distributed mode (seeDistributed mode configuration), periodic tasks can also be launched indistributed mode. In this case tasks will be split across allsahara-engineprocesses. This will reduce overall load.

Distributed periodic tasks are based on Hash Ring implementation and the Toozlibrary that provides group membership support for a set of backends. In orderto use periodic tasks distribution, the following steps are required:

• One of thesupported backends should be configured and started.

• Backend URL should be set in the sahara configuration file with theperiodic_coordinator_backend_url parameter. For example, if theZooKeeper backend is being used:

  [DEFAULT]periodic_coordinator_backend_url=kazoo://IP:PORT

• Tooz extras should be installed. When using Zookeeper as coordinationbackend,kazoo library should be installed. It can be done with pip:

  pip install tooz[zookeeper]

• Periodic tasks can be performed in parallel. Number of threads to runperiodic tasks on a single engine can be set withperiodic_workers_number parameter (only 1 thread will be launched bydefault). Example:

  [DEFAULT]periodic_workers_number=2

• coordinator_heartbeat_interval can be set to change the interval betweenheartbeat execution (1 second by default). Heartbeats are needed to makesure that connection to the coordination backend is active. Example:

  [DEFAULT]coordinator_heartbeat_interval=2

• hash_ring_replicas_count can be set to change the number of replicas foreach engine on a Hash Ring. Each replica is a point on a Hash Ring thatbelongs to a particular engine. A larger number of replicas leads to bettertask distribution across the set of engines. (40 by default). Example:

  [DEFAULT]hash_ring_replicas_count=100

External key manager usage¶

Sahara generates and stores several passwords during the course of operation.To harden sahara’s usage of passwords it can be instructed to use anexternal key manager for storage and retrieval of these secrets. To enablethis feature there must first be an OpenStack Key Manager service deployedwithin the stack.

With a Key Manager service deployed on the stack, sahara must be configuredto enable the external storage of secrets. Sahara uses thecastellan libraryto interface with the OpenStack Key Manager service. This library providesconfigurable access to a key manager. To configure sahara to use barbican asthe key manager, edit the sahara configuration file as follows:

[DEFAULT]use_barbican_key_manager=true

Enabling theuse_barbican_key_manager option will configure castellanto use barbican as its key management implementation. By default it willattempt to find barbican in the Identity service’s service catalog.

For added control of the barbican server location, optional configurationvalues may be added to specify the URL for the barbican API server.

[castellan]barbican_api_endpoint=http://{barbican controller IP:PORT}/barbican_api_version=v1

The specific values for the barbican endpoint will be dictated by theIP address of the controller for your installation.

With all of these values configured and the Key Manager service deployed,sahara will begin storing its secrets in the external manager.

Indirect instance access through proxy nodes¶

Sahara needs to access instances through SSH during cluster setup. Thisaccess can be obtained a number of different ways (seeNetworking configuration,Floating IP management,Custom network topologies). Sometimes it is impossible to provideaccess to all nodes (because of limited numbers of floating IPs or securitypolicies). In these cases access can be gained using other nodes of thecluster as proxy gateways. To enable this setis_proxy_gateway=truefor the node group you want to use as proxy. Sahara will communicate withall other cluster instances through the instances of this node group.

Note, ifuse_floating_ips=true and the cluster contains a node group withis_proxy_gateway=true, the requirement to havefloating_ip_poolspecified is applied only to the proxy node group. Other instances will beaccessed through proxy instances using the standard private network.

Note, the Cloudera Hadoop plugin doesn’t support access to Cloudera managerthrough a proxy node. This means that for CDH clusters only nodes withthe Cloudera manager can be designated as proxy gateway nodes.

Multi region deployment¶

Sahara supports multi region deployment. To enable this option eachinstance of sahara should have theos_region_name=<region>parameter set in the configuration file. The following example demonstratesconfiguring sahara to use theRegionOne region:

[DEFAULT]os_region_name=RegionOne

Non-root users¶

In cases where a proxy command is being used to access cluster instances(for example, when using namespaces or when specifying a custom proxycommand), rootwrap functionality is provided to allow users other thanroot access to the needed operating system facilities. To use rootwrapthe following configuration parameter is required to be set:

[DEFAULT]use_rootwrap=true

Assuming you elect to leverage the default rootwrap command(sahara-rootwrap), you will need to perform the following additional setupsteps:

• Copy the provided sudoers configuration file from the local project fileetc/sudoers.d/sahara-rootwrap to the system specific location, usually/etc/sudoers.d. This file is setup to allow a user namedsaharaaccess to the rootwrap script. It contains the following:

sahara ALL=(root) NOPASSWD: /usr/bin/sahara-rootwrap /etc/sahara/rootwrap.conf *

When using devstack to deploy sahara, please pay attention that you need tochange user in script fromsahara tostack.

• Copy the provided rootwrap configuration file from the local project fileetc/sahara/rootwrap.conf to the system specific location, usually/etc/sahara. This file contains the default configuration for rootwrap.
• Copy the provided rootwrap filters file from the local project fileetc/sahara/rootwrap.d/sahara.filters to the location specified in therootwrap configuration file, usually/etc/sahara/rootwrap.d. This filecontains the filters that will allow thesahara user to access theipnetnsexec,nc, andkill commands through the rootwrap(depending onproxy_command you may need to set additional filters).It should look similar to the followings:

[Filters]ip: IpNetnsExecFilter, ip, rootnc: CommandFilter, nc, rootkill: CommandFilter, kill, root

If you wish to use a rootwrap command other thansahara-rootwrap you canset the following parameter in your sahara configuration file:

[DEFAULT]rootwrap_command='sudo sahara-rootwrap /etc/sahara/rootwrap.conf'

For more information on rootwrap please refer to theofficial Rootwrap documentation

Object Storage access using proxy users¶

To improve security for clusters accessing files in Object Storage,sahara can be configured to use proxy users and delegated trusts foraccess. This behavior has been implemented to reduce the need forstoring and distributing user credentials.

The use of proxy users involves creating an Identity domain that will bedesignated as the home for these users. Proxy users will becreated on demand by sahara and will only exist during a job executionwhich requires Object Storage access. The domain created for theproxy users must be backed by a driver that allows sahara’s admin user tocreate new user accounts. This new domain should contain no roles, to limitthe potential access of a proxy user.

Once the domain has been created, sahara must be configured to use it byadding the domain name and any potential delegated roles that must be usedfor Object Storage access to the sahara configuration file. With thedomain enabled in sahara, users will no longer be required to entercredentials for their data sources and job binaries referenced inObject Storage.

[DEFAULT]use_domain_for_proxy_users=trueproxy_user_domain_name=sahara_proxyproxy_user_role_names=Member,SwiftUser

Volume instance locality configuration¶

The Block Storage service provides the ability to define volume instancelocality to ensure that instance volumes are created on the same hostas the hypervisor. TheInstanceLocalityFilter provides the mechanismfor the selection of a storage provider located on the same physicalhost as an instance.

To enable this functionality for instances of a specific node group, thevolume_local_to_instance field in the node group template should beset totrue and some extra configurations are needed:

• The cinder-volume service should be launched on every physical host and atleast one physical host should run both cinder-scheduler andcinder-volume services.

• InstanceLocalityFilter should be added to the list of default filters(scheduler_default_filters in cinder) for the Block Storageconfiguration.

• The Extended Server Attributes extension needs to be active in the Computeservice (this is true by default in nova), so that theOS-EXT-SRV-ATTR:host property is returned when requesting instanceinfo.

• The user making the call needs to have sufficient rights for the property tobe returned by the Compute service.This can be done by:

  • by changing nova’spolicy.json to allow the user access to theextended_server_attributes option.

  • by designating an account with privileged rights in the cinderconfiguration:

    os_privileged_user_name=os_privileged_user_password=os_privileged_user_tenant=

It should be noted that in a situation when the host has no space for volumecreation, the created volume will have anError state and can not be used.

Autoconfiguration for templates¶

Autoconfiguring templates

NTP service configuration¶

By default sahara will enable the NTP service on all cluster instances if theNTP package is included in the image (the sahara disk image builder willinclude NTP in all images it generates). The default NTP server will bepool.ntp.org; this can be overridden using thedefault_ntp_serversetting in theDEFAULT section of the sahara configuration file.

If you are creating cluster templates using the sahara UI and would like tospecify a different NTP server for a particular cluster template, use theURLofNTPserver setting in theGeneralParameters section when you createthe template. If you would like to disable NTP for a particular clustertemplate, deselect theEnableNTPservice checkbox in theGeneralParameters section when you create the template.

If you are creating clusters using the sahara CLI, you can specify another NTPserver or disable NTP service using the examples below.

If you want to enable configuring the NTP service, you should specify thefollowing configs for the cluster:

cluster_configs: {    "general": {        "URL of NTP server": "your_server.net",    }}

If you want to disable configuring NTP service, you should specify followingconfigs for the cluster:

"cluster_configs": {    "general": {        "Enable NTP service": false,    }}

CORS (Cross Origin Resource Sharing) Configuration¶

Sahara provides direct API access to user-agents (browsers) via the HTTPCORS protocol. Detailed documentation, as well as troubleshooting examples,may be found in the OpenStackAdministrator Guide.

To get started quickly, use the example configuration block below, replacingtheallowedorigin field with the host(s) from which your API expectsaccess.

[cors]allowed_origin=https://we.example.com:443max_age=3600allow_credentials=true[cors.additional_domain_1]allowed_origin=https://additional_domain_1.example.com:443[cors.additional_domain_2]allowed_origin=https://additional_domain_2.example.com:443

For more information on Cross Origin Resource Sharing, please review theW3CCORS specification.

Cleanup time for incomplete clusters¶

Sahara provides maximal time (in hours) for clusters allowed to be in statesother than “Active”, “Deleting” or “Error”. If a cluster is not in “Active”,“Deleting” or “Error” state and last update of it was longer thancleanup_time_for_incomplete_clusters hours ago then it will be deletedautomatically. You can enable this feature by adding appropriate configproperty in theDEFAULT section (by default it set up to0 value whichmeans that automatic clean up is disabled). For example, if you want cluster tobe deleted after 3 hours if it didn’t leave “Starting” state then you shouldspecify:

[DEFAULT]cleanup_time_for_incomplete_clusters=3

Security Group Rules Configuration¶

When auto_security_group is used, the amount of created security group rulesmay be bigger than the default values configured inneutron.conf. Then thedefault limit should be raised up to some bigger value which is proportional tothe number of cluster node groups. You can change it inneutron.conf file:

[quotas]quota_security_group=1000quota_security_group_rule=10000

Or you can execute openstack CLI command:

openstack quota set --secgroups 1000 --secgroup-rules 10000 $PROJECT_ID