Nginx and WSGI

Production StackStorm installations usenginx for SSL termination, servingWeb UI static content, and running st2auth and st2api as WSGI apps via gunicorn/uwsgi. StackStorm nginxconfigurations can be found at/etc/nginx/sites-enabled/st2*.conf.

st2auth andst2api can also run using a built-in simple Python server. This is used fordevelopment and strongly discouraged for any production. Be aware that some settings in/etc/st2.conf are only effective when running in development mode, and don’t apply whenrunning under WSGI servers. Refer to the comments inst2.conf.sample.

Configure MongoDB

StackStorm requires a connection to MongoDB to operate.

In/etc/st2/st2.conf include the following section:

[database]host=<MongoDB host>port=<MongoDB server port>db_name=<User define database name, usually st2>username=<username for db login>password=<password for db login>

Theusername andpassword properties are optional.

StackStorm also supportsMongoDB replica sets usingMongoDB URI string.

In/etc/st2/st2.conf include the following section:

[database]host=mongodb://<#MDB_NODE_1>,<#MDB_NODE_2>,<#MDB_NODE_3>/?replicaSet=<#MDB_REPLICA_SET_NAME>

• You can also add ports, usernames and passwords, etc to your connection string. Seehttps://docs.mongodb.com/manual/reference/connection-string/

• To understand more about setting up a MongoDB replica set, seehttps://docs.mongodb.com/manual/replication/

StackStorm also supports SSL/TLS to encrypt MongoDB connections. A few extra properties need be added tothe configuration apart from the ones outlined above.

In/etc/st2/st2.conf include the following section:

[database]...ssl=<True or False>ssl_keyfile=<Path to key file>ssl_certfile=<Path to certificate>ssl_cert_reqs=<One of none, optional or required>ssl_ca_certs=<Path to certificate form mongod>ssl_match_hostname=<True or False>

• ssl - Enable or disable connection over TLS/SSL or not. Default is False.

• ssl_keyfile - Private keyfile used to identify the local connection against MongoDB. Ifspecified ssl is assumed to be True.

• ssl_certfile - Certificate file used to identify the local connection. If specified ssl isassumed to be True.

• ssl_cert_reqs - Specifies whether a certificate is required from the other side of theconnection, and whether it will be validated if provided.

• ssl_ca_certs - Certificates file containing a set of concatenated CA certificates, which areused to validate certificates passed from MongoDB.

• ssl_match_hostname - Enable or disable hostname matching. Not recommended to disable anddefaults to True.

StackStorm also supports transport / network level compression since StackStorm v3.5.0. Example below showswhich configuration options need to be set to enable compression:

In/etc/st2/st2.conf include the following section:

[database]...compressors=<zstd|zlib>zlib_compression_level=6

• compressors - A comma delimited list of compression algorithm client supports for network /transport level compression which should be advertised to the server. Actual algorithm used willthen be determined based algorithm which is supported both by the client and the server.

• zlib_compression_level - Compression level to use (possible values are from-1 to9)whencompressors is set tozlib.

Configure RabbitMQ

StackStorm uses RabbitMQ for messaging between its services.

In/etc/st2/st2.conf include the following section:

[messaging]url=amqp://#RMQ_USER:#RMQ_PASSWD@#RMQ_HOST:#RMQ_PORT/#RMQ_VHOST

The#RMQ_VHOST property is optional and can be left blank.

StackStorm also supports SSL/TLS to encrypt RabbitMQ connections. To enable SSL, you need to setmessaging.ssl config option toTrue or pass?ssl=true query parameter at the end ofthe connection URL string.

For example:

[messaging]url=amqp://guest:[email protected]:5671/ssl=True

or

[messaging]url=amqp://guest:[email protected]:5671/?ssl=true

In addition to encrypted connection to RabbitMQ, some other SSL related options which aredocumented below are also supported:

[messaging]...ssl=<True or False>ssl_keyfile=<Path to key file>ssl_certfile=<Path to certificate>ssl_cert_reqs=<One of none, optional or required>ssl_ca_certs=<Path to CA certificate>login_method=<One of PLAIN, AMQPLAIN or EXTERNAL>

• ssl - Enable or disable connection over TLS/SSL or not. Default is False.

• ssl_keyfile - Private keyfile used to identify the local connection against RabbitMQ. Ifspecified ssl is assumed to be True.

• ssl_certfile - Certificate file used to identify the local connection. If specified ssl isassumed to be True.

• ssl_cert_reqs - Specifies whether a certificate is required from the other side of theconnection, and whether it will be validated if provided.

• ssl_ca_certs - Certificates file containing a set of concatenated CA certificates, which areused to validate certificates passed from RabbitMQ.

• login_method - Login method to use. Default isPLAIN. Other possibleoptions areAMQPLAIN andEXTERNAL.

StackStorm also supports compressing payloads which are sent over the message bus since StackStorm v3.5.0.Example below shows which configuration options need to be set to enable compression:

[messaging]...compression=<zstd|lzma|bz2|gzip>

StackStorm also supportsRabbitMQ cluster.

In/etc/st2/st2.conf include the following section:

[messaging]cluster_urls=amqp://#RMQ_USER:#RMQ_PASSWD@#RMQ_NODE_1:#RMQ_PORT/#RMQ_VHOST,amqp://#RMQ_USER:#RMQ_PASSWD@#RMQ_NODE_2:#RMQ_PORT/#RMQ_VHOST,amqp://#RMQ_USER:#RMQ_PASSWD@#RMQ_NODE_3:#RMQ_PORT/#RMQ_VHOST

• To understand more about setting up a RabbitMQ cluster, seehttps://www.rabbitmq.com/clustering.html

• RabbitMQ HA guide -https://www.rabbitmq.com/ha.html

Configure SSH

To run actions on remote hosts, StackStorm uses SSH. We recommend using public key-based based SSHaccess on all remote hosts.

The StackStorm ssh user and path to SSH key are set in/etc/st2/st2.conf. During installation,, theone-line install script configures ssh on the local box for the userstanley.

Follow these steps to configure astanley user on remote sytems:

useraddstanleymkdir-p/home/stanley/.sshchmod0700/home/stanley/.ssh# generate ssh keys and copy over public key to remote box.ssh-keygen-f/home/stanley/.ssh/stanley_rsa-P""cp${KEY_LOCATION}/stanley_rsa.pub/home/stanley/.ssh/stanley_rsa.pub# authorize key-based access.cat/home/stanley/.ssh/stanley_rsa.pub>>/home/stanley/.ssh/authorized_keyschmod0600/home/stanley/.ssh/authorized_keyschown-Rstanley:stanley/home/stanleyecho"stanley    ALL=(ALL)       NOPASSWD: SETENV: ALL">>/etc/sudoers.d/st2# ensure requiretty is not set to default in the /etc/sudoers file.sudosed-i-r"s/^Defaults\s+\+requiretty/# Defaults +requiretty/g"/etc/sudoers

To verify, run this from your StackStorm system:

# ssh should not require a password since the key is already providedssh-i/home/stanley/.ssh/stanley_rsa[email protected]# make sure that no password is requiredsudosu

[ssh_runner]use_ssh_config=True...

SUDO Access

StackStorm’sshell actions -local-shell-cmd,local-shell-script,remote-shell-cmd,remote-shell-script- are performed by a special user. By default, this user is namedstanley. This is configurable viast2.conf.

One way of setting up passwordless sudo is perform the below operation on each remote box:

echo"stanley    ALL=(ALL)       NOPASSWD: SETENV: ALL">>/etc/sudoers.d/st2

Configure Logging

By default, the logs can be found in/var/log/st2.

• With the standard logging setup you will see files likest2*.log andst2*.audit.log inthe log folder.

• Per-component logging configuration can be found in/etc/st2/logging.<component>.conf.Those files usePython logging configuration format.Log file location and other settings can be modified in these configuration files, e.g. tochange the output to use syslog instead.

• StackStorm ships with example configuration files to show how to use syslog - these are at/etc/st2/syslog.<component>.conf. To use them, edit/etc/st2/st2.conf, and change thelogging= lines to point to the syslog configuration file. You can also see moreinstructions and example configurations atexchange-misc/syslog.

• By default, log rotation is handled via logrotate. Default log rotation config(logrotate.conf) is included with all package-basedinstallations. Note thathandlers.RotatingFileHandler is used by default in/etc/st2/logging.*.conf, but themaxBytes andbackupCount args are not specified sono rotation is performed by default which then lets logrotate handle the rotation. If you wantPython services instead of logrotate to handle the log rotation, update the logging configs asshown below:

  [handler_fileHandler]class=handlers.RotatingFileHandlerlevel=DEBUGformatter=verboseConsoleFormatterargs=("logs/st2api.log", "a", 100000000, 5)

  In this case the log file will be rotated when it reaches 100000000 bytes (100MB) and a maximumof 5 old log files will be kept. For more information, seeRotatingFileHandler docs.

  Keep in mind that log level names need to be uppercase (e.g.DEBUG,INFO, etc.).

• Sensors run in their own process so it is recommended to not allow sensors to share the sameRotatingFileHandler. To configure a separate handler per sensor/etc/st2/logging.sensorcontainer.conf can be updated as follows, whereMySensor isthe sensor in themypack pack that will have its own log file:

  [loggers]keys=root,MySensor[handlers]keys=consoleHandler, fileHandler, auditHandler, MySensorFileHandler, MySensorAuditHandler[logger_MySensor]level=INFOhandlers=consoleHandler, MySensorFileHandler, MySensorAuditHandlerpropagate=0qualname=st2.SensorWrapper.mypack.MySensor[handler_MySensorFileHandler]class=handlers.RotatingFileHandlerlevel=INFOformatter=verboseConsoleFormatterargs=("logs/mysensor.log",)[handler_vSphereEventSensorAuditHandler]class=handlers.RotatingFileHandlerlevel=AUDITformatter=gelfFormatterargs=("logs/mysensor.audit.log",)

• Check out LogStash configuration and Kibana dashboard for pretty logging and audit atexchange-misc/logstash

Configure The Number of Action Runner Workers

In CentOS/RockyLinux/RHEL the number of action workers defaults to 10. In Ubuntu the number of workersdefaults to the number of CPU cores the machine has. You may wish to increase the number of workersin an HA setup or on system with plenty of resources.

The number of workers can be increased by modifying the environment variableWORKERS. To persistthe number ofst2actionrunner workers, create or edit the environment variable file for yourdistribution and add the number of workers, eg. 25:WORKERS=25. On RHEL/RockyLinux/CentOS we use the/etc/sysconfig/st2actionrunner file and on Ubuntu use the/etc/default/st2actionrunnerfile to pass custom environment variables to thest2actionrunner service/unit:

WORKERS=25

Authentication

Please refer toAuthentication to learn details of authentication, integrations withvarious identity providers, and managing API tokens.

Configure ChatOps

StackStorm brings native two-way ChatOps support. To learn more about ChatOps, and how to configure it manually, please refer toConfiguration section under ChatOps.

Configure secrets masking

In order to manage secrets masking on a system-wide basis you can also modify/etc/st2/st2.confand control secrets masking at 2 levels i.e. API and logs. Note that this feature only controlsexternal visibility of secrets and does not control how secrets are stored as well as managed byStackStorm.

• To mask secrets in API response. This is enabled on a per API basis and only available to adminusers.

  [api]...mask_secrets=True

• To mask secrets in logs:

  [log]...mask_secrets=True

For more information and limitations on secrets masking please refer toSecrets Masking.