Notice

This document is for a development version of Ceph.

Ceph Dashboard

Overview

The Ceph Dashboard is a web-based Ceph management-and-monitoring tool that canbe used to inspect and administer resources in the cluster. It is implementedas aCeph Manager Daemon module.

The original Ceph Dashboard shipped with Ceph Luminous and was a simpleread-only view into the run-time information and performance data of Cephclusters. It had a simple architecture. However, demand grew for richer,web-based management capabilities for users who prefer a WebUI over the CLI.

TheCeph Dashboard module adds web-based monitoring and administrationto the Ceph Manager. The architecture and functionality of this new module arederived from theopenATTIC Ceph management and monitoring tool. Development was originally driven by the openATTICteam atSUSE, with support from members of the Cephcommunity and from companies includingRed Hat.

The dashboard module’s backend code uses the CherryPy framework, and implementsa custom REST API. The WebUI implementation is based on Angular/TypeScript andincludes both functionality from the original dashboard and new featuresoriginally developed for the standalone version of openATTIC. The CephDashboard module is implemented as an application that provides a graphicalrepresentation of information and statistics through a web server hosted byceph-mgr.

Feature Overview

The dashboard provides the following features:

  • Multi-User and Role Management: The dashboard supports multiple useraccounts with different permissions (roles). User accounts and rolescan be managed via both the command line and the WebUI. The dashboardsupports various methods to enhance password security. Passwordcomplexity rules may be configured, requiring users to change their passwordafter the first login or after a configurable time period. SeeUser and Role Management for details.

  • Single Sign-On (SSO): The dashboard supports authentication

    via an external identity provider using the SAML 2.0 protocol or thse OAuth2 protocol. See:ref:dashboard-saml2-sso-support and :ref:dashboard-oauth2-sso-support for details.

  • SSL/TLS support: All HTTP communication between the web browser and thedashboard is secured via SSL. A self-signed certificate can be created witha built-in command, but it’s also possible to import custom certificatessigned and issued by a CA. SeeSSL/TLS Support for details.

  • Auditing: The dashboard backend can be configured to log allPUT,POSTandDELETE API requests in the Ceph audit log. SeeAuditing API Requestsfor instructions on how to enable this feature.

  • Internationalization (I18N): The language used for dashboard text can beselected at run-time.

The Ceph Dashboard offers the following monitoring and management capabilities:

  • Overall cluster health: Display performance and capacity metrics as wellas cluster status.

  • Embedded Grafana Dashboards: Ceph DashboardGrafana dashboards may be embedded in external applications and web pagesto surface information and performance metrics gathered bythePrometheus Module module. SeeEnabling the Embedding of Grafana Dashboards for details on how to configure this functionality.

  • Cluster logs: Display the latest updates to the cluster’s event andaudit log files. Log entries can be filtered by priority, date or keyword.

  • Hosts: Display a list of all cluster hosts along with theirstorage drives, which services are running, and which version of Ceph isinstalled.

  • Performance counters: Display detailed service-specific statistics foreach running service.

  • Monitors: List all Mons, their quorum status, and open sessions.

  • Monitoring: Enable creation, re-creation, editing, and expiration ofPrometheus’ silences, list the alerting configuration and allconfigured and firing alerts. Show notifications for firing alerts.

  • Configuration Editor: Display all available configuration options,their descriptions, types, default and currently set values. These may be edited as well.

  • Pools: List Ceph pools and their details (e.g. applications,pg-autoscaling, placement groups, replication size, EC profile, CRUSHrules, quotas etc.)

  • OSDs: List OSDs, their status and usage statistics as well asdetailed information like attributes (OSD map), metadata, performancecounters and usage histograms for read/write operations. Mark OSDsup/down/out, purge and reweight OSDs, perform scrub operations, modifyvarious scrub-related configuration options, select profiles toadjust the level of backfilling activity. List all drives associated with anOSD. Set and change the device class of an OSD, display and sort OSDs bydevice class. Deploy OSDs on new drives and hosts.

  • Device management: List all hosts known by the orchestrator. List alldrives attached to a host and their properties. Display drivehealth predictions and SMART data. Blink enclosure LEDs.

  • iSCSI: List all hosts that run the TCMU runner service, display allimages and their performance characteristics (read/write ops, traffic).Create, modify, and delete iSCSI targets (viaceph-iscsi). Display theiSCSI gateway status and info about active initiators.SeeEnabling iSCSI Management for instructions on how to configurethis feature.

  • RBD: List all RBD images and their properties (size, objects, features).Create, copy, modify and delete RBD images (incl. snapshots) and manage RBDnamespaces. Define various I/O or bandwidth limitation settings on a global,per-pool or per-image level. Create, delete and rollback snapshots of selectedimages, protect/unprotect these snapshots against modification. Copy or clonesnapshots, flatten cloned images.

  • RBD mirroring: Enable and configure RBD mirroring to a remote Ceph server.List active daemons and their status, pools and RBD images includingsync progress.

  • CephFS: List active file system clients and associated pools,including usage statistics. Evict active CephFS clients. Manage CephFSquotas and snapshots. Browse a CephFS directory structure.

  • Object Gateway: List all active object gateways and their performancecounters. Display and manage (add/edit/delete) object gateway users and theirdetails (e.g. quotas) as well as the users’ buckets and their details (e.g.placement targets, owner, quotas, versioning, multi-factor authentication).SeeEnabling the Object Gateway Management Frontend for configuration instructions.

  • NFS: Manage NFS exports of CephFS file systems and RGW S3 buckets via NFSGanesha. SeeNFS-Ganesha Management for details on how toenable this functionality.

  • Ceph Manager Modules: Enable and disable Ceph Manager modules, managemodule-specific configuration settings.

Overview of the Dashboard Landing Page

The landing page of Ceph Dashboard serves as the home page and features metricssuch as the overall cluster status, performance, and capacity. It provides real-timeupdates on any changes in the cluster and allows quick access to other sections of the dashboard.

../../_images/dashboard-landing-page.png

Note

You can change the landing page to the previous version from:Cluster>>ManagerModules>>Dashboard>>Edit.Editing theFEATURE_TOGGLE_DASHBOARD option will change the landing page, from one view to another.

Note that the previous version of the landing page will be disabled in future releases.

Details

Provides an overview of the cluster configuration, displaying various critical aspects of the cluster.

../../_images/details-card.png

Status

Provides a visual indication of cluster health, and displays cluster alerts grouped by severity.

../../_images/status-card-open.png

Capacity

  • Used: Displays the used capacity out of the total physical capacity provided by storage nodes (OSDs)

  • Warning: Displays thenearfull threshold of the OSDs

  • Danger: Displays thefull threshold of the OSDs

../../_images/capacity-card.png

Inventory

An inventory for all assets within the cluster.Provides direct access to subpages of the dashboard from each item of this card.

../../_images/inventory-card.png

Cluster Utilization

  • Used Capacity: Total capacity used of the cluster. The maximum value of the chart is the maximum capacity of the cluster.

  • IOPS (Input/Output Operations Per Second): Number of read and write operations.

  • Latency: Amount of time that it takes to process a read or a write request.

  • Client Throughput: Amount of data that clients read or write to the cluster.

  • Recovery Throughput: Amount of recovery data that clients read or write to the cluster.

../../_images/cluster-utilization-card.png

Supported Browsers

Ceph Dashboard is primarily tested and developed using the following webbrowsers:

Browser

Versions

Chrome andChromium based browsers

latest 2 major versions

Firefox

latest 2 major versions

Firefox ESR

latest major version

While Ceph Dashboard might work in older browsers, we cannot guarantee compatibility andrecommend keeping your browser up to date.

Enabling

If you have installedceph-mgr-dashboard from distribution packages, thepackage management system should take care of installing all requireddependencies.

If you’re building Ceph from source and want to start the dashboard from yourdevelopment environment, please see the filesREADME.rst andHACKING.rstin the source directorysrc/pybind/mgr/dashboard.

Within a running Ceph cluster, the Ceph Dashboard is enabled with:

cephmgrmoduleenabledashboard

Configuration

SSL/TLS Support

All HTTP connections to the dashboard are secured with SSL/TLS by default.

To get the dashboard up and running quickly, you can generate and install aself-signed certificate:

cephdashboardcreate-self-signed-cert

Note that most web browsers will complain about self-signed certificatesand require explicit confirmation before establishing a secure connection to thedashboard.

To properly secure a deployment and to remove the warning, acertificate that is issued by a certificate authority (CA) should be used.

For example, a key pair can be generated with a command similar to:

opensslreq-new-nodes-x509\-subj"/O=IT/CN=ceph-mgr-dashboard"-days3650\-keyoutdashboard.key-outdashboard.crt-extensionsv3_ca

Thedashboard.crt file should then be signed by a CA. Once that is done, youcan enable it for Ceph manager instances by running the following commands:

cephdashboardset-ssl-certificate-idashboard.crtcephdashboardset-ssl-certificate-key-idashboard.key

If unique certificates are desired for each manager instance,the name of the instance can be included as follows (where$name is the nameof theceph-mgr instance, usually the hostname):

cephdashboardset-ssl-certificate$name-idashboard.crtcephdashboardset-ssl-certificate-key$name-idashboard.key

SSL can also be disabled by setting this configuration value:

cephconfigsetmgrmgr/dashboard/sslfalse

This might be useful if the dashboard will be running behind a proxy which doesnot support SSL for its upstream servers or other situations where SSL is notwanted or required. SeeProxy Configuration for more details.

Warning

Use caution when disabling SSL as usernames and passwords will be sent to thedashboard unencrypted.

Note

You must restart Ceph manager processes after changing the SSLcertificate and key. This can be accomplished by either runningcephmgrfailmgr or by disabling and re-enabling the dashboard module (which alsotriggers the manager to respawn itself):

cephmgrmoduledisabledashboardcephmgrmoduleenabledashboard

Host Name and Port

Like most web applications, the dashboard binds to a TCP/IP address and TCP port.

By default, theceph-mgr daemon hosting the dashboard (i.e., the currentlyactive manager) will bind to TCP port 8443 or 8080 when SSL is disabled.

If no specific address has been configured, the web app will bind to::,which corresponds to all available IPv4 and IPv6 addresses.

These defaults can be changed via the configuration key facility on acluster-wide level (so they apply to all manager instances) as follows:

cephconfigsetmgrmgr/dashboard/server_addr$IPcephconfigsetmgrmgr/dashboard/server_port$PORTcephconfigsetmgrmgr/dashboard/ssl_server_port$PORT

Since eachceph-mgr hosts its own instance of the dashboard, it may benecessary to configure them separately. The IP address and port for a specificmanager instance can be changed with the following commands:

cephconfigsetmgrmgr/dashboard/$name/server_addr$IPcephconfigsetmgrmgr/dashboard/$name/server_port$PORTcephconfigsetmgrmgr/dashboard/$name/ssl_server_port$PORT

Replace$name with the ID of the ceph-mgr instance hosting the dashboard.

Note

The commandcephmgrservices will show you all endpoints that arecurrently configured. Look for thedashboard key to obtain the URL foraccessing the dashboard.

Username and Password

In order to be able to log in, you need to create a user account and associateit with at least one role. We provide a set of predefinedsystem roles thatyou can use. For more details please refer to theUser and Role Managementsection.

To create a user with the administrator role you can use the followingcommands:

cephdashboardac-user-create<username>-i<file-containing-password>administrator

Account Lock-out

It disables a user account if a user repeatedly enters the wrong credentialsfor multiple times. It is enabled by default to prevent brute-force or dictionaryattacks. The user can get or set the default number of lock-out attempts usingthese commands respectively:

cephdashboardget-account-lockout-attemptscephdashboardset-account-lockout-attempts<value:int>

Warning

This feature can be disabled by setting the default number of lock-out attempts to 0.However, by disabling this feature, the account is more vulnerable to brute-force ordictionary based attacks. This can be disabled by:

cephdashboardset-account-lockout-attempts0

Enable a Locked User

If a user account is disabled as a result of multiple invalid login attempts, thenit needs to be manually enabled by the administrator. This can be done by the followingcommand:

cephdashboardac-user-enable<username>

Accessing the Dashboard

You can now access the dashboard using your (JavaScript-enabled) web browser, bypointing it to any of the host names or IP addresses and the selected TCP portwhere a manager instance is running: e.g.,http(s)://<$IP>:<$PORT>/.

The dashboard page displays and requests a previously defined username andpassword.

Enabling the Object Gateway Management Frontend

When RGW is deployed with cephadm, the RGW credentials used by thedashboard will be automatically configured. You can also manually force thecredentials to be set up with:

cephdashboardset-rgw-credentials

This will create an RGW user with uiddashboard for each realm inthe system.

If you’ve configured a custom ‘admin’ resource in your RGW admin API, you should set it here also:

cephdashboardset-rgw-api-admin-resource<admin_resource>

If you are using a self-signed certificate in your Object Gateway setup,you should disable certificate verification in the dashboard to avoid refusedconnections, e.g. caused by certificates signed by unknown CA or not matchingthe host name:

cephdashboardset-rgw-api-ssl-verifyFalse

To set a custom hostname or address for an RGW gateway, set the value ofRGW_HOSTNAME_PER_DAEMONaccordingly:

cephdashboardset-rgw-hostname<gateway_name><hostname>

The setting can be unset using:

cephdashboardunset-rgw-hostname<gateway_name>

If the Object Gateway takes too long to process requests and the dashboard runsinto timeouts, you can set the timeout value to your needs:

cephdashboardset-rest-requests-timeout<seconds>

The default value is 45 seconds.

Enabling iSCSI Management

The Ceph Dashboard can manage iSCSI targets using the REST API provided by therbd-target-api service of theCeph iSCSI Gateway. Please make sure that it isinstalled and enabled on the iSCSI gateways.

Note

The iSCSI management functionality of Ceph Dashboard depends on the latestversion 3 of theceph-iscsi project.Make sure that your operating system provides the correct version, otherwisethe dashboard will not enable the management features.

If theceph-iscsi REST API is configured in HTTPS mode and its using a self-signedcertificate, you need to configure the dashboard to avoid SSL certificateverification when accessing ceph-iscsi API.

To disable API SSL verification run the following command:

cephdashboardset-iscsi-api-ssl-verificationfalse

The available iSCSI gateways must be defined using the following commands:

cephdashboardiscsi-gateway-list# Gateway URL format for a new gateway: <scheme>://<username>:<password>@<host>[:port]cephdashboardiscsi-gateway-add-i<file-containing-gateway-url>[<gateway_name>]cephdashboardiscsi-gateway-rm<gateway_name>

Enabling the Embedding of Grafana Dashboards

Grafana pulls data fromPrometheus. AlthoughGrafana can use other data sources, the Grafana dashboards we provide containqueries that are specific to Prometheus. Our Grafana dashboards thereforerequire Prometheus as the data source. The CephPrometheus Modulemodule exports its data in the Prometheus exposition format. These Grafanadashboards rely on metric names from the Prometheus module andNode exporter. The Node exporter is aseparate application that provides machine metrics.

Note

Prometheus’ security model presumes that untrusted users have access to thePrometheus HTTP endpoint and logs. Untrusted users have access to all the(meta)data Prometheus collects that is contained in the database, plus avariety of operational and debugging information.

However, Prometheus’ HTTP API is limited to read-only operations.Configurations cannot be changed using the API and secrets are notexposed. Moreover, Prometheus has some built-in measures to mitigate theimpact of denial of service attacks.

Please seePrometheus’ Security model<https://prometheus.io/docs/operating/security/> for more detailedinformation.

Installation and Configuration using cephadm

Grafana and Prometheus can be installed usingCephadm. They willautomatically be configured bycephadm. Please seeMonitoring Services documentation for more details on how to usecephadm for installing and configuring Prometheus and Grafana.

Manual Installation and Configuration

The following process describes how to configure Grafana and Prometheusmanually. After you have installed Prometheus, Grafana, and the Node exporteron appropriate hosts, proceed with the following steps.

  1. Enable the Ceph Exporter which comes as Ceph Manager module by running:

    cephmgrmoduleenableprometheus

    More details can be found in the documentation of thePrometheus Module.

  2. Add the corresponding scrape configuration to Prometheus. This may looklike:

    global:scrape_interval:5sscrape_configs:-job_name:'prometheus'static_configs:-targets:['localhost:9090']-job_name:'ceph'static_configs:-targets:['localhost:9283']-job_name:'node-exporter'static_configs:-targets:['localhost:9100']

    Note

    Please note that in the above example, Prometheus is configuredto scrape data from itself (port 9090), the Ceph manager moduleprometheus (port 9283), which exports Ceph internal data, and the NodeExporter (port 9100), which provides OS and hardware metrics for each host.

    Depending on your configuration, you may need to change the hostname inor add additional configuration entries for the NodeExporter. It is unlikely that you will need to change the default TCP ports.

    Moreover, you don’tneed to have more than one target for Ceph specificdata, provided by theprometheus mgr module. But it is recommended toconfigure Prometheus to scrape Ceph specific data from all existing Cephmanagers. This enables a built-in high availability mechanism, so thatservices run on a manager host will be restarted automatically on a differentmanager host if one Ceph Manager goes down.

  3. Add Prometheus as data source to Grafanausing the Grafana Web UI.

    Important

    The data source must be named “Dashboard1”.

  4. Install thevonage-status-panel and grafana-piechart-panel plugins using:

    grafana-clipluginsinstallvonage-status-panelgrafana-clipluginsinstallgrafana-piechart-panel
  5. Add Dashboards to Grafana:

    Dashboards can be added to Grafana by importing dashboard JSON files.Use the following command to download the JSON files:

    wgethttps://raw.githubusercontent.com/ceph/ceph/main/monitoring/ceph-mixin/dashboards_out/<Dashboard-name>.json

    You can find various dashboard JSON fileshere.

    For Example, for ceph-cluster overview you can use:

    wgethttps://raw.githubusercontent.com/ceph/ceph/main/monitoring/ceph-mixin/dashboards_out/ceph-cluster.json

    You may also author your own dashboards.

  6. Configure anonymous mode in/etc/grafana/grafana.ini:

    [auth.anonymous]enabled=trueorg_name=MainOrg.org_role=Viewer

    In newer versions of Grafana (starting with 6.2.0-beta1) a new setting namedallow_embedding has been introduced. This setting must be explicitlyset totrue for the Grafana integration in Ceph Dashboard to work, as thedefault isfalse.

    [security]allow_embedding=true

Enabling RBD-Image monitoring

Monitoring of RBD images is disabled by default, as it can significantly impactperformance. For more information please seeCeph Health Checks.When disabled, the overview and details dashboards will be empty in Grafana andmetrics will not be visible in Prometheus.

Configuring Dashboard

After you have set up Grafana and Prometheus, you will need to configure theconnection information that the Ceph Dashboard will use to access Grafana.

You need to tell the dashboard on which URL the Grafana instance isrunning/deployed:

cephdashboardset-grafana-api-url<grafana-server-url># default: ''

The format of url is :<protocol>:<IP-address>:<port>

Note

The Ceph Dashboard embeds Grafana dashboards viaiframe HTML elements.If Grafana is configured without SSL/TLS support, most browsers will block theembedding of insecure content if SSL support isenabled for the dashboard (which is the default). If youcan’t see the embedded Grafana dashboards after enabling them as outlinedabove, check your browser’s documentation on how to unblock mixed content.Alternatively, consider enabling SSL/TLS support in Grafana.

If you are using a self-signed certificate for Grafana,disable certificate verification in the dashboard to avoid refused connections,which can be a result of certificates signed by an unknown CA or that do notmatch the host name:

cephdashboardset-grafana-api-ssl-verifyFalse

You can also access Grafana directly to monitor your cluster.

Note

Ceph Dashboard configuration information can also be unset. For example, toclear the Grafana API URL we configured above:

cephdashboardreset-grafana-api-url

Alternative URL for Browsers

The Ceph Dashboard backend requires the Grafana URL to be able to verify theexistence of Grafana Dashboards before the frontend even loads them. Due to thenature of how Grafana is implemented in Ceph Dashboard, this means that twoworking connections are required in order to be able to see Grafana graphs inCeph Dashboard:

  • The backend (Ceph Mgr module) needs to verify the existence of the requestedgraph. If this request succeeds, it lets the frontend know that it can safelyaccess Grafana.

  • The frontend then requests the Grafana graphs directly from the user’sbrowser using an iframe. The Grafana instance is accessed directly without anydetour through Ceph Dashboard.

Now, it might be the case that your environment makes it difficult for theuser’s browser to directly access the URL configured in Ceph Dashboard. To solvethis issue, a separate URL can be configured which will solely be used to tellthe frontend (the user’s browser) which URL it should use to access Grafana.This setting won’t ever be changed automatically, unlike the GRAFANA_API_URLwhich is set byCephadm (only if cephadm is used to deploy monitoringservices).

To change the URL that is returned to the frontend issue the following command:

cephdashboardset-grafana-frontend-api-url<grafana-server-url>

If no value is set for that option, it will simply fall back to the value of theGRAFANA_API_URL option. If set, it will instruct the browser to use this URL toaccess Grafana.

Enabling SAML2 Single Sign-On (SSO)

The Ceph Dashboard supports external authentication of users via theSAML 2.0 protocol. You need tofirst create user accounts and associate them with desired roles, asauthorization is performed by the Dashboard. However, the authenticationprocess can be performed by an existing Identity Provider (IdP).

Note

Ceph Dashboard SSO support relies on onelogin’spython-saml library.Please ensure that this library is installed on your system, either by usingyour distribution’s package management or via Python’spip installer.

To configure SSO on Ceph Dashboard, you should use the following command:

cephdashboardssosetupsaml2<ceph_dashboard_base_url><idp_metadata>{<idp_username_attribute>}{<idp_entity_id>}{<sp_x_509_cert>}{<sp_private_key>}

Parameters:

  • <ceph_dashboard_base_url>: Base URL where Ceph Dashboard is accessible (e.g.,https://cephdashboard.local)

  • <idp_metadata>: URL to remote (http://,https://) or local (file://) path or content of the IdP metadata XML (e.g.,https://myidp/metadata,file:///home/myuser/metadata.xml).

  • <idp_username_attribute>(optional): Attribute that should be used to get the username from the authentication response. Defaults touid.

  • <idp_entity_id>(optional): Use this when more than one entity id exists on the IdP metadata.

  • <sp_x_509_cert> / <sp_private_key>(optional): File path of the certificate that should be used by Ceph Dashboard (Service Provider) for signing and encryption (these file paths should be accessible from the active ceph-mgr instance).

Note

The issuer value of SAML requests will follow this pattern:<ceph_dashboard_base_url>/auth/saml2/metadata

To display the current SAML 2.0 configuration, use the following command:

cephdashboardssoshowsaml2

Note

For more information aboutonelogin_settings, please check theonelogin documentation.

To disable SSO:

cephdashboardssodisable

To check if SSO is enabled:

cephdashboardssostatus

To enable SSO:

cephdashboardssoenablesaml2

Enabling OAuth2 Single Sign-On (SSO)

The Ceph Dashboard supports external authentication of users via theOAuth protocol. You need tohaveCephadm enabled as your orchestrator with an activeManagement Gateway andOAuth2 Proxy services.

From the IDP of choice, Keycloak is the current recomendation and tested solution,configure the IDP’s client used in theoauth2-proxy service configuration to validate the following redirect URLsfor login_url:https://<host_name>|<IP_address>/oauth2/callback andthe following logout_url:https://<host_name>|<IP_address>/ /oauth2/sign_out

Again, from the IDP, we will need a user with a valid role, this user will be the one to performauthorization against, we can create a role like: ‘administator’ to give admin level access to the user.

Make certain that theenable_auth flag has been included in thecephorchapplymgmt-gateway command and that it has been set totrue by running acommand of the following form:

cephorchapplymgmt-gateway--enable_auth=true--placement=<ceph-node-02>

To disable SSO:

cephdashboardssodisable

To check if SSO is enabled:

cephdashboardssostatus

To enable SSO:

cephdashboardssoenableoauth2

Enabling Prometheus Alerting

To use Prometheus for alerting you must definealerting rules.These are managed by theAlertmanager.If you are not yet using the Alertmanager,install it as it receivesand manages alerts from Prometheus.

Alertmanager capabilities can be consumed by the dashboard in three differentways:

  1. Use the notification receiver of the dashboard.

  2. Use the Prometheus Alertmanager API.

  3. Use both sources simultaneously.

All three methods notify you about alerts. You won’t be notifiedtwice if you use both sources, but you need to consume at least the Alertmanager APIin order to manage silences.

  1. Use the notification receiver of the dashboard

This allows you to get notifications asconfigured from the Alertmanager.You will get notified inside the dashboard once a notification is send out,but you are not able to manage alerts.

Add the dashboard receiver and the new route to your Alertmanagerconfiguration. This should look like:

route:receiver:'ceph-dashboard'...receivers:-name:'ceph-dashboard'webhook_configs:-url:'<url-to-dashboard>/api/prometheus_receiver'

Ensure that the Alertmanager considers your SSL certificate in termsof the dashboard as valid. For more information about the correctconfiguration checkout the<http_config> documentation.

  1. Use the API of Prometheus and the Alertmanager

This allows you to manage alerts and silences and will enable the “ActiveAlerts”, “All Alerts” as well as the “Silences” tabs in the “Monitoring”section of the “Cluster” menu entry.

Alerts can be sorted by name, job, severity, state and start time.Unfortunately it’s not possible to know when an alert was sent out through anotification by the Alertmanager based on your configuration, that’s why thedashboard will notify the user on any visible change to an alert and willnotify the changed alert.

Silences can be sorted by id, creator, status, start, updated and end time.Silences can be created in various ways, it’s also possible to expire them.

  1. Create from scratch

  2. Based on a selected alert

  3. Recreate from expired silence

  4. Update a silence (which will recreate and expire it (default Alertmanager behaviour))

To use it, specify the host and port of the Alertmanager server:

cephdashboardset-alertmanager-api-host<alertmanager-host:port># default: ''

For example:

cephdashboardset-alertmanager-api-host'http://localhost:9093'

To be able to see all configured alerts, you will need to configure the URL tothe Prometheus API. Using this API, the UI will also help you in verifyingthat a new silence will match a corresponding alert.

cephdashboardset-prometheus-api-host<prometheus-host:port># default: ''

For example:

cephdashboardset-prometheus-api-host'http://localhost:9090'

After setting up the hosts, refresh your browser’s dashboard window or tab.

  1. Use both methods

The behaviors of both methods are configured in a way that theyshould not disturb each other, through annoying duplicated notificationsmay pop up.

If you are using a self-signed certificate in your Prometheus or yourAlertmanager setup, you should disable certificate verification in thedashboard to avoid refused connections caused by certificates signed byan unknown CA or that do not match the host name.

  • For Prometheus:

cephdashboardset-prometheus-api-ssl-verifyFalse
  • For Alertmanager:

cephdashboardset-alertmanager-api-ssl-verifyFalse

User and Role Management

Password Policy

By default the password policy feature is enabled, which includes thefollowing checks:

  • Is the password longer than N characters?

  • Are the old and new password the same?

The password policy feature can be switched on or off completely:

cephdashboardset-pwd-policy-enabled<true|false>

The following individual checks can also be switched on or off:

cephdashboardset-pwd-policy-check-length-enabled<true|false>cephdashboardset-pwd-policy-check-oldpwd-enabled<true|false>cephdashboardset-pwd-policy-check-username-enabled<true|false>cephdashboardset-pwd-policy-check-exclusion-list-enabled<true|false>cephdashboardset-pwd-policy-check-complexity-enabled<true|false>cephdashboardset-pwd-policy-check-sequential-chars-enabled<true|false>cephdashboardset-pwd-policy-check-repetitive-chars-enabled<true|false>

Additionally the following options are available to configure passwordpolicy.

  • Minimum password length (defaults to 8):

cephdashboardset-pwd-policy-min-length<N>
  • Minimum password complexity (defaults to 10):

    cephdashboardset-pwd-policy-min-complexity<N>

    Password complexity is calculated by classifying each character inthe password. The complexity count starts by 0. A character is rated bythe following rules in the given order.

    • Increase by 1 if the character is a digit.

    • Increase by 1 if the character is a lower case ASCII character.

    • Increase by 2 if the character is an upper case ASCII character.

    • Increase by 3 if the character is a special character like!"#$%&'()*+,-./:;<=>?@[\]^_`{|}~.

    • Increase by 5 if the character has not been classified by one of the previous rules.

  • A list of comma separated words that are not allowed to be used in apassword:

    cephdashboardset-pwd-policy-exclusion-list<word>[,...]

User Accounts

The Ceph Dashboard supports multiple user accounts. Each user accountconsists of a username, a password (stored in encrypted form usingbcrypt),an optional name, and an optional email address.

If a new user is created via the Web UI, it is possible to set an option that theuser must assign a new password when they log in for the first time.

User accounts are stored in the monitors’ configuration database, and areavailable to allceph-mgr instances.

We provide a set of CLI commands to manage user accounts:

  • Show User(s):

    cephdashboardac-user-show[<username>]
  • Create User:

    cephdashboardac-user-create[--enabled][--force-password][--pwd_update_required]<username>-i<file-containing-password>[<rolename>][<name>][<email>][<pwd_expiration_date>]

    To bypass password policy checks use theforce-password option.Add the optionpwd_update_required so that a newly created user hasto change their password after the first login.

  • Delete User:

    cephdashboardac-user-delete<username>
  • Change Password:

    cephdashboardac-user-set-password[--force-password]<username>-i<file-containing-password>
  • Change Password Hash:

    cephdashboardac-user-set-password-hash<username>-i<file-containing-password-hash>

    The hash must be a bcrypt hash and salt, e.g.$2b$12$Pt3Vq/rDt2y9glTPSV.VFegiLkQeIpddtkhoFetNApYmIJOY8gau2.This can be used to import users from an external database.

  • Modify User (name, and email):

    cephdashboardac-user-set-info<username><name><email>
  • Disable User:

    cephdashboardac-user-disable<username>
  • Enable User:

    cephdashboardac-user-enable<username>

User Roles and Permissions

User accounts are associated with a set of roles that define whichdashboard functionality can be accessed.

The Dashboard functionality/modules are grouped within asecurity scope.Security scopes are predefined and static. The current available securityscopes are:

  • hosts: includes all features related to theHosts menuentry.

  • config-opt: includes all features related to management of Cephconfiguration options.

  • pool: includes all features related to pool management.

  • osd: includes all features related to OSD management.

  • monitor: includes all features related to monitor management.

  • rbd-image: includes all features related to RBD imagemanagement.

  • rbd-mirroring: includes all features related to RBD mirroringmanagement.

  • iscsi: includes all features related to iSCSI management.

  • rgw: includes all features related to RADOS Gateway (RGW) management.

  • cephfs: includes all features related to CephFS management.

  • nfs-ganesha: includes all features related to NFS Ganesha management.

  • manager: include all features related to Ceph Managermanagement.

  • log: include all features related to Ceph logs management.

  • grafana: include all features related to Grafana proxy.

  • prometheus: include all features related to Prometheus alert management.

  • dashboard-settings: allows to change dashboard settings.

Arole specifies a set of mappings between asecurity scope and a set ofpermissions. There are four types of permissions:

  • read

  • create

  • update

  • delete

See below for an example of a role specification, in the form of a Python dictionary:

# example of a role{'role':'my_new_role','description':'My new role','scopes_permissions':{'pool':['read','create'],'rbd-image':['read','create','update','delete']}}

The above role dictates that a user hasread andcreate permissions forfeatures related to pool management, and has full permissions forfeatures related to RBD image management.

The Dashboard provides a set of predefined roles that we callsystem roles, which can be used right away by a fresh Ceph Dashboardinstallation.

The list of system roles are:

  • administrator: allows full permissions for all security scopes.

  • read-only: allowsread permission for all security scopes exceptdashboard settings.

  • block-manager: allows full permissions forrbd-image,rbd-mirroring, andiscsi scopes.

  • rgw-manager: allows full permissions for thergw scope

  • cluster-manager: allows full permissions for thehosts,osd,monitor,manager, andconfig-opt scopes.

  • pool-manager: allows full permissions for thepool scope.

  • cephfs-manager: allows full permissions for thecephfs scope.

The list of available roles can be retrieved with the following command:

cephdashboardac-role-show[<rolename>]

You can also use the CLI to create new roles. The available commands are thefollowing:

  • Create Role:

    cephdashboardac-role-create<rolename>[<description>]
  • Delete Role:

    cephdashboardac-role-delete<rolename>
  • Add Scope Permissions to Role:

    cephdashboardac-role-add-scope-perms<rolename><scopename><permission>[<permission>...]
  • Delete Scope Permission from Role:

    cephdashboardac-role-del-scope-perms<rolename><scopename>

To assign roles to users, the following commands are available:

  • Set User Roles:

    cephdashboardac-user-set-roles<username><rolename>[<rolename>...]
  • Add Roles To User:

    cephdashboardac-user-add-roles<username><rolename>[<rolename>...]
  • Delete Roles from User:

    cephdashboardac-user-del-roles<username><rolename>[<rolename>...]

Example of User and Custom Role Creation

In this section we show a complete example of the commands thatcreate a user account that can manage RBD images, view and create Ceph pools,and has read-only access to other scopes.

  1. Create the user:

    cephdashboardac-user-createbob-i<file-containing-password>
  2. Create role and specify scope permissions:

    cephdashboardac-role-createrbd/pool-managercephdashboardac-role-add-scope-permsrbd/pool-managerrbd-imagereadcreateupdatedeletecephdashboardac-role-add-scope-permsrbd/pool-managerpoolreadcreate
  3. Associate roles to user:

    cephdashboardac-user-set-rolesbobrbd/pool-managerread-only

Proxy Configuration

In a Ceph cluster with multipleceph-mgr instances, only the dashboardrunning on the currently activeceph-mgr daemon will serve incoming requests.Connections to the dashboard’s TCP port on standbyceph-mgr instanceswill receive an HTTP redirect (303) to the active manager’s dashboard URL.This enables you to point your browser to anyceph-mgr instance inorder to access the dashboard.

If you want to establish a fixed URL to reach the dashboard or if you don’t wantto allow direct connections to the manager nodes, you could set up a proxy thatautomatically forwards incoming requests to the activeceph-mgrinstance.

Configuring a URL Prefix

If you are accessing the dashboard via a reverse proxy,you may wish to service it under a URL prefix. To get the dashboardto use hyperlinks that include your prefix, you can set theurl_prefix setting:

cephconfigsetmgrmgr/dashboard/url_prefix$PREFIX

so you can access the dashboard athttp://$IP:$PORT/$PREFIX/.

Disable the redirection

If the dashboard is behind a load-balancing proxy likeHAProxyyou might want to disable redirection to prevent situations in whichinternal (unresolvable) URLs are published to the frontend client. Use thefollowing command to get the dashboard to respond with an HTTP error (500 by default)instead of redirecting to the active dashboard:

cephconfigsetmgrmgr/dashboard/standby_behaviour"error"

To reset the setting to default redirection, use the following command:

cephconfigsetmgrmgr/dashboard/standby_behaviour"redirect"

Configure the error status code

When redirection is disabled, you may want to customize the HTTP statuscode of standby dashboards. To do so you need to run the command:

cephconfigsetmgrmgr/dashboard/standby_error_status_code503

Resolve IP address to hostname before redirect

Redirection from a standby dashboard to the active dashboard is done via themanager’s IP address, not via the manager’s hostname. In virtualizedenvironments, IP-address-based redirection reduces the incidence of error ascompared to hostname-based resolution. Because of the increased risk of errordue to hostname-based resolution, the option for hostname resolution isdisabled by default.

However, in some situations it might be helpful to redirect via the hostname.For example, if the configured TLS certificate matches only the hostnames andnot the IP addresses of those hosts, hostname redirection would be preferable.

To activate redirection from standby dashboards to active dashboards via themanager’s hostname, run the following command:

cephconfigsetmgrmgr/dashboard/redirect_resolve_ip_addrTrue

Disable hostname redirection by running the following command:

cephconfigsetmgrmgr/dashboard/redirect_resolve_ip_addrFalse

Warning

If you attempt to activate redirection by using the command above and youget the error messageEINVAL:unrecognizedconfigoption'mgr/dashboard/redirect_resolve_ip_addr', then you might be running arelease of Ceph prior to version 17.2.6. This feature was introduced in17.2.6, in this commit:https://github.com/ceph/ceph/pull/48219.

HAProxy example configuration

Below you will find an example configuration for SSL/TLS passthrough usingHAProxy.

Please note that this configuration works under the following conditions.If the dashboard fails over, the front-end client might receive a HTTP redirect(303) response and will be redirected to an unresolvable host. This happens whenfailover occurs between two HAProxy health checks. In this situation thepreviously active dashboard node will now respond with a 303 which points tothe new active node. To prevent that situation you should consider disablingredirection on standby nodes.

defaults  log global  option log-health-checks  timeout connect 5s  timeout client 50s  timeout server 450sfrontend dashboard_front  mode http  bind *:80  option httplog  redirect scheme https code 301 if !{ ssl_fc }frontend dashboard_front_ssl  mode tcp  bind *:443  option tcplog  default_backend dashboard_back_sslbackend dashboard_back_ssl  mode tcp  option httpchk GET /  http-check expect status 200  server x <HOST>:<PORT> check check-ssl verify none  server y <HOST>:<PORT> check check-ssl verify none  server z <HOST>:<PORT> check check-ssl verify none

Auditing API Requests

The REST API can log PUT, POST and DELETE requests to the Cephaudit log. This feature is disabled by default, but can be enabled with thefollowing command:

cephdashboardset-audit-api-enabled<true|false>

If enabled, the following parameters are logged per each request:

  • from - The origin of the request, e.g.https://[::1]:44410

  • path - The REST API path, e.g. /api/auth

  • method - e.g. PUT, POST or DELETE

  • user - The name of the user, otherwise ‘None’

The logging of the request payload (the arguments and their values) is enabledby default. Execute the following command to disable this behaviour:

cephdashboardset-audit-api-log-payload<true|false>

A log entry may look like this:

2018-10-2215:27:01.302514mgr.x[INF][DASHBOARD]from='https://[::ffff:127.0.0.1]:37022'path='/api/rgw/user/klaus'method='PUT'user='admin'params='{"max_buckets": "1000", "display_name": "Klaus Mustermann", "uid": "klaus", "suspended": "0", "email": "klaus.mustermann@ceph.com"}'

NFS-Ganesha Management

The dashboard requires enabling the NFS module which will be used to manageNFS clusters and NFS exports. For more information checkCephFS & RGW Exports over NFS.

Plug-ins

Plug-ins extend the functionality of the Ceph Dashboard in a modularand loosely coupled fashion.

Feature Toggles

This plug-in allows to enable or disable some features from the Ceph Dashboardon-demand. When a feature becomes disabled:

  • Its front-end elements (web pages, menu entries, charts, etc.) will becomehidden.

  • Its associated REST API endpoints will reject any further requests (404, NotFound Error).

The main purpose of this plug-in is to allow ad hoc customizations of theworkflows exposed by the dashboard. Additionally, it could allow for dynamicallyenabling experimental features with minimal configuration burden and no serviceimpact.

The list of features that can be enabled/disabled is:

  • Block (RBD):
    • Image Management:rbd

    • Mirroring:mirroring

    • iSCSI:iscsi

  • Filesystem (Cephfs):cephfs

  • Objects (RGW):rgw (including daemon, user and bucket management).

  • NFS:nfs-ganesha exports.

By default all features come enabled.

To retrieve a list of features and their current statuses:

cephdashboardfeaturestatus
Feature'cephfs':'enabled'Feature'iscsi':'enabled'Feature'nvmeof':'enabled'Feature'mirroring':'enabled'Feature'rbd':'enabled'Feature'rgw':'enabled'Feature'nfs':'enabled'

To enable or disable the status of a single or multiple features:

cephdashboardfeaturedisableiscsimirroring
Feature'iscsi':disabledFeature'mirroring':disabled

After a feature status has changed, the API REST endpoints immediately respondto that change, but it may take up to twenty (20) seconds for the front-end UIelements seconds to reflect the change.

Debug

This plugin allows to customize the behaviour of the dashboard according to thedebug mode. It can be enabled, disabled or checked with the following command:

cephdashboarddebugstatus
Debug:'disabled'
cephdashboarddebugenable
Debug:'enabled'
cephdashboarddebugdisable
Debug:'disabled'

By default, it’s disabled. This is the recommended setting for productiondeployments. If required, debug mode can be enabled without need of restarting.Currently, disabled debug mode equals to CherryPyproduction environment,while when enabled, it usestest_suite defaults (please refer toCherryPy Environments for moredetails).

It also adds request uuid (unique_id) to Cherrypy on versions that don’tsupport this. It additionally prints theunique_id to error responses andlog messages.

Message of the day (MOTD)

Displays a configured “message of the day” (MOTD) at the top of the CephDashboard.

The importance of an MOTD can be configured by its severity, which isinfo,warning ordanger. The MOTD can expire after a given time,this means it will not be displayed in the UI anymore. Use the followingsyntax to specify the expiration time:Ns|m|h|d|w, whereN is aninteger followed by seconds, minutes, hours, days and weeks. If the MOTDshould expire after 2 hours, use2h or5w for 5 weeks. Use0 toconfigure a MOTD that does not expire.

To configure a MOTD, run the following command:

cephdashboardmotdset<severity:info|warning|danger><expires><message>

To show the configured MOTD:

cephdashboardmotdget

To clear the configured MOTD run:

cephdashboardmotdclear

A MOTD with a severity ofinfo orwarning can be closed by the user.Theinfo MOTD is not displayed until the local storage cookies arecleared or a new MOTD with a different severity is displayed. A MOTD with awarning severity will be displayed again in a new session.

Troubleshooting the Dashboard

Locating the Dashboard

If you are unsure of the location of the Ceph Dashboard, run the following command:

cephmgrservices|jq.dashboard
"https://host:port"

The command returns the URL where the Ceph Dashboard is located:https://<host>:<port>/

Note

Many Ceph tools return results in JSON format. We suggest thatyou install thejq command-lineutility to facilitate working with JSON data.

Accessing the Dashboard

If you are unable to access the Ceph Dashboard, run the followingcommands:

  1. Verify the Ceph Dashboard module is enabled:

    cephmgrmodulels|jq.enabled_modules

    Ensure the Ceph Dashboard module is listed in the return value of thecommand. Example snipped output from the command above:

    ["dashboard","iostat","restful"]
  2. If it is not listed, activate the module with the following command:

    cephmgrmoduleenabledashboard
  3. Check the Ceph Dashboard and/orceph-mgr log files for any errors.

    • Check ifceph-mgr log messages are written to a file by:

      cephconfiggetmgrlog_to_file
      true
    • Get the location of the log file (it’s/var/log/ceph/<cluster-name>-<daemon-name>.logby default):

      cephconfiggetmgrlog_file
      /var/log/ceph/$cluster-$name.log
  4. Ensure the SSL/TLS support is configured properly:

    • Check if the SSL/TLS support is enabled:

      cephconfiggetmgrmgr/dashboard/ssl
    • If the command returnstrue, verify a certificate exists by:

      cephconfig-keygetmgr/dashboard/crt

      and:

      cephconfig-keygetmgr/dashboard/key
    • If it doesn’t returntrue, run the following command to generate a self-signedcertificate or follow the instructions outlined inSSL/TLS Support:

      cephdashboardcreate-self-signed-cert

Trouble Logging into the Dashboard

If you are unable to log into the Ceph Dashboard and you receive the followingerror, run through the procedural checks below:

../../_images/invalid-credentials.png
  1. Check that your user credentials are correct. If you are seeing thenotification message above when trying to log into the Ceph Dashboard, itis likely you are using the wrong credentials. Double check your usernameand password, and ensure that your keyboard’s caps lock is not enabled by accident.

  2. If your user credentials are correct, but you are experiencing the sameerror, check that the user account exists:

    cephdashboardac-user-show<username>

    This command returns your user data. If the user does not exist, it willprint:

    ErrorENOENT:User<username>doesnotexist
  3. Check if the user is enabled:

    cephdashboardac-user-show<username>|jq.enabled
    true

    Check ifenabled is set totrue for your user. If not the user isnot enabled, run:

    cephdashboardac-user-enable<username>

Please seeUser and Role Management for more information.

A Dashboard Feature is Not Working

When an error occurs on the backend, you will usually receive an errornotification on the frontend. Run through the following scenarios to debug.

  1. Check the Ceph Dashboard andceph-mgr logfile(s) for any errors. These canfound by searching for keywords, such as500 Internal Server Error,followed bytraceback. The end of a traceback contains more details aboutwhat exact error occurred.

  2. Check your web browser’s JavaScript Console for any errors.

Ceph Dashboard Logs

Dashboard Debug Flag

With this flag enabled, error traceback is included in backend responses.

To enable this flag via the Ceph Dashboard, navigate fromCluster toManagermodules. SelectDashboard module and click the edit button. Click thedebug checkbox and update.

To enable it via the CLI, run the following command:

cephdashboarddebugenable

Setting Logging Level of Dashboard Module

Setting the logging level to debug makes the log more verbose and helpful fordebugging.

  1. Increase the logging level of manager daemons:

    cephtellmgrconfigsetdebug_mgr20
  2. Adjust the logging level of the Ceph Dashboard module via the Dashboard orCLI:

    • Navigate fromCluster toManager modules. SelectDashboard moduleand click the edit button. Modify thelog_level configuration.

    • To adjust it via the CLI, run the following command:

      bin/cephconfigsetmgrmgr/dashboard/log_leveldebug

3. High log levels can result in considerable log volume, which caneasily fill up your filesystem. Set a calendar reminder for an hour, a day,or a week in the future to revert this temporary logging increase. This lookssomething like this:

cephconfiglog
...---11---2020-11-0711:11:11.960659---mgr.x/dashboard/log_level=debug---...
cephconfigreset11

Enable Centralized Logging in Dashboard

To learn more about centralized logging, seeCentralized Logging in Ceph

  1. Create the Loki service on any particular host using “Create Services” option.

  2. Similarly create the Promtail service which will be by default deployedon all the running hosts.

  3. To see debug-level messages as well as info-level events, run the following command via CLI:

    cephconfigsetmgrmgr/cephadm/log_to_cluster_leveldebug
  4. To enable logging to files, run the following commands via CLI:

    cephconfigsetgloballog_to_filetruecephconfigsetglobalmon_cluster_log_to_filetrue
  5. Click on the Daemon Logs tab under Cluster -> Logs.

  6. You can find some pre-defined labels there on clicking the Log browser button such as filename,job etc that can help you query the logs at one go.

  7. You can query the logs with LogQL for advanced search and perform somecalculations as well -https://grafana.com/docs/loki/latest/logql/.

Reporting issues from Dashboard

Ceph-Dashboard provides two ways to create an issue in the Ceph Issue Tracker,either using the Ceph command line interface or by using the Ceph Dashboarduser interface.

To create an issue in the Ceph Issue Tracker, a user needs to have an accounton the issue tracker. Under themyaccount tab in the Ceph Issue Tracker,the user can see their API access key. This key is used for authenticationwhen creating a new issue. To store the Ceph API access key, in the CLI run:

``cephdashboardset-issue-tracker-api-key-i<file-containing-key>``

Then on successful update, you can create an issue using:

``cephdashboardcreateissue<project><tracker_type><subject><description>``

The available projects to create an issue on are:#. dashboard#. block#. object#. file_system#. ceph_manager#. orchestrator#. ceph_volume#. core_ceph

The available tracker types are:#. bug#. feature

The subject and description are then set by the user.

The user can also create an issue using the Dashboard user interface. The settingsicon drop down menu on the top right of the navigation bar has the option toRaiseanissue. On clicking it, a modal dialog opens that has the option toselect the project and tracker from their respective drop down menus. The subjectand multiline description are added by the user. The user can then submit the issue.

Brought to you by the Ceph Foundation

The Ceph Documentation is a community resource funded and hosted by the non-profitCeph Foundation. If you would like to support this and our other efforts, please considerjoining now.