Notice
This document is for a development version of Ceph.
Ceph Dashboard
Overview
The Ceph Dashboard is a web-based Ceph management-and-monitoring tool that canbe used to inspect and administer resources in the cluster. It is implementedas aCeph Manager Daemon module.
The original Ceph Dashboard shipped with Ceph Luminous and was a simpleread-only view into the run-time information and performance data of Cephclusters. It had a simple architecture. However, demand grew for richer,web-based management capabilities for users who prefer a WebUI over the CLI.
TheCeph Dashboard module adds web-based monitoring and administrationto the Ceph Manager. The architecture and functionality of this new module arederived from theopenATTIC Ceph management and monitoring tool. Development was originally driven by the openATTICteam atSUSE, with support from members of the Cephcommunity and from companies includingRed Hat.
The dashboard module’s backend code uses the CherryPy framework, and implementsa custom REST API. The WebUI implementation is based on Angular/TypeScript andincludes both functionality from the original dashboard and new featuresoriginally developed for the standalone version of openATTIC. The CephDashboard module is implemented as an application that provides a graphicalrepresentation of information and statistics through a web server hosted byceph-mgr
.
Feature Overview
The dashboard provides the following features:
Multi-User and Role Management: The dashboard supports multiple useraccounts with different permissions (roles). User accounts and rolescan be managed via both the command line and the WebUI. The dashboardsupports various methods to enhance password security. Passwordcomplexity rules may be configured, requiring users to change their passwordafter the first login or after a configurable time period. SeeUser and Role Management for details.
- Single Sign-On (SSO): The dashboard supports authentication
via an external identity provider using the SAML 2.0 protocol or thse OAuth2 protocol. See:ref:dashboard-saml2-sso-support and :ref:dashboard-oauth2-sso-support for details.
SSL/TLS support: All HTTP communication between the web browser and thedashboard is secured via SSL. A self-signed certificate can be created witha built-in command, but it’s also possible to import custom certificatessigned and issued by a CA. SeeSSL/TLS Support for details.
Auditing: The dashboard backend can be configured to log all
PUT
,POST
andDELETE
API requests in the Ceph audit log. SeeAuditing API Requestsfor instructions on how to enable this feature.Internationalization (I18N): The language used for dashboard text can beselected at run-time.
The Ceph Dashboard offers the following monitoring and management capabilities:
Overall cluster health: Display performance and capacity metrics as wellas cluster status.
Embedded Grafana Dashboards: Ceph DashboardGrafana dashboards may be embedded in external applications and web pagesto surface information and performance metrics gathered bythePrometheus Module module. SeeEnabling the Embedding of Grafana Dashboards for details on how to configure this functionality.
Cluster logs: Display the latest updates to the cluster’s event andaudit log files. Log entries can be filtered by priority, date or keyword.
Hosts: Display a list of all cluster hosts along with theirstorage drives, which services are running, and which version of Ceph isinstalled.
Performance counters: Display detailed service-specific statistics foreach running service.
Monitors: List all Mons, their quorum status, and open sessions.
Monitoring: Enable creation, re-creation, editing, and expiration ofPrometheus’ silences, list the alerting configuration and allconfigured and firing alerts. Show notifications for firing alerts.
Configuration Editor: Display all available configuration options,their descriptions, types, default and currently set values. These may be edited as well.
Pools: List Ceph pools and their details (e.g. applications,pg-autoscaling, placement groups, replication size, EC profile, CRUSHrules, quotas etc.)
OSDs: List OSDs, their status and usage statistics as well asdetailed information like attributes (OSD map), metadata, performancecounters and usage histograms for read/write operations. Mark OSDsup/down/out, purge and reweight OSDs, perform scrub operations, modifyvarious scrub-related configuration options, select profiles toadjust the level of backfilling activity. List all drives associated with anOSD. Set and change the device class of an OSD, display and sort OSDs bydevice class. Deploy OSDs on new drives and hosts.
Device management: List all hosts known by the orchestrator. List alldrives attached to a host and their properties. Display drivehealth predictions and SMART data. Blink enclosure LEDs.
iSCSI: List all hosts that run the TCMU runner service, display allimages and their performance characteristics (read/write ops, traffic).Create, modify, and delete iSCSI targets (via
ceph-iscsi
). Display theiSCSI gateway status and info about active initiators.SeeEnabling iSCSI Management for instructions on how to configurethis feature.RBD: List all RBD images and their properties (size, objects, features).Create, copy, modify and delete RBD images (incl. snapshots) and manage RBDnamespaces. Define various I/O or bandwidth limitation settings on a global,per-pool or per-image level. Create, delete and rollback snapshots of selectedimages, protect/unprotect these snapshots against modification. Copy or clonesnapshots, flatten cloned images.
RBD mirroring: Enable and configure RBD mirroring to a remote Ceph server.List active daemons and their status, pools and RBD images includingsync progress.
CephFS: List active file system clients and associated pools,including usage statistics. Evict active CephFS clients. Manage CephFSquotas and snapshots. Browse a CephFS directory structure.
Object Gateway: List all active object gateways and their performancecounters. Display and manage (add/edit/delete) object gateway users and theirdetails (e.g. quotas) as well as the users’ buckets and their details (e.g.placement targets, owner, quotas, versioning, multi-factor authentication).SeeEnabling the Object Gateway Management Frontend for configuration instructions.
NFS: Manage NFS exports of CephFS file systems and RGW S3 buckets via NFSGanesha. SeeNFS-Ganesha Management for details on how toenable this functionality.
Ceph Manager Modules: Enable and disable Ceph Manager modules, managemodule-specific configuration settings.
Overview of the Dashboard Landing Page
The landing page of Ceph Dashboard serves as the home page and features metricssuch as the overall cluster status, performance, and capacity. It provides real-timeupdates on any changes in the cluster and allows quick access to other sections of the dashboard.

Note
You can change the landing page to the previous version from:Cluster>>ManagerModules>>Dashboard>>Edit
.Editing theFEATURE_TOGGLE_DASHBOARD
option will change the landing page, from one view to another.
Note that the previous version of the landing page will be disabled in future releases.
Details
Provides an overview of the cluster configuration, displaying various critical aspects of the cluster.

Status
Provides a visual indication of cluster health, and displays cluster alerts grouped by severity.

Capacity
Used: Displays the used capacity out of the total physical capacity provided by storage nodes (OSDs)
Warning: Displays thenearfull threshold of the OSDs
Danger: Displays thefull threshold of the OSDs

Inventory
An inventory for all assets within the cluster.Provides direct access to subpages of the dashboard from each item of this card.

Cluster Utilization
Used Capacity: Total capacity used of the cluster. The maximum value of the chart is the maximum capacity of the cluster.
IOPS (Input/Output Operations Per Second): Number of read and write operations.
Latency: Amount of time that it takes to process a read or a write request.
Client Throughput: Amount of data that clients read or write to the cluster.
Recovery Throughput: Amount of recovery data that clients read or write to the cluster.

Supported Browsers
Ceph Dashboard is primarily tested and developed using the following webbrowsers:
Browser | Versions |
---|---|
latest 2 major versions | |
latest 2 major versions | |
latest major version |
While Ceph Dashboard might work in older browsers, we cannot guarantee compatibility andrecommend keeping your browser up to date.
Enabling
If you have installedceph-mgr-dashboard
from distribution packages, thepackage management system should take care of installing all requireddependencies.
If you’re building Ceph from source and want to start the dashboard from yourdevelopment environment, please see the filesREADME.rst
andHACKING.rst
in the source directorysrc/pybind/mgr/dashboard
.
Within a running Ceph cluster, the Ceph Dashboard is enabled with:
cephmgrmoduleenabledashboard
Configuration
SSL/TLS Support
All HTTP connections to the dashboard are secured with SSL/TLS by default.
To get the dashboard up and running quickly, you can generate and install aself-signed certificate:
cephdashboardcreate-self-signed-cert
Note that most web browsers will complain about self-signed certificatesand require explicit confirmation before establishing a secure connection to thedashboard.
To properly secure a deployment and to remove the warning, acertificate that is issued by a certificate authority (CA) should be used.
For example, a key pair can be generated with a command similar to:
opensslreq-new-nodes-x509\-subj"/O=IT/CN=ceph-mgr-dashboard"-days3650\-keyoutdashboard.key-outdashboard.crt-extensionsv3_ca
Thedashboard.crt
file should then be signed by a CA. Once that is done, youcan enable it for Ceph manager instances by running the following commands:
cephdashboardset-ssl-certificate-idashboard.crtcephdashboardset-ssl-certificate-key-idashboard.key
If unique certificates are desired for each manager instance,the name of the instance can be included as follows (where$name
is the nameof theceph-mgr
instance, usually the hostname):
cephdashboardset-ssl-certificate$name-idashboard.crtcephdashboardset-ssl-certificate-key$name-idashboard.key
SSL can also be disabled by setting this configuration value:
cephconfigsetmgrmgr/dashboard/sslfalse
This might be useful if the dashboard will be running behind a proxy which doesnot support SSL for its upstream servers or other situations where SSL is notwanted or required. SeeProxy Configuration for more details.
Warning
Use caution when disabling SSL as usernames and passwords will be sent to thedashboard unencrypted.
Note
You must restart Ceph manager processes after changing the SSLcertificate and key. This can be accomplished by either runningcephmgrfailmgr
or by disabling and re-enabling the dashboard module (which alsotriggers the manager to respawn itself):
cephmgrmoduledisabledashboardcephmgrmoduleenabledashboard
Host Name and Port
Like most web applications, the dashboard binds to a TCP/IP address and TCP port.
By default, theceph-mgr
daemon hosting the dashboard (i.e., the currentlyactive manager) will bind to TCP port 8443 or 8080 when SSL is disabled.
If no specific address has been configured, the web app will bind to::
,which corresponds to all available IPv4 and IPv6 addresses.
These defaults can be changed via the configuration key facility on acluster-wide level (so they apply to all manager instances) as follows:
cephconfigsetmgrmgr/dashboard/server_addr$IPcephconfigsetmgrmgr/dashboard/server_port$PORTcephconfigsetmgrmgr/dashboard/ssl_server_port$PORT
Since eachceph-mgr
hosts its own instance of the dashboard, it may benecessary to configure them separately. The IP address and port for a specificmanager instance can be changed with the following commands:
cephconfigsetmgrmgr/dashboard/$name/server_addr$IPcephconfigsetmgrmgr/dashboard/$name/server_port$PORTcephconfigsetmgrmgr/dashboard/$name/ssl_server_port$PORT
Replace$name
with the ID of the ceph-mgr instance hosting the dashboard.
Note
The commandcephmgrservices
will show you all endpoints that arecurrently configured. Look for thedashboard
key to obtain the URL foraccessing the dashboard.
Username and Password
In order to be able to log in, you need to create a user account and associateit with at least one role. We provide a set of predefinedsystem roles thatyou can use. For more details please refer to theUser and Role Managementsection.
To create a user with the administrator role you can use the followingcommands:
cephdashboardac-user-create<username>-i<file-containing-password>administrator
Account Lock-out
It disables a user account if a user repeatedly enters the wrong credentialsfor multiple times. It is enabled by default to prevent brute-force or dictionaryattacks. The user can get or set the default number of lock-out attempts usingthese commands respectively:
cephdashboardget-account-lockout-attemptscephdashboardset-account-lockout-attempts<value:int>
Warning
This feature can be disabled by setting the default number of lock-out attempts to 0.However, by disabling this feature, the account is more vulnerable to brute-force ordictionary based attacks. This can be disabled by:
cephdashboardset-account-lockout-attempts0
Enable a Locked User
If a user account is disabled as a result of multiple invalid login attempts, thenit needs to be manually enabled by the administrator. This can be done by the followingcommand:
cephdashboardac-user-enable<username>
Accessing the Dashboard
You can now access the dashboard using your (JavaScript-enabled) web browser, bypointing it to any of the host names or IP addresses and the selected TCP portwhere a manager instance is running: e.g.,http(s)://<$IP>:<$PORT>/
.
The dashboard page displays and requests a previously defined username andpassword.
Enabling the Object Gateway Management Frontend
When RGW is deployed with cephadm, the RGW credentials used by thedashboard will be automatically configured. You can also manually force thecredentials to be set up with:
cephdashboardset-rgw-credentials
This will create an RGW user with uiddashboard
for each realm inthe system.
If you’ve configured a custom ‘admin’ resource in your RGW admin API, you should set it here also:
cephdashboardset-rgw-api-admin-resource<admin_resource>
If you are using a self-signed certificate in your Object Gateway setup,you should disable certificate verification in the dashboard to avoid refusedconnections, e.g. caused by certificates signed by unknown CA or not matchingthe host name:
cephdashboardset-rgw-api-ssl-verifyFalse
To set a custom hostname or address for an RGW gateway, set the value ofRGW_HOSTNAME_PER_DAEMON
accordingly:
cephdashboardset-rgw-hostname<gateway_name><hostname>
The setting can be unset using:
cephdashboardunset-rgw-hostname<gateway_name>
If the Object Gateway takes too long to process requests and the dashboard runsinto timeouts, you can set the timeout value to your needs:
cephdashboardset-rest-requests-timeout<seconds>
The default value is 45 seconds.
Enabling iSCSI Management
The Ceph Dashboard can manage iSCSI targets using the REST API provided by therbd-target-api
service of theCeph iSCSI Gateway. Please make sure that it isinstalled and enabled on the iSCSI gateways.
Note
The iSCSI management functionality of Ceph Dashboard depends on the latestversion 3 of theceph-iscsi project.Make sure that your operating system provides the correct version, otherwisethe dashboard will not enable the management features.
If theceph-iscsi
REST API is configured in HTTPS mode and its using a self-signedcertificate, you need to configure the dashboard to avoid SSL certificateverification when accessing ceph-iscsi API.
To disable API SSL verification run the following command:
cephdashboardset-iscsi-api-ssl-verificationfalse
The available iSCSI gateways must be defined using the following commands:
cephdashboardiscsi-gateway-list# Gateway URL format for a new gateway: <scheme>://<username>:<password>@<host>[:port]cephdashboardiscsi-gateway-add-i<file-containing-gateway-url>[<gateway_name>]cephdashboardiscsi-gateway-rm<gateway_name>
Enabling the Embedding of Grafana Dashboards
Grafana pulls data fromPrometheus. AlthoughGrafana can use other data sources, the Grafana dashboards we provide containqueries that are specific to Prometheus. Our Grafana dashboards thereforerequire Prometheus as the data source. The CephPrometheus Modulemodule exports its data in the Prometheus exposition format. These Grafanadashboards rely on metric names from the Prometheus module andNode exporter. The Node exporter is aseparate application that provides machine metrics.
Note
Prometheus’ security model presumes that untrusted users have access to thePrometheus HTTP endpoint and logs. Untrusted users have access to all the(meta)data Prometheus collects that is contained in the database, plus avariety of operational and debugging information.
However, Prometheus’ HTTP API is limited to read-only operations.Configurations cannot be changed using the API and secrets are notexposed. Moreover, Prometheus has some built-in measures to mitigate theimpact of denial of service attacks.
Please seePrometheus’ Security model<https://prometheus.io/docs/operating/security/> for more detailedinformation.
Installation and Configuration using cephadm
Grafana and Prometheus can be installed usingCephadm. They willautomatically be configured bycephadm
. Please seeMonitoring Services documentation for more details on how to usecephadm
for installing and configuring Prometheus and Grafana.
Manual Installation and Configuration
The following process describes how to configure Grafana and Prometheusmanually. After you have installed Prometheus, Grafana, and the Node exporteron appropriate hosts, proceed with the following steps.
Enable the Ceph Exporter which comes as Ceph Manager module by running:
cephmgrmoduleenableprometheus
More details can be found in the documentation of thePrometheus Module.
Add the corresponding scrape configuration to Prometheus. This may looklike:
global:scrape_interval:5sscrape_configs:-job_name:'prometheus'static_configs:-targets:['localhost:9090']-job_name:'ceph'static_configs:-targets:['localhost:9283']-job_name:'node-exporter'static_configs:-targets:['localhost:9100']
Note
Please note that in the above example, Prometheus is configuredto scrape data from itself (port 9090), the Ceph manager moduleprometheus (port 9283), which exports Ceph internal data, and the NodeExporter (port 9100), which provides OS and hardware metrics for each host.
Depending on your configuration, you may need to change the hostname inor add additional configuration entries for the NodeExporter. It is unlikely that you will need to change the default TCP ports.
Moreover, you don’tneed to have more than one target for Ceph specificdata, provided by theprometheus mgr module. But it is recommended toconfigure Prometheus to scrape Ceph specific data from all existing Cephmanagers. This enables a built-in high availability mechanism, so thatservices run on a manager host will be restarted automatically on a differentmanager host if one Ceph Manager goes down.
Add Prometheus as data source to Grafanausing the Grafana Web UI.
Important
The data source must be named “Dashboard1”.
Install thevonage-status-panel and grafana-piechart-panel plugins using:
grafana-clipluginsinstallvonage-status-panelgrafana-clipluginsinstallgrafana-piechart-panel
Add Dashboards to Grafana:
Dashboards can be added to Grafana by importing dashboard JSON files.Use the following command to download the JSON files:
wgethttps://raw.githubusercontent.com/ceph/ceph/main/monitoring/ceph-mixin/dashboards_out/<Dashboard-name>.json
You can find various dashboard JSON fileshere.
For Example, for ceph-cluster overview you can use:
wgethttps://raw.githubusercontent.com/ceph/ceph/main/monitoring/ceph-mixin/dashboards_out/ceph-cluster.json
You may also author your own dashboards.
Configure anonymous mode in
/etc/grafana/grafana.ini
:[auth.anonymous]enabled=trueorg_name=MainOrg.org_role=Viewer
In newer versions of Grafana (starting with 6.2.0-beta1) a new setting named
allow_embedding
has been introduced. This setting must be explicitlyset totrue
for the Grafana integration in Ceph Dashboard to work, as thedefault isfalse
.[security]allow_embedding=true
Enabling RBD-Image monitoring
Monitoring of RBD images is disabled by default, as it can significantly impactperformance. For more information please seeCeph Health Checks.When disabled, the overview and details dashboards will be empty in Grafana andmetrics will not be visible in Prometheus.
Configuring Dashboard
After you have set up Grafana and Prometheus, you will need to configure theconnection information that the Ceph Dashboard will use to access Grafana.
You need to tell the dashboard on which URL the Grafana instance isrunning/deployed:
cephdashboardset-grafana-api-url<grafana-server-url># default: ''
The format of url is :<protocol>:<IP-address>:<port>
Note
The Ceph Dashboard embeds Grafana dashboards viaiframe
HTML elements.If Grafana is configured without SSL/TLS support, most browsers will block theembedding of insecure content if SSL support isenabled for the dashboard (which is the default). If youcan’t see the embedded Grafana dashboards after enabling them as outlinedabove, check your browser’s documentation on how to unblock mixed content.Alternatively, consider enabling SSL/TLS support in Grafana.
If you are using a self-signed certificate for Grafana,disable certificate verification in the dashboard to avoid refused connections,which can be a result of certificates signed by an unknown CA or that do notmatch the host name:
cephdashboardset-grafana-api-ssl-verifyFalse
You can also access Grafana directly to monitor your cluster.
Note
Ceph Dashboard configuration information can also be unset. For example, toclear the Grafana API URL we configured above:
cephdashboardreset-grafana-api-url
Alternative URL for Browsers
The Ceph Dashboard backend requires the Grafana URL to be able to verify theexistence of Grafana Dashboards before the frontend even loads them. Due to thenature of how Grafana is implemented in Ceph Dashboard, this means that twoworking connections are required in order to be able to see Grafana graphs inCeph Dashboard:
The backend (Ceph Mgr module) needs to verify the existence of the requestedgraph. If this request succeeds, it lets the frontend know that it can safelyaccess Grafana.
The frontend then requests the Grafana graphs directly from the user’sbrowser using an iframe. The Grafana instance is accessed directly without anydetour through Ceph Dashboard.
Now, it might be the case that your environment makes it difficult for theuser’s browser to directly access the URL configured in Ceph Dashboard. To solvethis issue, a separate URL can be configured which will solely be used to tellthe frontend (the user’s browser) which URL it should use to access Grafana.This setting won’t ever be changed automatically, unlike the GRAFANA_API_URLwhich is set byCephadm (only if cephadm is used to deploy monitoringservices).
To change the URL that is returned to the frontend issue the following command:
cephdashboardset-grafana-frontend-api-url<grafana-server-url>
If no value is set for that option, it will simply fall back to the value of theGRAFANA_API_URL option. If set, it will instruct the browser to use this URL toaccess Grafana.
Enabling SAML2 Single Sign-On (SSO)
The Ceph Dashboard supports external authentication of users via theSAML 2.0 protocol. You need tofirst create user accounts and associate them with desired roles, asauthorization is performed by the Dashboard. However, the authenticationprocess can be performed by an existing Identity Provider (IdP).
Note
Ceph Dashboard SSO support relies on onelogin’spython-saml library.Please ensure that this library is installed on your system, either by usingyour distribution’s package management or via Python’spip installer.
To configure SSO on Ceph Dashboard, you should use the following command:
cephdashboardssosetupsaml2<ceph_dashboard_base_url><idp_metadata>{<idp_username_attribute>}{<idp_entity_id>}{<sp_x_509_cert>}{<sp_private_key>}
Parameters:
<ceph_dashboard_base_url>: Base URL where Ceph Dashboard is accessible (e.g.,https://cephdashboard.local)
<idp_metadata>: URL to remote (http://,https://) or local (file://) path or content of the IdP metadata XML (e.g.,https://myidp/metadata,file:///home/myuser/metadata.xml).
<idp_username_attribute>(optional): Attribute that should be used to get the username from the authentication response. Defaults touid.
<idp_entity_id>(optional): Use this when more than one entity id exists on the IdP metadata.
<sp_x_509_cert> / <sp_private_key>(optional): File path of the certificate that should be used by Ceph Dashboard (Service Provider) for signing and encryption (these file paths should be accessible from the active ceph-mgr instance).
Note
The issuer value of SAML requests will follow this pattern:<ceph_dashboard_base_url>/auth/saml2/metadata
To display the current SAML 2.0 configuration, use the following command:
cephdashboardssoshowsaml2
Note
For more information aboutonelogin_settings, please check theonelogin documentation.
To disable SSO:
cephdashboardssodisable
To check if SSO is enabled:
cephdashboardssostatus
To enable SSO:
cephdashboardssoenablesaml2
Enabling OAuth2 Single Sign-On (SSO)
The Ceph Dashboard supports external authentication of users via theOAuth protocol. You need tohaveCephadm enabled as your orchestrator with an activeManagement Gateway andOAuth2 Proxy services.
From the IDP of choice, Keycloak is the current recomendation and tested solution,configure the IDP’s client used in theoauth2-proxy service configuration to validate the following redirect URLsfor login_url:https://<host_name>|<IP_address>/oauth2/callback andthe following logout_url:https://<host_name>|<IP_address>/ /oauth2/sign_out
Again, from the IDP, we will need a user with a valid role, this user will be the one to performauthorization against, we can create a role like: ‘administator’ to give admin level access to the user.
Make certain that theenable_auth
flag has been included in thecephorchapplymgmt-gateway
command and that it has been set totrue
by running acommand of the following form:
cephorchapplymgmt-gateway--enable_auth=true--placement=<ceph-node-02>
To disable SSO:
cephdashboardssodisable
To check if SSO is enabled:
cephdashboardssostatus
To enable SSO:
cephdashboardssoenableoauth2
Enabling Prometheus Alerting
To use Prometheus for alerting you must definealerting rules.These are managed by theAlertmanager.If you are not yet using the Alertmanager,install it as it receivesand manages alerts from Prometheus.
Alertmanager capabilities can be consumed by the dashboard in three differentways:
Use the notification receiver of the dashboard.
Use the Prometheus Alertmanager API.
Use both sources simultaneously.
All three methods notify you about alerts. You won’t be notifiedtwice if you use both sources, but you need to consume at least the Alertmanager APIin order to manage silences.
Use the notification receiver of the dashboard
This allows you to get notifications asconfigured from the Alertmanager.You will get notified inside the dashboard once a notification is send out,but you are not able to manage alerts.
Add the dashboard receiver and the new route to your Alertmanagerconfiguration. This should look like:
route:receiver:'ceph-dashboard'...receivers:-name:'ceph-dashboard'webhook_configs:-url:'<url-to-dashboard>/api/prometheus_receiver'Ensure that the Alertmanager considers your SSL certificate in termsof the dashboard as valid. For more information about the correctconfiguration checkout the<http_config> documentation.
Use the API of Prometheus and the Alertmanager
This allows you to manage alerts and silences and will enable the “ActiveAlerts”, “All Alerts” as well as the “Silences” tabs in the “Monitoring”section of the “Cluster” menu entry.
Alerts can be sorted by name, job, severity, state and start time.Unfortunately it’s not possible to know when an alert was sent out through anotification by the Alertmanager based on your configuration, that’s why thedashboard will notify the user on any visible change to an alert and willnotify the changed alert.
Silences can be sorted by id, creator, status, start, updated and end time.Silences can be created in various ways, it’s also possible to expire them.
Create from scratch
Based on a selected alert
Recreate from expired silence
Update a silence (which will recreate and expire it (default Alertmanager behaviour))
To use it, specify the host and port of the Alertmanager server:
cephdashboardset-alertmanager-api-host<alertmanager-host:port># default: ''
For example:
cephdashboardset-alertmanager-api-host'http://localhost:9093'
To be able to see all configured alerts, you will need to configure the URL tothe Prometheus API. Using this API, the UI will also help you in verifyingthat a new silence will match a corresponding alert.
cephdashboardset-prometheus-api-host<prometheus-host:port># default: ''
For example:
cephdashboardset-prometheus-api-host'http://localhost:9090'
After setting up the hosts, refresh your browser’s dashboard window or tab.
Use both methods
The behaviors of both methods are configured in a way that theyshould not disturb each other, through annoying duplicated notificationsmay pop up.
If you are using a self-signed certificate in your Prometheus or yourAlertmanager setup, you should disable certificate verification in thedashboard to avoid refused connections caused by certificates signed byan unknown CA or that do not match the host name.
For Prometheus:
cephdashboardset-prometheus-api-ssl-verifyFalse
For Alertmanager:
cephdashboardset-alertmanager-api-ssl-verifyFalse
User and Role Management
Password Policy
By default the password policy feature is enabled, which includes thefollowing checks:
Is the password longer than N characters?
Are the old and new password the same?
The password policy feature can be switched on or off completely:
cephdashboardset-pwd-policy-enabled<true|false>
The following individual checks can also be switched on or off:
cephdashboardset-pwd-policy-check-length-enabled<true|false>cephdashboardset-pwd-policy-check-oldpwd-enabled<true|false>cephdashboardset-pwd-policy-check-username-enabled<true|false>cephdashboardset-pwd-policy-check-exclusion-list-enabled<true|false>cephdashboardset-pwd-policy-check-complexity-enabled<true|false>cephdashboardset-pwd-policy-check-sequential-chars-enabled<true|false>cephdashboardset-pwd-policy-check-repetitive-chars-enabled<true|false>
Additionally the following options are available to configure passwordpolicy.
Minimum password length (defaults to 8):
cephdashboardset-pwd-policy-min-length<N>
Minimum password complexity (defaults to 10):
cephdashboardset-pwd-policy-min-complexity<N>
Password complexity is calculated by classifying each character inthe password. The complexity count starts by 0. A character is rated bythe following rules in the given order.
Increase by 1 if the character is a digit.
Increase by 1 if the character is a lower case ASCII character.
Increase by 2 if the character is an upper case ASCII character.
Increase by 3 if the character is a special character like
!"#$%&'()*+,-./:;<=>?@[\]^_`{|}~
.Increase by 5 if the character has not been classified by one of the previous rules.
A list of comma separated words that are not allowed to be used in apassword:
cephdashboardset-pwd-policy-exclusion-list<word>[,...]
User Accounts
The Ceph Dashboard supports multiple user accounts. Each user accountconsists of a username, a password (stored in encrypted form usingbcrypt
),an optional name, and an optional email address.
If a new user is created via the Web UI, it is possible to set an option that theuser must assign a new password when they log in for the first time.
User accounts are stored in the monitors’ configuration database, and areavailable to allceph-mgr
instances.
We provide a set of CLI commands to manage user accounts:
Show User(s):
cephdashboardac-user-show[<username>]
Create User:
cephdashboardac-user-create[--enabled][--force-password][--pwd_update_required]<username>-i<file-containing-password>[<rolename>][<name>][<email>][<pwd_expiration_date>]
To bypass password policy checks use theforce-password option.Add the optionpwd_update_required so that a newly created user hasto change their password after the first login.
Delete User:
cephdashboardac-user-delete<username>
Change Password:
cephdashboardac-user-set-password[--force-password]<username>-i<file-containing-password>
Change Password Hash:
cephdashboardac-user-set-password-hash<username>-i<file-containing-password-hash>
The hash must be a bcrypt hash and salt, e.g.
$2b$12$Pt3Vq/rDt2y9glTPSV.VFegiLkQeIpddtkhoFetNApYmIJOY8gau2
.This can be used to import users from an external database.Modify User (name, and email):
cephdashboardac-user-set-info<username><name><email>
Disable User:
cephdashboardac-user-disable<username>
Enable User:
cephdashboardac-user-enable<username>
User Roles and Permissions
User accounts are associated with a set of roles that define whichdashboard functionality can be accessed.
The Dashboard functionality/modules are grouped within asecurity scope.Security scopes are predefined and static. The current available securityscopes are:
hosts: includes all features related to the
Hosts
menuentry.config-opt: includes all features related to management of Cephconfiguration options.
pool: includes all features related to pool management.
osd: includes all features related to OSD management.
monitor: includes all features related to monitor management.
rbd-image: includes all features related to RBD imagemanagement.
rbd-mirroring: includes all features related to RBD mirroringmanagement.
iscsi: includes all features related to iSCSI management.
rgw: includes all features related to RADOS Gateway (RGW) management.
cephfs: includes all features related to CephFS management.
nfs-ganesha: includes all features related to NFS Ganesha management.
manager: include all features related to Ceph Managermanagement.
log: include all features related to Ceph logs management.
grafana: include all features related to Grafana proxy.
prometheus: include all features related to Prometheus alert management.
dashboard-settings: allows to change dashboard settings.
Arole specifies a set of mappings between asecurity scope and a set ofpermissions. There are four types of permissions:
read
create
update
delete
See below for an example of a role specification, in the form of a Python dictionary:
# example of a role{'role':'my_new_role','description':'My new role','scopes_permissions':{'pool':['read','create'],'rbd-image':['read','create','update','delete']}}
The above role dictates that a user hasread andcreate permissions forfeatures related to pool management, and has full permissions forfeatures related to RBD image management.
The Dashboard provides a set of predefined roles that we callsystem roles, which can be used right away by a fresh Ceph Dashboardinstallation.
The list of system roles are:
administrator: allows full permissions for all security scopes.
read-only: allowsread permission for all security scopes exceptdashboard settings.
block-manager: allows full permissions forrbd-image,rbd-mirroring, andiscsi scopes.
rgw-manager: allows full permissions for thergw scope
cluster-manager: allows full permissions for thehosts,osd,monitor,manager, andconfig-opt scopes.
pool-manager: allows full permissions for thepool scope.
cephfs-manager: allows full permissions for thecephfs scope.
The list of available roles can be retrieved with the following command:
cephdashboardac-role-show[<rolename>]
You can also use the CLI to create new roles. The available commands are thefollowing:
Create Role:
cephdashboardac-role-create<rolename>[<description>]
Delete Role:
cephdashboardac-role-delete<rolename>
Add Scope Permissions to Role:
cephdashboardac-role-add-scope-perms<rolename><scopename><permission>[<permission>...]
Delete Scope Permission from Role:
cephdashboardac-role-del-scope-perms<rolename><scopename>
To assign roles to users, the following commands are available:
Set User Roles:
cephdashboardac-user-set-roles<username><rolename>[<rolename>...]
Add Roles To User:
cephdashboardac-user-add-roles<username><rolename>[<rolename>...]
Delete Roles from User:
cephdashboardac-user-del-roles<username><rolename>[<rolename>...]
Example of User and Custom Role Creation
In this section we show a complete example of the commands thatcreate a user account that can manage RBD images, view and create Ceph pools,and has read-only access to other scopes.
Create the user:
cephdashboardac-user-createbob-i<file-containing-password>
Create role and specify scope permissions:
cephdashboardac-role-createrbd/pool-managercephdashboardac-role-add-scope-permsrbd/pool-managerrbd-imagereadcreateupdatedeletecephdashboardac-role-add-scope-permsrbd/pool-managerpoolreadcreate
Associate roles to user:
cephdashboardac-user-set-rolesbobrbd/pool-managerread-only
Proxy Configuration
In a Ceph cluster with multipleceph-mgr
instances, only the dashboardrunning on the currently activeceph-mgr
daemon will serve incoming requests.Connections to the dashboard’s TCP port on standbyceph-mgr
instanceswill receive an HTTP redirect (303) to the active manager’s dashboard URL.This enables you to point your browser to anyceph-mgr
instance inorder to access the dashboard.
If you want to establish a fixed URL to reach the dashboard or if you don’t wantto allow direct connections to the manager nodes, you could set up a proxy thatautomatically forwards incoming requests to the activeceph-mgr
instance.
Configuring a URL Prefix
If you are accessing the dashboard via a reverse proxy,you may wish to service it under a URL prefix. To get the dashboardto use hyperlinks that include your prefix, you can set theurl_prefix
setting:
cephconfigsetmgrmgr/dashboard/url_prefix$PREFIX
so you can access the dashboard athttp://$IP:$PORT/$PREFIX/
.
Disable the redirection
If the dashboard is behind a load-balancing proxy likeHAProxyyou might want to disable redirection to prevent situations in whichinternal (unresolvable) URLs are published to the frontend client. Use thefollowing command to get the dashboard to respond with an HTTP error (500 by default)instead of redirecting to the active dashboard:
cephconfigsetmgrmgr/dashboard/standby_behaviour"error"
To reset the setting to default redirection, use the following command:
cephconfigsetmgrmgr/dashboard/standby_behaviour"redirect"
Configure the error status code
When redirection is disabled, you may want to customize the HTTP statuscode of standby dashboards. To do so you need to run the command:
cephconfigsetmgrmgr/dashboard/standby_error_status_code503
Resolve IP address to hostname before redirect
Redirection from a standby dashboard to the active dashboard is done via themanager’s IP address, not via the manager’s hostname. In virtualizedenvironments, IP-address-based redirection reduces the incidence of error ascompared to hostname-based resolution. Because of the increased risk of errordue to hostname-based resolution, the option for hostname resolution isdisabled by default.
However, in some situations it might be helpful to redirect via the hostname.For example, if the configured TLS certificate matches only the hostnames andnot the IP addresses of those hosts, hostname redirection would be preferable.
To activate redirection from standby dashboards to active dashboards via themanager’s hostname, run the following command:
cephconfigsetmgrmgr/dashboard/redirect_resolve_ip_addrTrue
Disable hostname redirection by running the following command:
cephconfigsetmgrmgr/dashboard/redirect_resolve_ip_addrFalse
Warning
If you attempt to activate redirection by using the command above and youget the error messageEINVAL:unrecognizedconfigoption'mgr/dashboard/redirect_resolve_ip_addr'
, then you might be running arelease of Ceph prior to version 17.2.6. This feature was introduced in17.2.6, in this commit:https://github.com/ceph/ceph/pull/48219.
HAProxy example configuration
Below you will find an example configuration for SSL/TLS passthrough usingHAProxy.
Please note that this configuration works under the following conditions.If the dashboard fails over, the front-end client might receive a HTTP redirect(303) response and will be redirected to an unresolvable host. This happens whenfailover occurs between two HAProxy health checks. In this situation thepreviously active dashboard node will now respond with a 303 which points tothe new active node. To prevent that situation you should consider disablingredirection on standby nodes.
defaults log global option log-health-checks timeout connect 5s timeout client 50s timeout server 450sfrontend dashboard_front mode http bind *:80 option httplog redirect scheme https code 301 if !{ ssl_fc }frontend dashboard_front_ssl mode tcp bind *:443 option tcplog default_backend dashboard_back_sslbackend dashboard_back_ssl mode tcp option httpchk GET / http-check expect status 200 server x <HOST>:<PORT> check check-ssl verify none server y <HOST>:<PORT> check check-ssl verify none server z <HOST>:<PORT> check check-ssl verify none
Auditing API Requests
The REST API can log PUT, POST and DELETE requests to the Cephaudit log. This feature is disabled by default, but can be enabled with thefollowing command:
cephdashboardset-audit-api-enabled<true|false>
If enabled, the following parameters are logged per each request:
from - The origin of the request, e.g.https://[::1]:44410
path - The REST API path, e.g. /api/auth
method - e.g. PUT, POST or DELETE
user - The name of the user, otherwise ‘None’
The logging of the request payload (the arguments and their values) is enabledby default. Execute the following command to disable this behaviour:
cephdashboardset-audit-api-log-payload<true|false>
A log entry may look like this:
2018-10-2215:27:01.302514mgr.x[INF][DASHBOARD]from='https://[::ffff:127.0.0.1]:37022'path='/api/rgw/user/klaus'method='PUT'user='admin'params='{"max_buckets": "1000", "display_name": "Klaus Mustermann", "uid": "klaus", "suspended": "0", "email": "klaus.mustermann@ceph.com"}'
NFS-Ganesha Management
The dashboard requires enabling the NFS module which will be used to manageNFS clusters and NFS exports. For more information checkCephFS & RGW Exports over NFS.
Plug-ins
Plug-ins extend the functionality of the Ceph Dashboard in a modularand loosely coupled fashion.
Feature Toggles
This plug-in allows to enable or disable some features from the Ceph Dashboardon-demand. When a feature becomes disabled:
Its front-end elements (web pages, menu entries, charts, etc.) will becomehidden.
Its associated REST API endpoints will reject any further requests (404, NotFound Error).
The main purpose of this plug-in is to allow ad hoc customizations of theworkflows exposed by the dashboard. Additionally, it could allow for dynamicallyenabling experimental features with minimal configuration burden and no serviceimpact.
The list of features that can be enabled/disabled is:
- Block (RBD):
Image Management:
rbd
Mirroring:
mirroring
iSCSI:
iscsi
Filesystem (Cephfs):
cephfs
Objects (RGW):
rgw
(including daemon, user and bucket management).NFS:
nfs-ganesha
exports.
By default all features come enabled.
To retrieve a list of features and their current statuses:
cephdashboardfeaturestatus
Feature'cephfs':'enabled'Feature'iscsi':'enabled'Feature'nvmeof':'enabled'Feature'mirroring':'enabled'Feature'rbd':'enabled'Feature'rgw':'enabled'Feature'nfs':'enabled'
To enable or disable the status of a single or multiple features:
cephdashboardfeaturedisableiscsimirroring
Feature'iscsi':disabledFeature'mirroring':disabled
After a feature status has changed, the API REST endpoints immediately respondto that change, but it may take up to twenty (20) seconds for the front-end UIelements seconds to reflect the change.
Debug
This plugin allows to customize the behaviour of the dashboard according to thedebug mode. It can be enabled, disabled or checked with the following command:
cephdashboarddebugstatus
Debug:'disabled'
cephdashboarddebugenable
Debug:'enabled'
cephdashboarddebugdisable
Debug:'disabled'
By default, it’s disabled. This is the recommended setting for productiondeployments. If required, debug mode can be enabled without need of restarting.Currently, disabled debug mode equals to CherryPyproduction
environment,while when enabled, it usestest_suite
defaults (please refer toCherryPy Environments for moredetails).
It also adds request uuid (unique_id
) to Cherrypy on versions that don’tsupport this. It additionally prints theunique_id
to error responses andlog messages.
Message of the day (MOTD)
Displays a configured “message of the day” (MOTD) at the top of the CephDashboard.
The importance of an MOTD can be configured by its severity, which isinfo
,warning
ordanger
. The MOTD can expire after a given time,this means it will not be displayed in the UI anymore. Use the followingsyntax to specify the expiration time:Ns|m|h|d|w
, whereN
is aninteger followed by seconds, minutes, hours, days and weeks. If the MOTDshould expire after 2 hours, use2h
or5w
for 5 weeks. Use0
toconfigure a MOTD that does not expire.
To configure a MOTD, run the following command:
cephdashboardmotdset<severity:info|warning|danger><expires><message>
To show the configured MOTD:
cephdashboardmotdget
To clear the configured MOTD run:
cephdashboardmotdclear
A MOTD with a severity ofinfo
orwarning
can be closed by the user.Theinfo
MOTD is not displayed until the local storage cookies arecleared or a new MOTD with a different severity is displayed. A MOTD with awarning
severity will be displayed again in a new session.
Troubleshooting the Dashboard
Locating the Dashboard
If you are unsure of the location of the Ceph Dashboard, run the following command:
cephmgrservices|jq.dashboard
"https://host:port"
The command returns the URL where the Ceph Dashboard is located:https://<host>:<port>/
Note
Many Ceph tools return results in JSON format. We suggest thatyou install thejq command-lineutility to facilitate working with JSON data.
Accessing the Dashboard
If you are unable to access the Ceph Dashboard, run the followingcommands:
Verify the Ceph Dashboard module is enabled:
cephmgrmodulels|jq.enabled_modules
Ensure the Ceph Dashboard module is listed in the return value of thecommand. Example snipped output from the command above:
["dashboard","iostat","restful"]
If it is not listed, activate the module with the following command:
cephmgrmoduleenabledashboard
Check the Ceph Dashboard and/or
ceph-mgr
log files for any errors.Check if
ceph-mgr
log messages are written to a file by:cephconfiggetmgrlog_to_file
true
Get the location of the log file (it’s
/var/log/ceph/<cluster-name>-<daemon-name>.log
by default):cephconfiggetmgrlog_file
/var/log/ceph/$cluster-$name.log
Ensure the SSL/TLS support is configured properly:
Check if the SSL/TLS support is enabled:
cephconfiggetmgrmgr/dashboard/ssl
If the command returns
true
, verify a certificate exists by:cephconfig-keygetmgr/dashboard/crt
and:
cephconfig-keygetmgr/dashboard/key
If it doesn’t return
true
, run the following command to generate a self-signedcertificate or follow the instructions outlined inSSL/TLS Support:cephdashboardcreate-self-signed-cert
Trouble Logging into the Dashboard
If you are unable to log into the Ceph Dashboard and you receive the followingerror, run through the procedural checks below:

Check that your user credentials are correct. If you are seeing thenotification message above when trying to log into the Ceph Dashboard, itis likely you are using the wrong credentials. Double check your usernameand password, and ensure that your keyboard’s caps lock is not enabled by accident.
If your user credentials are correct, but you are experiencing the sameerror, check that the user account exists:
cephdashboardac-user-show<username>
This command returns your user data. If the user does not exist, it willprint:
ErrorENOENT:User<username>doesnotexist
Check if the user is enabled:
cephdashboardac-user-show<username>|jq.enabled
true
Check if
enabled
is set totrue
for your user. If not the user isnot enabled, run:cephdashboardac-user-enable<username>
Please seeUser and Role Management for more information.
A Dashboard Feature is Not Working
When an error occurs on the backend, you will usually receive an errornotification on the frontend. Run through the following scenarios to debug.
Check the Ceph Dashboard and
ceph-mgr
logfile(s) for any errors. These canfound by searching for keywords, such as500 Internal Server Error,followed bytraceback
. The end of a traceback contains more details aboutwhat exact error occurred.Check your web browser’s JavaScript Console for any errors.
Ceph Dashboard Logs
Dashboard Debug Flag
With this flag enabled, error traceback is included in backend responses.
To enable this flag via the Ceph Dashboard, navigate fromCluster toManagermodules. SelectDashboard module and click the edit button. Click thedebug checkbox and update.
To enable it via the CLI, run the following command:
cephdashboarddebugenable
Setting Logging Level of Dashboard Module
Setting the logging level to debug makes the log more verbose and helpful fordebugging.
Increase the logging level of manager daemons:
cephtellmgrconfigsetdebug_mgr20
Adjust the logging level of the Ceph Dashboard module via the Dashboard orCLI:
Navigate fromCluster toManager modules. SelectDashboard moduleand click the edit button. Modify the
log_level
configuration.To adjust it via the CLI, run the following command:
bin/cephconfigsetmgrmgr/dashboard/log_leveldebug
3. High log levels can result in considerable log volume, which caneasily fill up your filesystem. Set a calendar reminder for an hour, a day,or a week in the future to revert this temporary logging increase. This lookssomething like this:
cephconfiglog
...---11---2020-11-0711:11:11.960659---mgr.x/dashboard/log_level=debug---...cephconfigreset11
Enable Centralized Logging in Dashboard
To learn more about centralized logging, seeCentralized Logging in Ceph
Create the Loki service on any particular host using “Create Services” option.
Similarly create the Promtail service which will be by default deployedon all the running hosts.
To see debug-level messages as well as info-level events, run the following command via CLI:
cephconfigsetmgrmgr/cephadm/log_to_cluster_leveldebug
To enable logging to files, run the following commands via CLI:
cephconfigsetgloballog_to_filetruecephconfigsetglobalmon_cluster_log_to_filetrue
Click on the Daemon Logs tab under Cluster -> Logs.
You can find some pre-defined labels there on clicking the Log browser button such as filename,job etc that can help you query the logs at one go.
You can query the logs with LogQL for advanced search and perform somecalculations as well -https://grafana.com/docs/loki/latest/logql/.
Reporting issues from Dashboard
Ceph-Dashboard provides two ways to create an issue in the Ceph Issue Tracker,either using the Ceph command line interface or by using the Ceph Dashboarduser interface.
To create an issue in the Ceph Issue Tracker, a user needs to have an accounton the issue tracker. Under themyaccount
tab in the Ceph Issue Tracker,the user can see their API access key. This key is used for authenticationwhen creating a new issue. To store the Ceph API access key, in the CLI run:
``cephdashboardset-issue-tracker-api-key-i<file-containing-key>``
Then on successful update, you can create an issue using:
``cephdashboardcreateissue<project><tracker_type><subject><description>``
The available projects to create an issue on are:#. dashboard#. block#. object#. file_system#. ceph_manager#. orchestrator#. ceph_volume#. core_ceph
The available tracker types are:#. bug#. feature
The subject and description are then set by the user.
The user can also create an issue using the Dashboard user interface. The settingsicon drop down menu on the top right of the navigation bar has the option toRaiseanissue
. On clicking it, a modal dialog opens that has the option toselect the project and tracker from their respective drop down menus. The subjectand multiline description are added by the user. The user can then submit the issue.
Brought to you by the Ceph Foundation
The Ceph Documentation is a community resource funded and hosted by the non-profitCeph Foundation. If you would like to support this and our other efforts, please considerjoining now.