Repository files navigation

Monocle is designed for development teams to provide:

• analytics on project changes
• boards to display changes by criteria

It helps teams and individual to better organize daily dutiesand to detect anomalies in the way changes are produced and reviewed.

Monocle supports GitHub, GitLab and Gerrit.

How to get started with Monocle:

• Explore the website and blog:changemetrics.io
• Try on the demo instance:demo.changemetrics.io
• Chat with us in the project Matrix room:#monocle:matrix.org
• Ask and Discuss about Monocle on theDiscussions page
• Run your own instance:Read the installation guide
• Hack on it:Read the contributing guide
• Checkoutwho use Monocle

The activity view:

The developer board:

The peers strength view:

The process below describes how to index changes from GitHub repositoriesand then how to start the web UI to browse metrics.

The deployment is based on Docker via aDocker composedefinition.This compose definition can also be used byPodman compose, then alldocker-compose commands from this documentation could be replaced bypodman-compose.

Alternatively, the following deployment method are supported:

• deploy on Kubernetes
• deploy from source code

git clone https://github.com/change-metrics/monocle.gitcd monocle# Init a .secrets file with a default API key for the crawler processecho CRAWLERS_API_KEY=$(uuidgen)> .secrets

Theconfig.yaml file is used by the crawler and api services.

To crawl GitHub public repositories, you must generate a personal accesstoken on GitHub (w/o any specific rights) athttps://github.com/settings/tokens.

Then create the config fileetc/config.yaml. Here is an example your could start with.Make sure to writeGITHUB_TOKEN=<github_token> in the.secrets file:

---workspaces:  -name:monoclecrawlers:      -name:github-tektoncdprovider:github_organization:tektoncdgithub_repositories:            -operator            -pipelineupdate_since:'2021-01-01'

To crawl the full tektoncd GitHub organization then remove thegithub_repositories entry from the file.Check the sectionWorkspaces for a complete description ofthe configuration.

The docker-compose file is set to use the container image of the latest stablerelease of Monocle (1.11.2). It is adviced to use the latest stable Monocle version.However, as our CI publishes the latest (devel) container image then it is possibleto run the very last version. To do so setCOMPOSE_MONOCLE_VERSION tolatest inthe.env file.

Please refer to theEnvironment variables section.

Start Monocle:

docker-compose up -d

Ensure services are running and healthy:

docker-compose ps

Inspect services logs:

docker-compose logs -f

You should be able to access the web UI athttp://localhost:8080.

SeeTroubleshooting section if needed.

The Monocle configuration is splitted between a configuration file and various environmentvariables.

Environment variables configure system settings and secrets. Any changes requirea restart of the Monocle API.

The configuration file contains non sensitive data and could be exposed in a Git repository.A CI/CD flow could be enabled on top of the Git repository to allow Monocle' users topropose configuration changes. The Monocle API automatically reload the configuration when it changes.

For a local deployment, default settings are fine.

The following settings are available in the.env file (See.env.sample for a sample file):

• COMPOSE_ES_XMS and COMPOSE_ES_XMX to change the ElasticSearch JVM HEAP SIZE. By default 512m.
• COMPOSE_MONOCLE_VERSION to use a specific version. By default it uses latest stable release version.
• COMPOSE_MONOCLE_API_ADDR to set binding address where the Monocle API is exposed by the container.
• COMPOSE_MONOCLE_API_PORT to set binding port where the Monocle API is exposed by the container.
• COMPOSE_MONOCLE_PUBLIC_URL to configure the public URL to access the UI and API.The URL is required for user redirection during the authentication. By default it ishttp://localhost:8080
• COMPOSE_MONOCLE_WEBAPP_TITLE to change the title of the web application. By default it isMonocle.

The following settings are available in the.secrets file:

• MONOCLE_JWK_GEN_KEY to set the local JWT issuer key. The key size must be 64 characters minimum.By default the key is automatically generated.
• MONOCLE_OIDC_<PROVIDER_NAME>_CLIENT_SECRET to set the secret used by Monocle to request an ID Token (Unset by default).
• MONOCLE_ADMIN_TOKEN to set thetoken to access admin endpoints.By default not set and endpoints deactivated.

The Monocle configuration file is used by the API and crawlers processes. The format of the file is YAML.

The file configures the following:

• Monocle workspaces
• TheAbout WEB Application endpoint
• TheAuthentication system

You might want to use Dhall to manage it or to better understand the schema (dhall-monocle).

A workspace uses a dedicated ElasticSearch index. A workspace defines:

• crawlers -details
• projects -details
• identities -details
• groups -details
• search aliases -details

Monocle provides two kinds of crawlers:

• Change: A crawler to fetch Changes proposed to a repository. Monocle supports Gerrit (Reviews), GitHub (Pull-Requests), GitLab (Merge-Requests).
• TaskData: A crawler to fetch task data related to a repository. Monocle supports GitHub (issues),and BugZilla (Bugs).

The.secrets file is used to store credentials and API keys used by crawlers to authenticate on providers.

Thecrawlers value is a list of crawler. Each crawler is composed of:

• name: an abitrary name used to identify the crawler.
• update_since: the crawler will fetch changes that has been created/updated since that date.
• provider: provider settings

workspaces:  -name:democrawlers:      -name:spinnakerupdate_since:"2020-01-01"provider:{}

A GitHub provider settings

provider:github_organization:spinnaker# Optional settingsgithub_repositories:      -pipelinegithub_url:https://github.com/api/graphqlgithub_token:GITHUB_TOKEN

github_organization is the only mandatory key. Ifgithub_repositories is not specified thenthe crawler will crawl the whole organization repositories. If specified then it will crawl onlythe listed repositories. To crawl repositories from a personnal GitHub account, you need to setgithub_organization to the account name and list repositories under thegithub_repositories key.

github_url might be specified in case of an alternate url. Default is "https://github.com/api/graphql".

github_token might be specified to use an alternate environment variable name to look for thetoken value. Default is "GITHUB_TOKEN"

Regarding the Github tokens (classic):

• To crawl public repositories, no specific scope are needed (no checkbox selected).
• To crawl privates repositories, you must set the "repo" scope.

Regarding the Github fine grained tokens (new):

• To crawl public repositories, keep the "Public Repositories (read-only)" checkbox selected. Nothing else is needed.
• To crawl privates repositories, select "All repositories" or "Only select repositories", then in "Repository permissions"select "Pull Requests", "Contents" as "Read-only".

The GitHub provider can also be configured to crawl Pull-Requests created by specific GitHub users.For instance the following crawler's provider will fetch Pull-Requests and related events forusers john and jane:

provider:github_users:      -john      -jane# Optional settingsgithub_url:https://github.com/api/graphqlgithub_token:GITHUB_TOKEN

A Gerrit provider settings

provider:gerrit_url:https://review.opendev.orggerrit_repositories:      -openstack/nova      -openstack/neutron# Optional settingsgerrit_login:monoclegerrit_password:GERRIT_PASSWORDgerrit_prefix:opendev/

gerrit_url is mandatory and must be the url of the Gerrit provider.gerrit_repositories is mandatory and is the list of repositories from which the crawler will fetch Reviews from.

gerrit_login might be specified to authenticate on the provider API.gerrit_passwordmight be specified to use an alternate environment variable name to look for thepassword. Default is "GERRIT_PASSWORD"

gerrit_prefix might be set to configure the crawler to prepend the repository name with a prefix.

A GitLab provider settings

provider:gitlab_organization:redhat/centos-stream/ci-cd/zuul# Optional settingsgitlab_repositories:      -jobs-configgitlab_url:https://gitlab.com/api/graphqlgitlab_token:GITLAB_TOKEN

gitlab_organization is the only mandatory key. Ifgitlab_repositories is not specified thenthe crawler will crawl the whole organization repositories. If specified then it will crawl onlythe listed repositories.

gitlab_url might be specified in case of an alternate url. Default is "https://gitlab.com/api/graphql".

gitlab_token might be specified to use an alternate environment variable name to look for thetoken value. Default is "GITLAB_TOKEN"

To crawl privates repositories, you must generate a Personal Access Token with the "read_api" scope.

Monocle provides additional crawlers to attach tasks/issues/RFEs to changes based on amatch onchange_url. Then, Changes are enhanced with information about relatedtasks such as apriority or ascore.

For GitHub:

The GitHub TaskData crawler run automatically whenever repositories are specified into aGitHub Changes crawler definition.

When using the "Fine grained" token for private repositories, then make sure to select the "Issues" permission.

For Bugzilla:

A BugZilla provider settings

provider:bugzilla_url:https://redhat.bugzilla.combugzilla_products:      -Awesome product# Optional settingsbugzilla_token:BUGZILLA_TOKEN

bugzilla_product must be specified. The crawler will crawl listed products for bugs thatcontain an external bug referencesext_bz_bug_id. The crawler assumes that the externalreference is used to link to a change (Pull-Request/Review).

bugzilla_token might be specified to use an alternate environment variable name to look for thetoken value. Default is "BUGZILLA_TOKEN"

Note that this crawler is managed by thecrawler container.

Projects could be defined within a workspace configuration. A project is identified by a name and allows to set the following filter attributes:

• repository_regex
• branch_regex
• file_regex

Here is an example of configuration.

workspaces:  -name:examplecrawlers:      -name:openstackprovider:gerrit_url:https://review.opendev.orggerrit_repositories:            -^openstack/.*update_since:"2021-01-01"projects:      -name:computerepository_regex:".*nova.*"      -name:compute-testsfile_regex:"test[s]/.*"repository_regex:".*nova.*"      -name:deploymentrepository_regex:".*tripleo.*|.*puppet.*|.*ansible.*"branch_regex:"master"

The monocle API endpointapi/1/get_projects can be queried toretrieved the list defined projects for a given workspace. See theMonocle OpenAPI.

The monocle query endpoint handles the query parameter:project.

Monocle is able to index changes from multiple code review systems. A contributormight get different identities across code review systems. Thus Monocle providesa configuration section to define aliases for contributors.

Let say a Monocle workspace is configured to fetch changes from github.com andreview.opendev.org (Gerrit) and we would like that John's metrics are merged undertheJohn Doe identity.

workspaces:  -name:exampleidents:      -ident:John Doealiases:          -github.com/john-doe          -review.opendev.org/John Doe/12345          -AuthProviderUID:jdoecrawlers:      -name:github-containersprovider:github_organization:containersgithub_token:<github_token>update_since:'2000-01-01'      -name:gerrit-opendevprovider:gerrit_url:https://review.opendev.orggerrit_repositories:            -^openstack/.*update_since:'2000-01-01'

A contributor id on github.com or a GitHub enterprise instance is formated as<domain>/<login>.

A contributor id on a Gerrit instance is formated as<domain>/<Full Name>/<gerrit-user-id>.

Database objects must be updated to reflect the configuration. Onceconfig.yaml is updated, run the following commands:

docker-compose run --rm --no-deps api monocle janitor update-idents --elastic elastic:9200 --config /etc/monocle/config.yaml

A group in Monocle permits to group authors of Changes and filter them from the web interface.

Group memberships are defined through theidents section of the configuration.

Here is an example:

workspaces:  -name:exampleidents:      -ident:John Doealiases:          -github.com/john-doe          -review.opendev.org/John Doe/12345groups:          -devs      -ident:Jane Doealiases:          -github.com/jane-doegroups:          -devs          -ptl

The Monocle configuration file provides a way to define aliases to be used in search queries.A use case could be to groupbot authors to provide an easy way to exclude them from search results.

Here is an example:

workspaces:  -name:examplesearch_aliases:      -name:botsalias:'(author:"github-actions" or author:"bedevere-bot")'

Then the query could be: "from:now-3weeks and not bot".

This section configures information to be displayed by the WEB Application on theAbout modal.

Here is an example:

about:links:    -category:Communityname:Monocle configurationurl:https://github.com/change-metrics/demo-node-config

This section configures global crawlers setting.

Note that,crawlers is a config root level parameter.

Here is an example:

crawlers:# Wait one hour between iterationloop_delay_sec:3600

Monocle supports user authentication via an OIDC (OpenID Connect) provider.

Once authenticated, Monocle is able to display personalized content like resolving theself query value such as in the query:author: self.

Monocle supports one provider at a time.

Note that by default and if authentication system is not configured or misconfigured,Monocle provides anunauthentcated login mechanism which allows a user to enter itsauthor name.

Here is an example of configuration:

auth:auth_provider:oidc_client_id:my_client_idoidc_issuer_url:https://accounts.google.comoidc_provider_name:my provideroidc_user_claim:emailenforce_auth:false

• oidc_client_id: is the client id that is provided by your OIDC provider.
• oidc_issuer_url: is the URL used to discover your OIDC's providerconfiguration (via the.well-known/openid-configuration endpoint).
• oidc_provider_name: is the name displayed on the authentication modal in the WEB Application.
• oidc_user_claim: (optional) is the JWT claim used as/(or to discover matching) Monocle User ID. (Default:sub)
• enfore_auth: (optional) if set toTrue then Monocle requires a valid JWT to access any API endpoints and theWEB Application requires the user to login to navigate. (Default:False)

The redirect URI to configure on the OIDC provider is:<MONOCLE_PUBLIC_URL>/api/2/auth/cb.

Furthermore, two additional settings (environment variables) are required to enable the Authentication provider:

• MONOCLE_PUBLIC_URL: is used to redirect the OIDC provider to the callback endpoint and the user tothe Monocle WEB Application after a successful authentication.
• MONOCLE_OIDC_<PROVIDER_NAME>_CLIENT_SECRET: Monocle looks for this variable to discover the client secretwhich has been provided by your OIDC provider.

Monocle API generates the following log line when the Authentication provider has been properly set up:

2022-10-11 10:39:11 INFO    Monocle.Main:149: AuthSystemReady {"provider":"my provider"}

Monocle issues its own JWTs to the WEB Application users. JWTs issued by the OIDC provider are used to getthe authenticated user's information such as its unique uid via theoidc_user_claim. Monocle uses the UID (from the OIDC provider) to discover through theidents settings a matchingMonocle Ident. Amatching ident defines analias which isAuthProviderUID:<user_unique_id>. In case ofno matching ident then the user claim value is used.

A Monocle JWT is valid for 24 hours, once expired the user must login again to the OIDC provider.

Monocle can issue service tokens to be used by applications that consume the Monocle API. To requesta token your need to perform the following call:

curl -XPOST -d'{"token": "<admin_token>"}' -H"Content-type: application/json"<monocle_public_url>/auth/get{"jwt":"eyJhbGciOiJIUzUxMiJ9.eyJkYXQiOnsiYURlZmF1bHRNdWlkIjoiYm90IiwiYU11aWRNYXAiOnt9fX0.bmj5vcxXxz2LmkrVKxX8jd-aYzHeTng_QBzR_9YZwCb9ToKA59TVlN1wZf6hhPlUX1VU82Y94gVCREDifintZg"}

Then set theAuthorization header to access Monocle API:

curl -XPOST -d'{"void": ""}' -H"Content-type: application/json" -H'Authorization: Bearer eyJhbGciOiJIUzUxMiJ9.eyJkYXQiOnsiYURlZmF1bHRNdWlkIjoiYm90IiwiYU11aWRNYXAiOnt9fX0.bmj5vcxXxz2LmkrVKxX8jd-aYzHeTng_QBzR_9YZwCb9ToKA59TVlN1wZf6hhPlUX1VU82Y94gVCREDifintZg'<monocle_public_url>/auth/whoami

TheMONOCLE_ADMIN_TOKEN must be set to enable this endpoint.

Here are the expected environment variables that need to be added to the.secrets file:

• CRAWLERS_API_KEY: an arbitrary api key used by the crawler to index data.
• GITHUB_TOKEN: an API key for GitHub crawler.
• GITLAB_TOKEN: an API key for GitLab crawler.

Open the sampleconfig.yaml.

ElasticSearch could need some capabilities to run in containermode. Take a look at the logs to see if it started correctly:

$docker-compose logs elastic

For example, you could need to increase this system parameter:

$sudo sysctl -w vm.max_map_count=262144

or make the data directory writable for other:

$chmod og+w data

To delete a workspace (a workspace is an elasticsearch index):

#List indexes with:docker-compose run --rm --no-deps api curl http://elastic:9200/_aliases?pretty=true#Delete an index withdocker-compose run --rm --no-deps api curl -XDELETE http://elastic:9200/<index-name>

ElasticSearch sets defaults settings on new indexes. The default setting for queries basedon regex is set to a value that might not fit your usage especially when your project definitionsuses regex above that limit. However the limit could be increased using the following command:

docker-compose run --rm --no-deps api curl \-XPUT http://localhost:9200/monocle.changes.1.<index-name>/_settings \-H "Content-Type: application/json" -d '{"index": {"max_regex_length": 50000}}'

Crawler default ciphers can be restrictive and not able to work with some load balancer of remoteserver advertising an unsupported cipher. If you experiencewrong signature type errors in thecrawler container, you should consider changing ciphers using theTLS_CIPHER environment variablein your docker-compose configuration file. You can findadditional information on Fedora changelog.

To disable TLS verification, set theTLS_NO_VERIFY environment variable to1.

If you're accessing the web service remotely, make sure to setMONOCLE_PUBLIC_URL to the right URL in your.env file as this will be used to redirect requests from the UI. If not you'll see the UI fail with a "Network error" message.

To wipe any data related to a workspace's crawler:

docker-compose stop crawlerdocker-compose run --rm --no-deps api monocle janitor wipe-crawler-data --elastic elastic:9200 --config /etc/monocle/config.yaml --workspace<workspace> --crawler-name<crawler-name>docker-compose start crawler

Monocle crawlers keep track of the last date (commit date) when a successful document fetch happened. The commandbelow can be used to force a crawler to fetch (again) documents since another date.

docker-compose run --rm --no-deps api monocle janitor set-crawler-commit-date --elastic elastic:9200 --config /etc/monocle/config.yaml --workspace<workspace> --crawler-name<crawler-name> --commit-date 2023-01-01

Monocle is composed of the following services:

1. an Elasticsearch data store.
2. an API service to serve the Web APP and API requests.
3. a crawler service to retrieve change from providers.

The APIs are defined usingprotobuf and served over HTTP throughMonocle OpenAPI.

Most of the metrics are exposed via theapi/2/metric/get endpoint. Theapi/2/metric/list lists theavailable metrics. The Monocle Web UI leverages those metrics via theCatalog page.

The detail about the query and response format is described in theMonocle OpenAPI document.

Note that there is an effort in progress to ensure that all metrics displayed by the Monocle Web UI use theapi/2/metric endpoint.

To list the available metrics:

curl'https://demo.changemetrics.io/api/2/metric/list' \  -H'Content-Type: application/json' \  --data'{"void":""}'

To get theChanges created metric as atrend with automatic interval:

curl'https://demo.changemetrics.io/api/2/metric/get' \  -H'Content-Type: application/json' \  --data'{"index":"python","username":"","query":"from:now-3weeks","metric":"changes_created","trend":{"interval":""}}'

To setup the monitoring:

1. Create the prometheus service by providing the API and CRAWLER location

export API_TARGET=localhost:8080export CRAWLER_TARGET=localhost:9001mkdir -p /srv/prometheuspodman create --network host -v /srv/prometheus:/var/lib/prometheus:Z -e API_TARGET=${API_TARGET} -e CRAWLER_TARGET=${CRAWLER_TARGET} --name monocle-prometheus quay.io/change-metrics/monocle-prometheus:latest

2. Create the grafana service (on the prometheus host)

mkdir -p /srv/grafanapodman create --network host -v /srv/grafana:/var/lib/grafana:Z -e GRAFANA_PASS=secret --name monocle-grafana quay.io/change-metrics/monocle-grafana:latest

3. Starts the services with systemd

mkdir -p ~/.config/systemd/user/for service in prometheus grafana; do podman generate systemd -n monocle-$service > ~/.config/systemd/user/monocle-$service.service; donesystemctl --user daemon-reloadfor service in prometheus grafana; do systemctl --user start monocle-$service; done

4. Check metrics with grafana dashboard athttp://localhost:19030/ (login with 'admin:secret')

If you use Monocle, please open aPull Request to get added to this list.