Repository files navigation

django-watchman exposes a status endpoint for your backing services likedatabases, caches, etc.

The full documentation is athttp://django-watchman.rtfd.org.

We're in love with django-watchman. External monitoring is a vital part of our service offering. Using django-watchman we can introspect the infrastructure of an application via a secure URL. It's very well written and easy to extend. We've recommended it to many of our clients already.

— Hany Fahim, CEO,VM Farms.

1. Installdjango-watchman:

   pip install django-watchman

2. Addwatchman to yourINSTALLED_APPS setting like this:

   INSTALLED_APPS = (    ...    'watchman',)

3. Include the watchman URLconf in your projecturls.py like this:

   re_path(r'^watchman/', include('watchman.urls')),

4. Start the development server and visithttp://127.0.0.1:8000/watchman/ toget a JSON response of your backing service statuses:

   {    "databases": [        {            "default": {                "ok": true            }        }    ],    "caches": [        {            "default": {                "ok": true            }        }    ],    "storage": {"ok": true}}

Visithttp://127.0.0.1:8000/watchman/dashboard/ to get a human-friendly HTMLrepresentation of all of your watchman checks.

If you want to protect the status endpoint, you can use theWATCHMAN_TOKENS setting.This is a comma-separated list of tokens.When this setting is added, you must pass one of the tokens in as thewatchman-tokenGET parameter:

GET http://127.0.0.1:8000/watchman/?watchman-token=:token

Or by setting theAuthorization: WATCHMAN-TOKEN header on the request:

curl -X GET -H "Authorization: WATCHMAN-TOKEN Token=\":token\"" http://127.0.0.1:8000/watchman/

If you want to change the token name, you can set theWATCHMAN_TOKEN_NAME.The value of this setting will be theGET parameter that you must pass in:

WATCHMAN_TOKEN_NAME = 'custom-token-name'GET http://127.0.0.1:8000/watchman/?custom-token-name=:token

DEPRECATION WARNING:WATCHMAN_TOKEN was replaced by theWATCHMAN_TOKENS setting to support multiple authentication tokens in django-watchman0.11.It will continue to work until it's removed in django-watchman1.0.

If you want to protect the status endpoint with a customizedauthentication/authorization decorator, you can addWATCHMAN_AUTH_DECORATORto your settings. This needs to be a dotted-path to a decorator, and defaultstowatchman.decorators.token_required:

WATCHMAN_AUTH_DECORATOR = 'django.contrib.admin.views.decorators.staff_member_required'

Note that thetoken_required decorator does not protect a view unlessWATCHMAN_TOKENS is set in settings.

django-watchman allows you to customize the checks which are run by modifyingtheWATCHMAN_CHECKS setting. Insettings.py:

WATCHMAN_CHECKS = (    'module.path.to.callable',    'another.module.path.to.callable',)

You can also import the watchman.constants to include the DEFAULT_CHECKS and PAID_CHECKS in yoursettings.py:

from watchman import constants as watchman_constantsWATCHMAN_CHECKS = watchman_constants.DEFAULT_CHECKS + ('module.path.to.callable', )

Checks take no arguments, and must return adict whose keys are applied to the JSON response.

Use thewatchman.decorators.check decorator to capture exceptions:

from watchman.decorators import checkdef custom_check():    return {"custom_check": _custom_check()}@checkdef _custom_check():    return {"ok": True, "extra_info": "if helpful"}

In the absence of any checks, a 404 is thrown, which is then handled by thejson_view decorator.

A subset of checks may be run, by passing?check=module.path.to.callable&check=...in the request URL. Only the callables given in the querystring which are alsoinWATCHMAN_CHECKS should be run, eg:

curl -XGET http://127.0.0.1:8080/watchman/?check=watchman.checks.caches

You can skip any number of checks, by passing?skip=module.path.to.callable&skip=...in the request URL. Only the checks inWATCHMAN_CHECKS which are not in thequerystring should be run, eg:

curl -XGET http://127.0.0.1:8080/watchman/?skip=watchman.checks.email

If your application has a large number of databases or caches configured,watchman may open too many connections as it checks each database or cache.

You can set theWATCHMAN_DATABASES orWATCHMAN_CACHES settings in orderto override the default set of databases and caches to be monitored.

If you want to simply check that your application is running and able to handlerequests, you can call ping:

GEThttp://127.0.0.1:8000/watchman/ping/

It will return the textpong with a 200 status code. Calling this doesn'trun any of the checks.

If you would like a "bare" status view (one that doesn't report any details,justHTTP 200 if checks pass, andHTTP 500 if any checks fail), youcan use thebare_status view by putting the following intourls.py:

import watchman.views# ...re_path(r'^status/?$', watchman.views.bare_status),

You can also run your checks without starting the webserver and making requests.This can be useful for testing your configuration before enabling a server,checking configuration on worker servers, etc. Run the management command like so:

python manage.py watchman

By default, successful checks will not print any output. If all checks passsuccessfully, the exit code will be0. If a check fails, the exit code willbe1, and the error message including stack trace will be printed tostderr.

If you'd like to see output for successful checks as well, set verbosity to2 or higher:

python manage.py watchman -v 2{"storage": {"ok": true}}{"caches": [{"default": {"ok": true}}]}{"databases": [{"default": {"ok": true}}]}

If you'd like to run a subset of checks, use-c and a comma-separated listof python module paths:

python manage.py watchman -c watchman.checks.caches,watchman.checks.databases -v 2{"caches": [{"default": {"ok": true}}]}{"databases": [{"default": {"ok": true}}]}

If you'd like to skip certain checks, use-s and a comma-separated list ofpython module paths:

python manage.py watchman -s watchman.checks.caches,watchman.checks.databases -v 2{"storage": {"ok": true}}

Use-h to see a full list of options:

python manage.py watchman -h

Watchman can return the version of watchman which is running to help you keeptrack of whether or not your sites are using an up-to-date version. This isdisabled by default to prevent any unintended information leakage for websiteswithout authentication. To enable, update theEXPOSE_WATCHMAN_VERSIONsetting:

EXPOSE_WATCHMAN_VERSION = True

By default, watchman will return a500 HTTP response code, even if there's afailing check. You can specify a different response code for failing checksusing theWATCHMAN_ERROR_CODE setting:

WATCHMAN_ERROR_CODE = 200

watchman includes log messages using a logger calledwatchman.You can configure this by configuring theLOGGING section of your Djangosettings file.

Here is a simple example that would log to the console:

LOGGING = {    'version': 1,    'disable_existing_loggers': False,    'handlers': {        'console': {            'class': 'logging.StreamHandler',        },    },    'loggers': {        'watchman': {            'handlers': ['console'],            'level': 'DEBUG',        },    },}

More information is available in theDjango documentation.

If you're using APM and watchman is being often hit for health checks (such as an ELB onAWS), you will find some stats based on averages will be affected (average transactiontime, apdex, etc):

You can disable APM instrumentation for watchman by using theWATCHMAN_DISABLE_APMsetting:

WATCHMAN_DISABLE_APM = True

This currently supports the following agents:

• Datadog
• New Relic

Please open an issue if there's another APM you use which is being affected.

For each cache indjango.conf.settings.CACHES:

• Set a test cache item
• Get test item
• Delete test item

For each database indjango.conf.settings.DATABASES:

• Verify connection by callingconnections[database].introspection.table_names()

Send a test email toto@example.com usingdjango.core.mail.send_mail.

If you're using a 3rd party mail provider, this check could end up costing youmoney, depending how aggressive you are with your monitoring. For this reason,this check isnot enabled by default.

For reference, if you were using Mandrill, and hitting your watchman endpointonce per minute, this would cost you ~$5.60/month.

Custom Settings

• WATCHMAN_EMAIL_SENDER (default:watchman@example.com): Specify an email to be the sender of the test email
• WATCHMAN_EMAIL_RECIPIENTS (default:[to@example.com]): Specify a list of email addresses to send the test email
• WATCHMAN_EMAIL_HEADERS (default:{}): Specify a dict of custom headers to be added to the test email

Usingdjango.core.files.storage.default_storage:

• Write a test file
• Check the test file's size
• Read the test file's contents
• Delete the test file

By default the test file gets written on the root of the djangoMEDIA_ROOT.

There are two reasons why you may need to override this. You can usethe settingWATCHMAN_STORAGE_PATH to accomplish this.

1) Django triggers a django.core.exceptions.SuspiciousFileOperation<https://docs.djangoproject.com/en/5.0/ref/exceptions/#suspiciousoperation>on the storage check.

2) If for whatever reason, the base ofMEDIA_ROOT is not writable bythe user that runs Django.

In either case, choose a path within and relative toMEDIA_ROOT.

WATCHMAN_STORAGE_PATH = "django-watchman/"

If theMEDIA_ROOT is already defined:

from os.path import join as joinpathWATCHMAN_STORAGE_PATH = joinpath(MEDIA_ROOT, "django-watchman/")

By default, django-watchman will run checks against your databases(watchman.checks.databases), caches (watchman.checks.caches), andstorage (watchman.checks.storage).

Paid checks are checks which may cost you money if they are run regularly.

Currently there is only one "paid" check -watchman.checks.email. Manytimes email is sent using managed services like SendGrid or Mailgun. You canenable it by setting theWATCHMAN_ENABLE_PAID_CHECKS toTrue, or byoverriding theWATCHMAN_CHECKS setting.

A sample project is available along with a Dockerfile to make it easy to tryout django-watchman. It includes examples of how to write simple custom checks.

One of the custom checks will always fail, so if you want to see what responseslook like with 100% succeeding checks, add ?skip=sample_project.checks.fail_custom_check

• Docker <https://www.docker.com/get-docker>

1. Build and run the Docker image with the current local code:make run
2. Visit watchman json endpoint in your browser:http://127.0.0.1:8000/watchman/
3. Visit watchman json endpoint in your browser:http://127.0.0.1:8000/watchman/?skip=sample_project.checks.fail_custom_check
4. Visit watchman dashboard in your browser:http://127.0.0.1:8000/watchman/dashboard/
5. Visit watchman dashboard in your browser:http://127.0.0.1:8000/watchman/dashboard/?skip=sample_project.checks.fail_custom_check
6. Visit watchman ping in your browser:http://127.0.0.1:8000/watchman/ping/
7. Visit watchman bare status in your browser:http://127.0.0.1:8000/watchman/bare/