Tutorial Part 2: Auto-generating conn-check config for a Django app

Hello World (again)

Let’s assume that you’ve actually created the Hello World service from part 1 as a Django app, and you think to yourself:

“Hang on, aren’t all these connections I want conn-check to check for me already defined in my Django settings module?”

conn-check-configs

Yes, yes they are, and with the handy-dandy conn-check-configs package you can automatically generate conn-check config YAML from a range of standard Django settings values (in theory from other environments too, such as Juju, but for now just Django).

exempli gratia

Given the following settings.py in our HWaaS app:

INSTALLED_APPS = [
    'hwaas'
]
DATABASES = {
    'default': {
            'ENGINE': 'django.db.backends.postgresql_psycopg2',
            'HOST': 'gibson.hwass.internal',
            'NAME': 'hwaas_production',
            'PASSWORD': '123456asdf',
            'PORT': 11211,
            'USER': 'hwaas',
}
CACHES = {
    'default': {
        'LOCATION': 'freeside.hwaas.internal:11211',
        'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache',
    },
}
PROXY_HOST = 'countzero.hwaas.internal'
PROXY_PORT = 8080
TRANSLATE_API_KEY = 'BLAH'

We can create a settings-to-conn-check.py script with the least possible effort like so:

#!/usr/bin/env python
from conn_check_configs.django import run

if __name__ == '__main__':
    run()

This will output postgresql and memcached checks to similar our hand-written config:

$ chmod +x settings-to-conn-check.py
$ ./settings-to-conn-check.py -f cc-config.yaml -m hwaas.settings
$ cat cc-config.yaml
- type: postgresql
  database: hwaas_production
  host: gibson.hwaas.internal
  port: 5432
  username: hwaas
  password: 123456asdf
- type: memcached
  host: freeside.hwaas.internal
  port: 11211

Customising generated checks

In order to generate the checks we need for Squid / Google Translate API, we can add some custom callbacks:

#!/usr/bin/env python
from conn_check_configs.django import run, EXTRA_CHECK_MAKERS


def make_proxied_translate_check(settings, options):
    checks = []
    if settings['PROXY_HOST']:
        checks.append({
            'type': 'http',
            'url': 'https://www.googleapis.com/language/translate/v2?q='
                   'Hello%20World&target=de&source=en&key={}'.format(
                       settings['TRANSLATE_API_KEY']),
            'proxy_host': settings['PROXY_HOST'],
            'proxy_port': int(settings.get('PROXY_PORT', 8080)),
            'expected_code': 200,
        })
    return checks

EXTRA_CHECK_MAKERS.append(make_proxied_translate_check)


if __name__ == '__main__':
    run()

In the above we define a callable which takes 2 params, settings which is a wrapper around the Django settings module, and options which is an object containing the command line arguments that were passed to the script.

The settings module is not the direct settings module but a dict-like wrapper so that you can access the settings just a like a dict (using indices, .get method, etc.)

To ensure make_proxied_translate_check is collected and called by the main run function we add it to the EXTRA_CHECK_MAKERS list.

The above generates our required HTTP check:

- type: http
  url: https://www.googleapis.com/language/translate/v2?q=Hello%20World&target=de&source=en&key=BLAH
  proxy_host: countzero.hwaas.internal
  proxy_port: 8080
  expected_code: 200

A note on statstd checks

Getting more operational visibility on how HWaaS runs would be great, wouldn’t it?

So let’s add some metrics collection using StatsD, and as luck would have it we can get a lot for nearly free with the django-statsd, after adding it to our dependencies we update our settings.py to include:

INSTALLED_APPS = [
    'hwaas'
    'django_statsd',
]
MIDDLEWARE_CLASSES = [
    'django_statsd.middleware.GraphiteMiddleware',
]
STATSD_CLIENT = 'django_statsd.clients.normal'
STATSD_HOST = 'bigend.hwaas.internal'
STATSD_PORT = 10021

Note: You don’t actually need the django-statsd app to have conn-check-configs generate statsd checks, only the use of STATSD_HOST and STATSD_PORT in your settings module matters.

Another run of our settings-to-conn-check.py script will result in the extra statsd check:

- type: udp
  host: bigend.hwaas.internal
  port: 10021
  send: conncheck.test:1|c
  expect:

As you can see this is just a generic UDP check that attempts to send an incremental counter metric to the statsd host.

Unfortunately the fire-and-forget nature of this use of statsd/UDP will not error in a number of common situations (the simplest being that statsd is not running on the target host, or even a routing issue along the way).

It will catch simple problems such as not being able to open up the local UDP port to send from, but that’s usually not enough.

If you use a third-party implementation of statsd, such as txStatsD then you might have the ability to define a pair of health check strings, for example by changing the send and expect values in the STATSD_CHECK dict we can send and expect different strings:

#!/usr/bin/env python
from conn_check_configs.django import run, STATSD_CHECK

STATSD_CHECK['send'] = 'Hakuna'
STATSD_CHECK['expect'] = 'Matata'

if __name__ == '__main__':
    run()

Which generates this check:

- type: udp
  host: bigend.hwaas.internal
  port: 10021
  send: Hakuna
  expect: Matata

In the above we would configure our txStatD (for example) instance to respond to the string Hakuna with the string Matata, which would catch pretty much all the possible issues with contacting our metrics service.