Repository files navigation

Apub/sub-based implementation of aRaspberry Pi data pipeline built onredis and centered around sensors.

The general philosophy is that a "sensor" is any entity (physical or not) that operates primarilyin an output-only mode (configuration doesn't necessarily count as an "input" so is allowable).

These outputs are treated ephemerally, in a "fire and forget" manner, creating a data stream.The intent is for interested entities to subscribe to the data stream and transform, interpretand/or persist it according to their requirements.

This is still very much an active work-in-progress! However, as it is functional and actively deployed I figured it was worth making public in the event it might help others in their projects.

Those current deployments consist of myatmospheric particulate matter sensor and my garden monitoring bots, with the entire system handling (on average) about a half-million units of sensor data per day.

• Hardware:
  • a Raspberry Pi running a recent Raspbian build (for sensor nodes)
  • any Debian-like system running apt (for managment/non-sensor nodes)
• somesensors, configured appropriately (most easily done withraspi-config):
  • depending on your chosen sensors: I2C enabled, SPI enabled, 1-wire enabled
  • other sensors (LM335, Soil, TEPT5700) require anexternal ADC which itself will require SPI

• Clone the repo
• cd into repo dir
• ./setup.sh (you might need to enter yoursudo password to install requirements)
• source env/bin/activate and go!

• sensors-src: source daemon that manages all specified sensors and publishes their data as configured
• downsample: a very flexible data stream downsampler/forwarder/transformer. example uses:
  • an at-frequency forwarder (set-r 1)
  • a loopback downsampler (set-o to the same as-i)
  • a many-to-one reducer (set-t tokey)
  • a one-to-many exploder (set-m toflatten:[options])
  • a bounded ephemeral cache (set-t tolist:[options], where thelimit=X option sets the bound)
  • ...
  • profit!
• sqlite-sink: anSQLite sink for data streams. examples:
  • sink to an SQLite database on a different host a downsampled data stream:
    1. on the source device:
       • downsample -i redis://localhost -o redis://sql-db-host -r ... -p ...
    2. on the sink host "sql-db-host":
       • sqlite-sink path-to-db.sqlite3
         • (lots of "TODOs" here, obviously)
• oled-display: anOLED display driver for consuming & display some sensor data, among other things
• thingspeak: a simple example of aThingSpeak data forwarder for the SPS30 particulate matter sensor data.Example resulting data set.

The following are currently supported (with the required drivers / interfaces setup of course):

• SPS30 I2C particulate matter air quality sensor.Source.
• BME680 temperature / humidity / barometeric pressure / volatile organic compound I2C sensor. My setup usesAdaFruit's awesome breakout board for ease-of-integration.Source.
• BME280 temperature / humidity / barometeric pressure I2C sensor.Source.
• DHTXX(DHT11/DHT22) temperature / humidity sensors. I useDFRobot'sbreakouts but you can find similar breakout's from many (re)sellers online.Source.
• DS18S20 high-precision 1-wire temperature sensor.Source.
• LM335 precision analog temperature sensor.Source.
• Capacative soil moisture sensors such as the DFRobotSEN0114 or anysimple-to-build capactive analog moisture sensor.Source.
• TEPT5700 ambient light sensor.Source.
• "Virtual" sensors such asSysInfo andNetInfo that do not require any additional hardware.

Variants of these sensors could likely be made to work with this system via simple modifications if any are required at all.

• A simple Python wrapper driver for the SPS30 sensor is includedhere, wrapping the includedembedded-sps I2C driver from Sensiron implemented in C (forked to include a shared object build step for Python consumption).
• A simple Python driver for the74HC595 8-bit shift register is includedhere.
• The venerableMCP3008 10-bit 8-channel analog-to-digital converter is directly supported by thebase analog sensor implementation. Other ADCs would be quite simple to adapt (even more so if I implemented a HAL... #TODO).
  • channel (zero-indexed) is specified inconfig.json

• The old now-archived repository from whence this all came
  • might still contain some useful stuff: D3-based "live" analog data plotting code inrpjctrl, and I'm still usingledCount.py on some of my units because I'm too lazy to re-write it properly