Repository files navigation

You can watch my PyCon YouTube video about this project here:

PyCon 2020 Talk:

Jeff Bass - Yin Yang Ranch: Building a Distributed ComputerVision Pipeline using Python, OpenCV and ZMQ

PyCon 2020 Talk Video about this project

PyCon 2020 Talk Presentation slides

If you have a question about my PyCon 2020 talk, open an issue!For more about that seeyin-yang-ranch issue 2.

This project is a collection of Python programs and Raspberry Pi hardwareprojects to help manage a small urban permaculture farm calledYin Yang Ranch.The 2 acre farm is an ongoing science project to build living soil, capture rainin barrels, and grow a variety of plants and fruit trees that can feed birds,bees, butterflies and people. We are in Southern California about 10 miles fromthe Malibu coast. Drought and limited rainfall are the toughest climate issues.Monitoring and observation are important, so I built a Raspberry Pi Camera systemto read the water meter and monitor temperatures to optimize irrigation. I can send atext message to the system ("Susan") to ask about water usage or temperatures:

This repository contains the software and the hardware designs used to buildour measurement and monitoring systems.yin-yang-ranch is a continuouslyevolving project with a lot of hardware hacking and software refactoring. Iam open-sourcing everything in case it might be helpful to others.My projects use Raspberry Pi computers, PiCameras, various sensors and relatedelectronics. I control the hardware with Python programs that use computervision, OpenCV, Numpy, pandas, and the PyZMQ messaging library. I use theRaspberry Pi GPIO Python module to control lights (e.g., to light the watermeter) and irrigation valves.

The Yin Yang Ranch project is made up of 4 repositories on GitHub:

1. yin-yang-ranch: this repository. Overall project design andlibrarian prototype code.
2. imageZMQ: Transporting OpenCV images.
3. imagenode: Capture and Send Images and Sensor Data.
4. imagehub: Receive and Store Images and Event Logs.

imageZMQ moves images taken by Raspberry Pi computers to hub computersfor image processing.imagenode runs on multiple RPi computers, continuouslycapturing images, detecting motion, and gathering sensor data (e.g. air andsoil temperatures).imagehub runs on a Mac or a Linux computer and receivesimages and event messages from 8-10 Raspberry Pi computers simultaneously.I use a variety of computer vision techniques implementedin Python. I have programs that can read the water meter. Or tell if thatcritter moving behind the barn is a coyote or a raccoon.

I also have a website atyin-yang-ranch.comthat will someday display some dashboards on weather, compost temperatures,solar power generation and when the last coyote was spotted. It is just a fewpictures of the ranch for now while I am developing the dashboard software.

The overall system design is a hub and spoke network with ZMQ messaging betweenRaspberry PiCameras and imagehubs. One image hub can simultaneously receiveimages from about 10 PiCameras. A librarian program gathers event messages andimages from the imagehubs. A communications program uses the event logs toanswer queries about images and events, as shown in the SMS text exchange picturedabove. By distributing computer vision processing pipelines across Raspberry Picomputers and more powerful computers like Macs, each computer can do what itdoes best. A Raspberry Pi can take pictures with the PiCamera and adjustcamera settings, control additional lighting, crop, flip and grayscale images,as well as detect motion. A Mac can store and index images from many RaspberryPi computers simultaneously. It can perform more complex image processing likereading the changing digits of the water meter or using image classificationtechniques to label a coyote or a raccoon in an image stream. My current setup hasabout a dozen Raspberry Pis with PiCamera modules and 2 linux laptops with webcamsattached to a single imagehub.

The project contains code repositories for each part of the design shown above:

• imagenode: image capture on Raspberry Pi and other computers using PiCameras,webcams and various OpenCV techniques for image rotation, threshholding,dilation, differencing and motion detection. Also sends sensor data, such astemperature and humidity, from sensors attached to the RPi's.Seeimagenode: Capture and Send Images and Sensor Data.
• imageZMQ: Python classes that transport OpenCV images from imagenodes toimagehubs. The imageZMQ package is pip installable.Seeimagezmq: Transporting OpenCV images.
• imagehub: receives event messages, images and sensor data from multipleRaspberry Pi and other computers via imagezmq. Stores them to disk files. Notethat theimagenodes don't store any data to the RPi SD cards, but send alltheir data to theimagehub for storage.Seeimagehub: Receiving and saving images and event data from multiple Raspberry Pi's.
• librarian: reads theimagehub event logs and storedimages to answer questions about them. A prototype of thelibrarian codeis contained in this repository. It can answer simple queries likethose in the SMS texting example above.SeeThe Librarian Prototype.Also, for an excellent alternative to my own librarian design, see thisapproach.
• commhub: provides a very simple natural language interface for answeringquestions about events and images (is the water running? was a coyote sightedtoday?). It parses the inbound questions and provides simple answers using datafrom theimagehub event logs. Thecommhub has methods for differentchannels of communication with end users. The prototypecommhub code inthis repository implements 2 communications channels: 1) SMS texting (using GoogleVoice and its Gmail interface) and 2) a terminal window CLI text interface.
• commagents: are separate Python programs connecting each communication channelto thecommhub. For example, an SMS/texting agent (example shown above),is implemented asgmail_watcher.py in this repository. Futurecommagentssuch as a Twilio SMS texting agent, an email agent and a webchat agent arebeing developed.
• yin-yang-ranch (this GitHub repository): contains overall projectdocumentation and design. This repository also contains prototype Pythonprograms for thelibrarian,commhub and an examplecommagent(in thelibrarian-prototype folder). There is also example data from myfarm in thetest-data folder. That folder containsimagehub logsand captured images from my farm (including images of coyotes, a bobcat, themail truck and an Amazon delivery ;-)

This distributed design allows each computer to do what it does best. ARaspberry Pi with a PiCamera can watch a water meter for needle motion, thentransmit only those images show the water flow changes (from flowingto not flowing or vice versa). The logic for motion detection and imageselection runs in the Raspberry Piimagenode, which only sends relevant imagesto theimagehub, saving network bandwidth. Theimagehub stores the eventmessages and images from multiple nodes at the same time. Thelibrarianprogram answers user queries about images and event messages. A morecomplete "which computer does what" explanation can be found inDistributing tasks among the multiple computers.

The system is written in Python and uses these packages. Higher versions willusually work fine, but these specific ones are known to work. See each specificrepository above for more software details.

• Python 3.6 through 3.11
• OpenCV 3.3 and 4.0+
• Raspian OS Buster, Stretch and Raspbian Jessie using legacy PiCamera
• Raspberry Pi OS Bookworm and Bullseye using PiCamera2
• PyZMQ 20.0+
• imagezmq 1.1.1+

The project uses a wide variety of electronics hardware:

• Raspberry Pi computers with both PiCameras and webcams.
• Mac and Linux laptops (some with webcams as image nodes).
• Temperature and humidity sensors.
• Lighting control electronics (e.g., to light the water meter).
• Motion detection sensors (both PIR and ultrasonic).
• Infrared lighting arrays (to watch for coyotes and raccoons at night).
• Irrigation actuators to turn water on and off.
• Solar panel monitoring hardware with programs to optimize power use and trackthe daily, monthly and annual sunshine energy reaching the farm. Hours andintensity of sunlight are big factors in photosynthesis, plant growth ratesand water requirements.

This is what a water meter looks like:

The water meter project uses computer vision to manage water use on the farm.I can use computer vision to determine if water is flowing or not, readthe gallons used per hour or per day, and save some of the images for analysis.The project also watches for unusual water flow due to leaks or broken irrigationcontrols and sends alerts. When the water is flowing, the large analog needlespins clockwise. Each full rotation of the needle causes the rightmostdigit of the digital meter to advance by one digit. The small "blue star" dialis a "leak detector" that spins even when a very small amount of water isflowing (like a dripping faucet).

The Raspberry Pi sits in a mason jar on top of the water meter cover. ThePiCamera and the array of LED lights is underneath the water meter cover andaimed at the water meter face. Here is a picture of the water meter as seen bythe PiCamera:

For more details on the water meter camera electronics and buildout, seeWater Meter Camera Hardware Details.

Raspberry Pi nodes around the farm can monitor temperature and detect motionof critters wandering about. Here is a log that shows motion detected behindthe barn, along with a couple of pictures that were taken when the coyoteactivated the motion detection in theimagenode RPi running in the barn:

Here is what the back of the barn looks like with the infrared "PiNoir"style PiCamera, a temperature sensor and the infrared floodlight that lights thearea after dark without putting out white light:

For more details on the infrared camera, infrared floodlight and temperaturesensor, seeCritter Infrared Camera and Temperature Sensor Details.

Another PiCameraimagenode watches the driveway and entrance area. It seesthe mail truck come and go, and spots an occasional hawk. It uses a RaspberryPi Zero W computer and a PiCamera that are encased in a "fake security camera"housing that cost about $5:

And here is what it looks like assembled and mounted in our driveway. You cansee the PiCamera behind the housing lens:

For more details on the Pi Zero based driveway camera and its enclosure,including the assembly pictures and some "action shots", seeDriveway Camera Hardware Example.

Theyin-yang-ranch projects are in early development and testing.Prototypes for all the modules in the design diagram above are working, and theearly experiments have provided a lot of data to help with designchanges and code refactoring. I have pushed theimageZMQ,imagenodeandimagehub as separate repositories on GitHub (see links above).

Thelibrarian and its communications programs have prototypes in thisrepository. They are documentedhere.Thelibrarian is currently being refactored with a new design, but theprototype is what was used to generate the SMS texting example above. It hasbeen running for about 3 years. It will eventually be pushed to its ownGitHub repository.

TheimageZMQ repositorycontains test programs that show how images can be sent from multiple RaspberryPi computers simultaneously to a hub computer. Theimagenode andimagehubprograms are evolutions of theimageZMQ test programstiming_send_images.pyandtiming_receive_imnages.py.The Python code in those two programs is a brief "pseudo code" outline for thecode that is in theimagenode andimagehubprograms. Links to the fullimagenode andimagehub repositories areabove.

Theyin-yang-ranch projects are in very early development and testing. Iwelcome questions and comments. The easiest way to make a comment or ask aquestion about the project is to open an issue. If your issue is specific toimageZMQ,imagenode orimagehub, it will be easiest if you openan issue in the appropriate project. Issues about the overall project design orabout my PyCon 2020 presentation should go into this repository.

Animagenode &imagehub user and code contributor @sbkirby has designeda completely different approach to building animagehub andlibrariancombination using a broad mix of tools in addition to Python including Node-Red,MQTT, MariaDB and OpenCV in Docker containers. He has posted it in thisGithub repository.I like his approach a lot, although I'm still working on a mostly Pythonapproach to my own librarian that is an extension of the prototype librarian inthis repository.

• The Raspberry Pi Foundation and their remarkable Raspberry Pi tiny singleboard computers. Even their $10 Pi Zero runs Linux and OpenCV and can do seriouscomputer vision image acquisition and processing.Raspberry Pi Foundation
• Adafruit an amazing resource for electronics makers with helpful tutorials andelectronic components of all kinds.Adafruit
• ZeroMQ is a great network messaging library with great documentationatZeroMQ.org.
• OpenCV and its Python bindings provide great scaffolding for computervision projects large or small:OpenCV.org.