Repository files navigation

A Rock-Paper-Scissors game using computer vision and machine learning on Raspberry Pi.

By Julien de la Bruère-Terreault (drgfreeman@tuta.io)

Click on image to accessvideo on YouTube.

• See myDrGFreeman/rps-cv-data-science repository where I posted different notebooks demonstrating some cool data science analysis on the image dataset resulting from this project.

• MagPi article in Simplified Chinese contributed byTommy Zihao

This project results from a challenge my son gave me when I was teaching him the basics of computer programming making a simple text based Rock-Paper-Scissors game in Python. At that time I was starting to experiment with computer vision with a Raspberry Pi and an old USB webcam so my son naively asked me:

"Could you make a Rock-Paper-Scissors game that uses the camera to detect hand gestures?"

I accepted the challenge and about a year and a lot of learning later, I completed the challenge with a functional game.

The game uses a Raspberry Pi computer and Raspberry Pi camera installed on a 3D printed support with LED strips to achieve consistent images.

The pictures taken by the camera are processed and fed to an image classifier that determines whether the gesture corresponds to "Rock", "Paper" or "Scissors" gestures.

The image classifier uses aSupport Vector Machine, a class ofmachine learning algorithm. The image classifier has been priorly "trained" with a bank of labeled images corresponding to the "Rock", "Paper", "Scissors" gestures captured with the Raspberry Pi camera.

The image below shows the processing pipeline for the training of the image classifier (top portion) and the prediction of gesture for new images captured by the camera during the game (bottom portion). Clickhere for full size image.

The project depends on and has been tested with the following libraries:

• OpenCV >= 3.3.0 with bindings for Python 3*
• Python >= 3.4+
• Numpy >= 1.13.0
• Scikit-Learn >= 0.18.2
• Scikit-Image >= 0.13.0
• Pygame >= 1.9.3
• Picamera

* Followthis guide for installation of OpenCV on the Raspberry Pi. Install Python libraries within the same virtual environment as OpenCV using thepip install <package_name> command. Picamera is installed by default onRaspbian images.

• Raspberry Pi 3 Model B or B+ computer
• Raspberry Pi Camera Module V2
• A green background to allow background subtraction in the captured images.
• A physical setup for the camera to ensure consistent lighting and camera position. The 3D models I used are availableon Thingiverse.

• capture.py
  This file opens the camera in "capture mode", to capture and label images that will later be used to train the image classifier. The captured images are automatically named and stored in a folder structure.

• train.py
  This script reads and processes the training images in preparation for training the image classifier. The processed image data is then used to train the support vector machine image classifier. The trained classifier is stored in theclf.pkl file read byplay.py.

• playgui.py
  This file runs the actual Rock-Paper-Scissors game using the camera and the trained image classifier in a graphical user interface (GUI). Images from each play are captured and added to the image bank, creating additional images to train the classifier.

• play.py
  This file runs the actual Rock-Paper-Scissors game similarly to playgui.py except the game output is done in the terminal and OpenCV window (no GUI).

* Note that the due to memory limitations on the Raspberry Pi, thetrain.py script may not run properly on the Raspberry Pi with training sets of more than a few hundred images. Consequently, it is recommended to run these on a more powerful computer. This computer must also have OpenCV, Python 3.4+ and the numpy, scikit-learn and scikit-image Python libraries installed.

• rpscv.gui
  This module defines the RPSGUI class and associated methods to manage the gamegraphical user interface (GUI).

• rpscv.imgproc
  This module provides the image processing functions used by the various other Python files.

• rpscv.utils
  This module provides functions and constants used by the various other Python files.

• rpscv.camera
  This module defines the Camera class, a wrapper around the picamera library, with specific methods for the project such as white balance calibration.

Typical output fromtrain.py (on PC with Intel Core I7-6700 @3.4GHz, 16GB RAM, Anaconda distribution):

(rps-cv) jul@rosalind:~/pi/git/rps-cv$ python train.py+0.0: Importing libraries+3.75: Generating image dataCompleted processing 1708 images  rock: 562 images  paper: 568 images  scissors: 578 images+99.51: Generating test set+99.64: Defining pipeline+99.64: Defining cross-validation+99.64: Defining grid searchGrid search parameters:GridSearchCV(cv=StratifiedKFold(n_splits=5, random_state=42, shuffle=True),       error_score='raise',       estimator=Pipeline(steps=[('pca', PCA(copy=True, iterated_power='auto', n_components=None,       random_state=None, svd_solver='auto', tol=0.0, whiten=False)), ('clf', SVC(C=1.0,       cache_size=200, class_weight=None, coef0=0.0, decision_function_shape=None, degree=3,       gamma='auto', kernel='rbf', max_iter=-1, probability=False, random_state=None, shrinking=True,       tol=0.001, verbose=False))]),       fit_params={}, iid=True, n_jobs=4,       param_grid={'clf__C': array([   1.     ,    3.16228,   10.     ,   31.62278,  100.     ]),                   'clf__gamma': array([ 0.0001 ,  0.00032,  0.001  ,  0.00316,  0.01   ]),                   'pca__n_components': [60]},       pre_dispatch='2*n_jobs', refit=True, return_train_score=True,       scoring='f1_micro', verbose=1)+99.64: Fitting classifierFitting 5 folds for each of 25 candidates, totalling 125 fits[Parallel(n_jobs=4)]: Done  42 tasks      | elapsed:  2.1min[Parallel(n_jobs=4)]: Done 125 out of 125 | elapsed:  5.9min finishedGrid search best score: 0.9910406616126809Grid search best parameters:  pca__n_components: 60  clf__C: 10.0  clf__gamma: 0.00031622776601683794+458.66: Validating classifier on test setClassifier f1-score on test set: 0.9922178988326849Confusion matrix:[[84  1  0] [ 1 84  0] [ 0  0 87]]Classification report:             precision    recall  f1-score   support       rock       0.99      0.99      0.99        85      paper       0.99      0.99      0.99        85   scissors       1.00      1.00      1.00        87avg / total       0.99      0.99      0.99       257+458.72: Writing classifier to clf.pkl+467.25: Done!

Initial screen:

Computer wins the play:

Player wins the play:

Tie:

Game over, player wins the game: