Prerequisites#

To follow this tutorial you’ll need a Raspberry Pi 4 or 5, a camera for it and allthe other standard accessories.

• Raspberry Pi 4 Model B 2GB+

• Raspberry Pi Camera Module

• Heat sinks and Fan (optional but recommended)

• 5V 3A USB-C Power Supply

• SD card (at least 8gb)

• SD card read/writer

Raspberry Pi Setup#

PyTorch only provides pip packages for Arm 64bit (aarch64) so you’ll need to install a 64 bit version of the OS on your Raspberry Pi

You’ll need to install theofficial rpi-imager to install Rasbperry Pi OS.

32-bit Raspberry Pi OS will not work.

Installation will take at least a few minutes depending on your internet speed and sdcard speed. Once it’s done it should look like:

Time to put your sdcard in your Raspberry Pi, connect the camera and boot it up.

# This enables the extended features such as the camera.start_x=1# This needs to be at least 128M for the camera processing, if it's bigger you can just leave it as is.gpu_mem=128

Installing PyTorch and picamera2#

PyTorch and all the other libraries we need have ARM 64-bit/aarch64 variants so you can just install them via pip and have it work like any other Linux system.

$sudoaptinstall-ypython3-picamera2python3-libcamera$pipinstalltorchtorchvision--break-system-packages

We can now check that everything installed correctly:

$python-c"import torch; print(torch.__version__)"

Video Capture#

Test the camera is working first, by runninglibcamera-hello in a terminal.

For video capture we’re going to be using picamera2 to capture the video frames.

The model we’re using (MobileNetV2) takes in image sizes of224x224 so wecan request that directly from picamera2 at 36fps. We’re targeting 30fps for themodel but we request a slightly higher framerate than that so there’s alwaysenough frames.

frompicamera2importPicamera2picam2=Picamera2()# print available sensor modesprint(picam2.sensor_modes)config=picam2.create_still_configuration(main={"size":(224,224),"format":"BGR888",},display="main")picam2.configure(config)picam2.set_controls({"FrameRate":36})picam2.start()

To capture the frames we can callcapture_image to return aPIL.Imageobject that we can use with PyTorch.

# read frameimage=picam2.capture_image("main")# show frame for testingimage.show()

This data reading and processing takes about3.5ms.

Image Preprocessing#

We need to take the frames and transform them into the format the model expects. This is the same processing as you would do on any machine with the standard torchvision transforms.

fromtorchvisionimporttransformspreprocess=transforms.Compose([# convert the frame to a CHW torch tensor for trainingtransforms.ToTensor(),# normalize the colors to the range that mobilenet_v2/3 expecttransforms.Normalize(mean=[0.485,0.456,0.406],std=[0.229,0.224,0.225]),])input_tensor=preprocess(image)# The model can handle multiple images simultaneously so we need to add an# empty dimension for the batch.# [3, 224, 224] -> [1, 3, 224, 224]input_batch=input_tensor.unsqueeze(0)

Model Choices#

There’s a number of models you can choose from to use with different performancecharacteristics. Not all models provide aqnnpack pretrained variant so fortesting purposes you should chose one that does but if you train and quantizeyour own model you can use any of them.

We’re usingmobilenet_v2 for this tutorial since it has good performance andaccuracy.

Raspberry Pi 4 Benchmark Results:

MobileNetV2: Quantization and JIT#

For optimal performance we want a model that’s quantized and fused. Quantizedmeans that it does the computation using int8 which is much more performant thanthe standard float32 math. Fused means that consecutive operations have beenfused together into a more performant version where possible. Commonly thingslike activations (ReLU) can be merged into the layer before (Conv2d)during inference.

The aarch64 version of pytorch requires using theqnnpack engine.

importtorchtorch.backends.quantized.engine='qnnpack'

For this example we’ll use a prequantized and fused version of MobileNetV2 that’s provided out of the box by torchvision.

fromtorchvisionimportmodelsnet=models.quantization.mobilenet_v2(pretrained=True,quantize=True)

We then want to jit the model to reduce Python overhead and fuse any ops. Jit gives us ~30fps instead of ~20fps without it.

net=torch.jit.script(net)

Putting It Together#

We can now put all the pieces together and run it:

importtimeimporttorchfromtorchvisionimportmodels,transformsfrompicamera2importPicamera2torch.backends.quantized.engine='qnnpack'picam2=Picamera2()# print available sensor modesprint(picam2.sensor_modes)config=picam2.create_still_configuration(main={"size":(224,224),"format":"BGR888",},display="main")picam2.configure(config)picam2.set_controls({"FrameRate":36})picam2.start()preprocess=transforms.Compose([transforms.ToTensor(),transforms.Normalize(mean=[0.485,0.456,0.406],std=[0.229,0.224,0.225]),])net=models.quantization.mobilenet_v2(pretrained=True,quantize=True)# jit model to take it from ~20fps to ~30fpsnet=torch.jit.script(net)started=time.time()last_logged=time.time()frame_count=0withtorch.no_grad():whileTrue:# read frameimage=picam2.capture_image("main")# preprocessinput_tensor=preprocess(image)# create a mini-batch as expected by the modelinput_batch=input_tensor.unsqueeze(0)# run modeloutput=net(input_batch)# do something with output ...print(output.argmax())# log model performanceframe_count+=1now=time.time()ifnow-last_logged>1:print(f"{frame_count/(now-last_logged)} fps")last_logged=nowframe_count=0

Running it shows that we’re hovering at ~30 fps on a Raspberry Pi 4 and ~41 fps on a Raspberry Pi 5.

This is with all the default settings in Raspberry Pi OS. If you disabled the UIand all the other background services that are enabled by default it’s moreperformant and stable.

If we checkhtop we see that we have almost 100% utilization.

To verify that it’s working end to end we can compute the probabilities of theclasses anduse the ImageNet class labelsto print the detections.

top=list(enumerate(output[0].softmax(dim=0)))top.sort(key=lambdax:x[1],reverse=True)foridx,valintop[:10]:print(f"{val.item()*100:.2f}%{classes[idx]}")

mobilenet_v3_large running in real time:

Detecting an orange:

Detecting a mug:

Troubleshooting: Performance#

PyTorch by default will use all of the cores available. If you have anythingrunning in the background on the Raspberry Pi it may cause contention with themodel inference causing latency spikes. To alleviate this you can reduce thenumber of threads which will reduce the peak latency at a small performancepenalty.

torch.set_num_threads(2)

Forshufflenet_v2_x1_5 using2threads instead of4threadsincreases best case latency to72ms from60ms but eliminates thelatency spikes of128ms.

Next Steps#

You can create your own model or fine tune an existing one. If you fine tune onone of the models fromtorchvision.models.quantizedmost of the work to fuse and quantize has already been done for you so you candirectly deploy with good performance on a Raspberry Pi.

See more:

• Quantization for more information on how to quantize and fuse your model.

• Transfer Learning Tutorialfor how to use transfer learning to fine tune a pre-existing model to your dataset.