- Raspberry Pi Real-Time Object Detection and Tracking
- 1. Introduction
- 2. Dependency
- 3. What's in this repository
- 3.1. Camera Test
- 3.2. Camera Streaming over HTTP
- 3.3. Motion Detection
- 3.4. Color-based Object Detection and Tracking
- 3.5. Shape-based Object Detection and Tracking
- 3.6. Feature-based Object Detection and Tracking (with ORB)
- 3.7. Face Detection and Tracking
- 3.8. Object Detection using YOLO
- 3.9. Object Detection using TensorFlow Lite
- 4. How to Run
- 5. Benchmark
- 6. Q&A
- License
Table of contents generated with markdown-toc
Using a Raspberry Pi and a camera module for computer vision with OpenCV, YOLO, and TensorFlow Lite. The aim of this project is to provide a starting point for using RPi & CV in your own DIY / maker projects. Computer vision based on cameras is very powerful and will bring your project to the next level. This allows you to track complicated objects that would otherwise not be possible with other types of sensors (infrared, ultrasonic, LiDAR, etc).
Note the code is based on Python and OpenCV meaning it is cross-platform. You can run this on other Linux-based platforms as well, e.g. x86/x64 PC, IPC, Jetson, Banana Pi, LattaPanda, BeagleBoard, etc.
This project is dependent on the following packages:
- Python >= 3.6.9 (tested with Python 3.12 and 3.13)
- OpenCV-Python
- OpenCV-Contrib-Python
- NumPy
- SciPy
- Matplotlib
- Ultralytics (with CPU-only PyTorch on Raspberry Pi)
ai-edge-litert(optional, for the TensorFlow Lite example)
- Support Raspberry Pi 1 Model B, Raspberry Pi 2, Raspberry Pi Zero, and Raspberry Pi 3/4/5 (preferable)
- Raspberry Pi 5, Raspberry Pi 4 (on Compute Module 5 and Compute Module 4), Raspberry Pi 3 Model B and Raspberry Pi Zero 2 W have been tested, see 2.4. Tested platforms.
- Both Raspberry Pi OS and Ubuntu are supported.
- Different boards will have very varied performances.
- RPi 3/4/5 are preferable as they have more powerful CPUs;
- RPi 1/2 may be struggling and produce very low FPS, in which case you can further reduce the camera resolution (160 x 120).
- Nvidia Jetson is supported on a best-effort basis: the examples run on a Jetson Nano (see 5.2. Results), but this project is built for the Raspberry Pi. For Jetson-specific features (TensorRT, GPU detection, CSI camera setup), use nvidia-jetson-object-detection.
- Any USB camera supported by Raspberry Pi
- To see a list of all supportive cameras, visit http://elinux.org/RPi_USB_Webcams
- The official RPi camera module is supported through
Picamera2.- All scripts use
Picamera2automatically when it is installed (sudo apt install python3-picamera2on Raspberry Pi OS), and fall back to a USB camera otherwise. - On a Compute Module, the camera has to be declared in
/boot/firmware/config.txt. For example, for a Camera Module v2 (IMX219) on theCAM1connector, setcamera_auto_detect=0and adddtoverlay=imx219,cam1, then reboot. Useov5647for Camera Module v1,imx708for v3, andimx477for the HQ camera. Runrpicam-hello --list-camerasto check that the camera is detected. - On Ubuntu,
python3-picamera2is not available from the Ubuntu archive, so use a USB camera.
- All scripts use
- CSI cameras on Nvidia Jetson are opened automatically through GStreamer. This needs JetPack's own OpenCV, not
pip install opencv-python; see nvidia-jetson-object-detection for setup.
All the examples have been tested on the following setups:
| Board | OS | Python | Camera |
|---|---|---|---|
| Raspberry Pi Compute Module 5 (RPi 5) | Ubuntu 24.04 LTS | 3.12 | Sony PlayStation Eye (USB) |
| Raspberry Pi Compute Module 4 (RPi 4) | Raspberry Pi OS (Debian 13 trixie) | 3.13 | Camera Module v2.1 (IMX219) on CAM1 |
| Raspberry Pi 3 Model B (RPi 3) | Raspberry Pi OS (Debian 13 trixie) | 3.13 | Camera Module v1 (OV5647) |
| Raspberry Pi Zero 2 W | Raspberry Pi OS Lite (Debian 13 trixie) | 3.13 | Camera Module v2 (IMX219) |
| Nvidia Jetson Nano 4GB | JetPack 4.6.1 (Ubuntu 18.04) | 3.8 (see 5.2) | — (processing FPS only) |
See 5. Benchmark for the frame rate of each example on these boards.
Currently, the following applications are implemented:
src/camera-test: Test if the camera is workingsrc/camera-stream: Stream the camera to a web browsersrc/motion-detection: Detect any motion in the framesrc/object-tracking-color: Object detection & tracking based on colorsrc/object-tracking-shape: Object detection & tracking based on shapesrc/object-tracking-feature: Object detection & tracking based on features using ORBsrc/face-detection: Face detection & trackingsrc/object-detection-yolo: Object detection using YOLO (RPi 3/4/5 only)src/object-detection-tflite: Object detection using TensorFlow Lite (RPi 3/4/5 only)
Test the RPi and OpenCV environment. You are expected to see a pop-up window that has video streams from your USB camera if everything is set up correctly. If the window does not appear, you need to check both of (1) your environment; (2) camera connection.
Stream the camera as MJPEG over HTTP, so you can watch it from any browser on the same network, without a display attached to the Pi. With the Raspberry Pi camera, frames are JPEG-encoded by the Pi's hardware encoder through Picamera2, so CPU usage stays low. With a USB camera, frames are read and JPEG-encoded with OpenCV.
python3 src/camera-stream/cv_camera_stream.py
Then open http://<raspberry-pi-ip>:8000/ in a browser (the raw stream is at /stream.mjpg, which can also be opened in VLC). The script runs in the foreground by default. To keep it running in the background after you log out (like nohup), add --nohup:
python3 src/camera-stream/cv_camera_stream.py --nohup
It prints the process ID to stop it with (kill <PID>), and writes its output to camera_stream.log in the current directory. The resolution and port are set by IMAGE_WIDTH, IMAGE_HEIGHT and PORT at the top of the script. If a USB camera does not support the requested resolution, it uses the closest one it has, and the script prints the resolution it actually streams at. Tested at about 30 FPS with a Camera Module v2 (IMX219) on a Compute Module 4 at 1280 x 720, and with a PlayStation Eye USB camera on a Compute Module 5 at 640 x 480.
Like the other examples, the script uses the Raspberry Pi camera when Picamera2 is installed and a camera is connected, and the USB camera otherwise. The Raspberry Pi camera does not need OpenCV. In the foreground, press Ctrl+C to stop it. The stream has no authentication, so anyone on your network can view it. Only one program can use the camera at a time, so stop the stream before running the other examples.
Detect object movements in the camera frame and print a warning message when motion is sustained.
Pipeline: the current and previous frames are converted to grayscale, Gaussian-blurred to suppress sensor noise, then compared with a per-pixel cv2.absdiff. The resulting diff is thresholded, dilated, and passed through cv2.findContours; only connected regions with area ≥ MIN_MOTION_AREA count as motion, and a debounce (MOTION_CONSECUTIVE_FRAMES) rejects one-frame transients. All thresholds are constants at the top of the script, so you can trade sensitivity for false-positive rate without touching the loop.
Track an object based on its color in HSV and print its center position. You can choose your own color by clicking on the object of interest. Click multiple times on different points so a full color space is covered. You can hard code the parameters so you don't need to pick them again for the next run. The following demo shows how I track a Nintendo game controller in real-time:
Detect and track geometric shapes in the frame:
- Circles via
cv2.HoughCircles() - Triangles and rectangles via contour approximation (
cv2.findContours()+cv2.approxPolyDP())
Contours below MIN_CONTOUR_AREA pixels are filtered out to reduce noise; the epsilon used for polygon approximation and the Canny edge thresholds are configurable at the top of the script.
Detect and track an object using its feature. The algorithm I selected here is ORB (Oriented FAST and Rotated BRIEF) for its fast calculation speed to enable real-time detection. To use the example, please prepare an Arduino UNO board in hand (or replace the simple.png).
Detecting face using Harr Cascade detector.
Use YOLO (You Only Look Once) for object detection.
Note this code is based on Ultralytics YOLO. The instruction can be found at their website: Quick Start Guide: Raspberry Pi with Ultralytics YOLO11. Double check if you need to use it in a commercialised project!
Use TensorFlow Lite to recognise objects.
To use it, download the model and labels into the module directory:
cd src/object-detection-tflite
# COCO labels
wget https://raw.githubusercontent.com/google-coral/test_data/master/coco_labels.txt
# EfficientDet-Lite0 model (INT8 quantized, ~4MB, with built-in NMS post-processing)
wget -O efficientdet_lite0.tflite https://storage.googleapis.com/download.tensorflow.org/models/tflite/task_library/object_detection/rpi/lite-model_efficientdet_lite0_detection_metadata_1.tflite
Then install the TensorFlow Lite runtime and run:
pip install ai-edge-litert
python3 src/object-detection-tflite/object_detection_tflite.py
Run the installation script. It creates a virtual environment under ./venv and installs all Python dependencies from requirements.txt:
chmod +x install.sh
./install.sh
On Raspberry Pi, the venv is created with --system-site-packages so that libcamera and picamera2 (installed in the system Python by Raspberry Pi OS) remain accessible from inside it. The script also installs the CPU-only build of PyTorch first, because the default PyPI build for ARM64 pulls in several GB of CUDA packages that a Raspberry Pi cannot use.
The face detection example uses the Haar cascade files from the libopencv-dev system package (installed by install.sh), since the OpenCV 5 pip wheels no longer ship them.
The examples need opencv-contrib-python: in OpenCV 5, face detection's CascadeClassifier is only in the contrib build. Ultralytics depends on opencv-python, which installs into the same cv2 folder, so whichever of the two is installed last wins. install.sh therefore removes opencv-python and reinstalls opencv-contrib-python at the end (pip check then reports that ultralytics requires opencv-python; this is expected and harmless). If you later upgrade ultralytics, or install the packages yourself, and face detection reports that OpenCV has no CascadeClassifier, run:
pip uninstall -y opencv-python
pip install --force-reinstall --no-deps opencv-contrib-python
Activate the environment in your shell before running any of the scripts:
source venv/bin/activate
The TensorFlow Lite example (§3.9) has an extra optional dependency — install it into the same venv only if you plan to use it:
pip install ai-edge-litert
Run scripts in the /src folder by:
python3 src/$FOLDER_NAME$/$SCRIPT_NAME$.py
To stop the code, press the ESC key on your keyboard.
Changing the resolution will significantly impact the FPS. By default it is set to be 320 x 240, but you can change it to any value that your camera supports at the beginning of each source code (defined by IMAGE_WIDTH and IMAGE_HEIGHT). Typical resolutions are:
- 160 x 120
- 320 x 240
- 640 x 480 (480p)
- 1280 x 720 (720p)
- 1920 x 1080 (1080p: make sure your camera supports this high resolution.)
benchmark/benchmark_fps.py runs each example for a fixed time and reports its average frame rate as a Markdown table. It counts the frames each example prints, skips a warm-up period (model loading, camera start-up), and stops each example before starting the next one.
python3 benchmark/benchmark_fps.py
Each example is measured in two ways:
- Camera FPS: the example runs as normal on the live camera. This is the frame rate you see in practice, but it can never exceed the camera's frame rate (often 30 FPS), and several examples also wait 30 ms per frame in
cv2.waitKey(). - Processing FPS: a short clip is recorded from the camera first, then fed to each example as fast as it can take frames, with no windows and no
waitKey()delay. This shows how much processing headroom the board has, and the time per frame shows the cost of each algorithm. The examples' own code is not changed: the benchmark replaces the camera with the clip when it starts them.
See 5.2. Results for results on the RPi 5, RPi 4, RPi 3, RPi Zero 2 W and Jetson Nano. Processing FPS does not include drawing the windows, which also takes time when a display is attached. Camera FPS also depends on the lighting, as many cameras lower their frame rate in low light.
Options:
--mode camera/--mode processing: measure only one of the two (default: both).--video clip.mp4: use the first frames of a video file for processing FPS instead of recording from the camera, so no camera is needed with--mode processing.--headless: do not open any windows for camera FPS, so it can be run over SSH without a display. Frame timing is kept the same as with a display. (Processing FPS never opens windows.)--only face yolo: run only some of the examples (motion-detection,color,shape,orb,face,tflite,yolo).--duration 20/--warmup 5: seconds measured per example, and seconds skipped after its first frame.--verbose: show the output of each example.
To draw the results as a chart, put them in benchmark/results.csv (one row per example, one column per board; a cell is the FPS, the FPS followed by gpu if it ran on the GPU, oom if it ran out of memory, or empty if not measured) and run:
python3 benchmark/plot_results.py
This saves a grouped bar chart to images/benchmark_fps.png, plus images/benchmark_fps_dark.png for dark backgrounds (--theme light or --theme dark for only one, --csv and --out for other files).
The TensorFlow Lite example is skipped if its model and labels have not been downloaded (see 3.9). The camera test (§3.1) and camera streaming (§3.2) examples are not included, as they do no processing.
Measured with the benchmark script (20 s per example after a 5 s warm-up, OpenCV 5.0). See 5.1. Run the benchmark to measure your own board.
Processing FPS shows how fast each board runs the example's processing, without the camera's frame rate limit (time per frame in brackets). So that the boards can be compared directly, every board was fed the same 640 x 480 clip, recorded once with the PlayStation Eye (python3 benchmark/benchmark_fps.py --mode processing --video clip.avi):
The same results as a table:
| Example | RPi 5 (CM5) | RPi 4 (CM4) | RPi 3 | RPi Zero 2 W | Jetson Nano |
|---|---|---|---|---|---|
| Motion Detection | 2318 FPS (0.4 ms) | 645 FPS (1.6 ms) | 362 FPS (2.8 ms) | 321 FPS (3.1 ms) | 697 FPS (1.4 ms) |
| Color-based Tracking | 1573 FPS (0.6 ms) | 440 FPS (2.3 ms) | 251 FPS (4.0 ms) | 222 FPS (4.5 ms) | 467 FPS (2.1 ms) |
| Shape-based Tracking | 577 FPS (1.7 ms) | 202 FPS (4.9 ms) | 119 FPS (8.4 ms) | 110 FPS (9.1 ms) | 295 FPS (3.4 ms) |
| Feature-based Tracking (ORB) | 53.6 FPS (18.7 ms) | 24.3 FPS (41.2 ms) | 13.0 FPS (76.9 ms) | 11.1 FPS (90.4 ms) | 23.9 FPS (41.8 ms) |
| Face Detection | 164 FPS (6.1 ms) | 79.1 FPS (12.6 ms) | 42.5 FPS (23.5 ms) | 36.9 FPS (27.1 ms) | 75.7 FPS (13.2 ms) |
| TensorFlow Lite (EfficientDet-Lite0) | 33.2 FPS (30.2 ms) | 12.3 FPS (81.6 ms) | 6.1 FPS (165 ms) | 5.1 FPS (194 ms) | 8.6 FPS (116 ms) |
| YOLO11n (Ultralytics, 640x480) | 2.1 FPS (475 ms) | 1.3 FPS (765 ms) | 0.6 FPS (1578 ms) | out of memory | 7.1 FPS (140 ms, GPU) |
Notes on these results:
- RPi Zero 2 W and YOLO11n: YOLO11n needs about 430 MB of RAM to process one 640 x 480 frame (peak, measured on the RPi 3), but the Zero 2 W has only 512 MB in total (about 290 MB free with no desktop running), so it runs out of memory. Use the TensorFlow Lite example on the Zero 2 W instead.
- Jetson Nano (for comparison): all the examples run on the CPU except YOLO11n, which runs on the GPU. Measured with Python 3.8 and the Ultralytics PyTorch builds for JetPack 4. For a faster TensorRT setup, see nvidia-jetson-object-detection.
Camera FPS is the frame rate you see when running the examples on the live camera (python3 benchmark/benchmark_fps.py --headless, RPi 5 and RPi 4 only):
| Example | RPi 5 (CM5) | RPi 4 (CM4) |
|---|---|---|
| Motion Detection | 30.0 FPS | 33.2 FPS |
| Color-based Tracking | 29.1 FPS | 27.5 FPS |
| Shape-based Tracking | 28.0 FPS | 26.4 FPS |
| Feature-based Tracking (ORB) | 30.0 FPS | 22.6 FPS |
| Face Detection | 26.6 FPS | 22.8 FPS |
| TensorFlow Lite (EfficientDet-Lite0) | 30.0 FPS | 11.2 FPS |
| YOLO11n (Ultralytics, 640x480) | 2.1 FPS | 0.8 FPS |
Camera FPS is limited by different things:
- The camera: the PlayStation Eye runs at up to 30 FPS, and the Camera Module v2 at about 33 FPS at 320 x 240. Motion detection reaches the camera's frame rate on both boards.
- The
cv2.waitKey()delay: color and shape tracking wait 33 ms per frame, which keeps them below 30 FPS. Face detection waits 30 ms per frame, which, with 12.7 ms of detection on the RPi 4, gives about 23 FPS. - The CPU: ORB and TensorFlow Lite on the RPi 4, and YOLO on both boards. For these, camera FPS is close to processing FPS.
Camera FPS also depends on the lighting. In a dim room (about 16 lux), the Camera Module v2 lengthened its exposure to 66 ms per frame and ran at only 15 FPS, which capped all the fast examples at 15 FPS. The RPi 4 results above were measured with the room light on (about 240 lux).
Q: Does this support Nvidia Jetson?
A: Yes. I have tested with my Jetson Nano 4GB. Note that Jetson has its own repo now: nvidia-jetson-object-detection.
Q: Does this support the Raspberry Pi camera?
A: This is implemented in issue [#16].
Q: Does this support Raspberry Pi 5?
A: Yes. All examples have been tested on a Raspberry Pi Compute Module 5, which uses the same processor as the Raspberry Pi 5. See 2.4. Tested platforms.
Q: Can we run this project on Ubuntu?
A: Yes. All examples have been tested on Ubuntu 24.04 LTS on a Raspberry Pi Compute Module 5 with a USB camera. picamera2 is not available from the Ubuntu archive, so use a USB camera on Ubuntu.
Q: I am using virtual environment and get a message "no module called libcamera" issue
A: A simple solution would be (Reference):
python3 -m venv --system-site-packages env
Thanks VgerTest for issue [#20].
© This source code is licensed under the MIT License.