Repository files navigation

This repository is part of the camera course in the Sensor Fusion Nanodegree. In this project, I implemented several key components to track 3D objects over time, compute Time-to-Collision (TTC) using both Lidar and camera data, and test various detector/descriptor combinations.

This project involves the following tasks:

1. Matching 3D Objects Over Time
2. Computing TTC Based on Lidar Measurements
3. Computing TTC Based on Camera Measurements
4. Conducting Various Tests with the Framework

Developed an algorithm to match 3D objects over time using keypoint correspondences. This involved:

• Extracting keypoints and descriptors from successive frames.
• Matching these keypoints to track objects across frames.
• Implementation Details: ThematchBoundingBoxes method incamFusion_Student.cpp correctly assigns the matches based on counts. Check out line 285 for implementation details.

Implemented a method to compute the Time-to-Collision (TTC) using Lidar measurements. This process includes:

• Filtering Lidar points to focus on the region of interest.
• Calculating TTC using the relative velocity and distance of the objects.
• Implementation Details: The Lidar-based TTC uses the median distance instead of the closest Lidar point. Check out line 234 inmatching2D_Student.cpp for implementation.

For camera-based TTC computation:

• Associated keypoint matches to regions of interest (ROI).
• Computed TTC based on the keypoint matches within the ROI.
• Integrated this with the object tracking mechanism.
• Implementation Details:
  • Associate Keypoint Correspondences with Bounding Boxes: Check out line 137 inmatching2D_Student.cpp.
  • Compute Camera-based TTC: Check out line 175 inmatching2D_Student.cpp.

To identify the most suitable detector/descriptor combination for TTC estimation:

• Tested different combinations of detectors (e.g., SIFT, SURF, ORB) and descriptors.
• Analyzed the performance and reliability of these combinations.
• Investigated potential sources of errors from both camera and Lidar sensors.
• Observations:
  • The TTC Lidar jumped from 12.515 to 15.74 in one instance, indicating an erroneous prediction due to the constant velocity model assumption. This frame is right after another frame with a significant TTC drop, highlighting the limitations of disregarding acceleration.
  • For my experiments, the SHI TOMASI (detector) + BRISK (descriptor) combination gave stable and reliable results.
  • The Lidar predictions remained constant regardless of the descriptor/detector combination.
  • The ORB + BRIEF combination resulted in a Camera TTC of 130 s.

• src/: Source files for the project, including implementations for keypoint matching, TTC computation, and testing.
• dat/: contains weights-images/:contains KITTI Sequence (LiDAR + RGB Images)
• results/: Directory to store output results.
• CMakeLists.txt: CMake configuration file.
• README.md: Project documentation.

• cmake >= 2.8
  • All OSes:click here for installation instructions
• make >= 4.1 (Linux, Mac), 3.81 (Windows)
  • Linux: make is installed by default on most Linux distros
  • Mac:install Xcode command line tools to get make
  • Windows:Click here for installation instructions
• Git LFS
  • Weight files are handled usingLFS
  • Install Git LFS before cloning this Repo.
• OpenCV >= 4.1
  • This must be compiled from source using the-D OPENCV_ENABLE_NONFREE=ON cmake flag for testing the SIFT and SURF detectors.
  • The OpenCV 4.1.0 source code can be foundhere
• gcc/g++ >= 5.4
  • Linux: gcc / g++ is installed by default on most Linux distros
  • Mac: same deal as make -install Xcode command line tools
  • Windows: recommend usingMinGW

1. Clone this repo.
2. Download dat and images folders from the driveLink and place in the same repo.
3. Make a build directory in the top level project directory:mkdir build && cd build
4. Compile:cmake .. && make
5. Run it:./3D_object_tracking.

The table below presents Time-to-Collision (TTC) values calculated using different combinations of detectors and descriptors. The TTC values are provided for both Lidar and Camera data across various test scenarios.