Robotics / Case study
Camera images and LiDAR measurements brought together to locate the cones that define a Formula Student track.
01 / Contribution
I worked on the StereoLabs ZED camera pipeline and its integration with UPM Racing’s perception stack. My 2023 degree thesis followed the camera path from a 2D bounding box to a colored cone in 3D and examined how those estimates lined up with LiDAR detections.
Teammates developed the LiDAR processing. The shared ROS stack combined the two streams and published cone observations for downstream planning and mapping.
02 / ROS pipeline
The camera provided color and depth estimates. LiDAR provided positions from point-cloud clusters.
Camera
zed_ros_wrapper supplied the image and depth map. YOLOv3-tiny via darknet_ros produced color classes and bounding boxes. The cam_cones_publisher node sampled depth at the center of each box and back-projected it using the camera intrinsics.
ROS output /CamConeList
LiDAR
The Ouster OS1 driver supplied the point cloud. After prefiltering and LineFit ground removal, the clustering node grouped the remaining points and estimated a center for each candidate cone.
ROS output /LidarConeList
Fusion
Calibration used at least four cones seen by both sensors to estimate the rotation and translation into the LiDAR frame.
ROS output /ground_truth/cones
The historical topic name refers to estimated cone observations.
Design decision
With no additional sensors available, we wanted either sensing path to remain useful on its own. We explored projecting camera images onto the LiDAR point cloud, but also kept independent 3D cone detections from both sensors. Matching those detections let us combine LiDAR’s more precise positions with camera color while retaining a fallback if either sensor failed.
The intended fallback was straightforward: without LiDAR, stereo vision could still supply cone positions and colors, with lower positional accuracy. Without cameras, LiDAR could still locate cones. After a completed lap, the SLAM map could retain the previously observed cone information; on the first lap, planning would need to infer the track without color labels. The simple association step supported that redundancy-first design.
03 / Original thesis evidence
In one controlled setup, four cones were placed 3–6 m from the LiDAR. The plotted camera detections were more dispersed than the LiDAR clusters. The sensors also measured different points on each cone. LiDAR used the cluster center, while the camera sampled a point on the visible surface.
When boxes overlapped, the center of a rear cone’s box could land on the foreground cone, giving it the wrong depth. The thesis proposed matching cone detections between the left and right images before estimating depth as a follow-up.
04 / Interactive reconstruction
This browser example uses synthetic 2D points in a shared frame to model the thesis’s association step. Adjust the limits to explore which pairs pass.
Interactive example · synthetic readings
Find the nearest camera detection for each LiDAR return. Both tolerances must pass for the pair to combine the LiDAR position with the camera color.
The plot shows both sensor streams. Enable JavaScript to adjust the tolerances and see which pairs match.
Simplified 2D view after transforming the camera readings into the LiDAR frame.
The historical case study draws on Juan Segura Núñez de Arenas’s degree thesis, February 2023, pp. 19–20, 33–39, 44–50, 53, and 61–66.