A Review of Intel Real Sense-Based Obstacle Detection for Unmanned Ground Vehicles

Abstract

The process of obstacle detection involves the identification of obstacles that lie ahead in the route, Intel RealSense camera has been a prominent solution for identifying real-time applications in recent years. Autonomous Unmanned Ground Vehicle(UGVs) face a difficult time in detecting obstacles because of stationary objects, moving objects and adverse weather conditions. We need to get an accurate distance between the UGVs and obstacles, So Intel RealSense cameras use stereo vision to calculate the depth data. Based on D415, D435 and D455 Intel RealSense depth sensors, we have examined and compared various obstacle detection methods.

Introduction

I. INTRODUCTION

In the present era of autonomous technology, the Unmanned Ground Vehicles (UGVs) system is mostly used on highways, roads and parking lots, it is crucial to find things quickly that show up on the road out of nowhere. [1] These items include stationary objects such as branches, parked vehicles, road signs, etc. and dynamic objects such as pedestrians, wildlife, moving vehicles, objects falling from vehicles, etc.

The depth camera can detect obstacles and accurately measure the distance between the object and UGV on the path. However, the dynamic and unstructured nature of real-world environments poses formidable challenges for these autonomous vehicles. UGVs operate in a variety of adverse weather conditions, including rain, snow, fog and glare lights. Adverse weather can severely affect sensor performance, making it challenging to detect and respond to obstacles effectively.

The Intel RealSense cameras are significantly advanced in computer vision and depth detecting technology. The depth cameras can be used in a wide spectrum of real-time applications. The depth data goes beyond 2D imagery, it has the ability to perceive 3D images. In 3D scanning, they facilitate the creation of detailed, three-dimensional representations of objects and spaces.

This review paper explores an extensive analysis of Intel RealSense depth sensors, including models such as the D400 series (D415, D435, D455) each of which has its own depth sensing technologies. The D400 series is useful in robotics and augmented reality because it uses structured light projection and active infrared stereo for accurate 3D perception. This paper compares the efficiency of these sensors in detecting obstacles, considering their unique features like accuracy, frame rate, field of view, and range.

II. BACKGROUND WORKS

Obstacle detection is a critical task for autonomous vehicles, and many researchers have proposed various techniques to tackle obstacle detection. Two types of vision-based techniques have been developed, namely (a) Monocular vision-based technique (b) Stereo vision-based approach. In our survey, we will give a review of the Stereo vision-based Intel RealSense cameras.

A. Monocular vision-based Technique

Monocular vision is a type of vision that uses only one camera [5,6]. Monocular vision-based techniques are methods that use monocular images to extract information about the 3D world.

There are several different monocular vision-based vehicle detection algorithms. Some of the most common algorithms rely on features such as shape, texture, and color to detect obstacles. [12] Introduces a novel approach, emphasizing simplicity and clarity by introducing a straightforward approach in monocular vision-based vehicle detection, emphasizing ease of application and understanding. However, a potential disadvantage lies in the limited depth perception inherent to monocular vision, which may impact the accuracy of distance estimations in obstacle detection.[13] The proposed distance estimation method for in-car cameras stands out for its robustness, leveraging attitude angle information and deep learning to achieve superior accuracy across diverse traffic scenarios. However, potential disadvantages include the reliance on complex deep learning frameworks, which may pose challenges in terms of computational efficiency and model training requirements.

Considering the limitations of monocular vision, such as restricted depth perception, narrow field of view, and susceptibility to varying lighting conditions. Stereo vision, employing two cameras, enhances depth perception and expands the field of view, addressing the shortcomings of monocular systems. This dual-camera setup provides more robust and accurate spatial information, improving obstacle detection and overall reliability in diverse driving environments.

B. Stereo vision Based Technique

Stereo vision is a technique used in computer vision and 3D reconstruction that simulates the human binocular vision system. It involves the use of two or more cameras to capture images of the same scene from different viewpoints simultaneously. [7] These images are then processed to extract in-depth information and create a three-dimensional understanding of the environment. By analyzing the disparities between corresponding points in the images, stereo vision enables the calculation of the distance to objects in the scene. This technique relies on the principle of triangulation, where the relative positions and orientations of the cameras are used to determine the depth of each point in the scene. In stereo vision, the process begins with the calibration of the cameras to accurately determine their intrinsic and extrinsic parameters. Once calibrated, the images captured by the cameras are rectified to ensure that corresponding points lie on the same horizontal scan line. Matching algorithms are then employed to find corresponding points in the rectified images, using techniques like correlation-based matching or feature-based matching. By comparing the disparities between corresponding points, the depth of the scene is computed using the triangulation principle. The resulting depth map provides a comprehensive understanding of the 3D structure of the environment, enabling applications such as 3D reconstruction, object detection, and scene analysis.[14] The Instant Scene Modeler (iSM) leverages stereo vision, utilizing on-board sensors to rapidly generate calibrated photo-realistic 3D models of unknown environments for enhanced situational awareness in unmanned vehicles. The system's stereo image processing enables accurate depth perception, contributing to improved mission planning and execution.  [15] Obstacle detection (OD) techniques for intelligent ground vehicles, focusing on stereo vision and 2D/3D sensors. It categorizes various OD models based on stereo vision, emphasizing the importance of real-time performance, and assessing trade-offs between representation accuracy and computational efficiency in achieving comprehensive environmental perception for autonomous vehicles.

III. OBSTACLE DETECTION

This section assesses how efficiently the Intel RealSense cameras detect obstacles for unmanned ground vehicles and the working efficiency of each RealSense camera in different fields.

A. The Technology of Intel RealSense Cameras

RealSense cameras use a combination of stereo vision and infrared projection to create depth maps and 3D point clouds. Stereo vision works by comparing two images of the same scene taken from different angles. The infrared projector emits a pattern of light onto the scene, and the camera uses this pattern to calculate the distance to each object in the scene. RealSense cameras produce depth maps with high accuracy, even in challenging conditions such as low light and outdoor environments. They also have a long range and a wide field of view. The Intel RealSense open-source SDK is a set of tools and libraries that developers can use to build applications with RealSense cameras. Through SDK we can access the depth data and color data from RealSense cameras.

The D415 and D455 are compact and affordable RealSense cameras that are ideal for a variety of applications. They have a range of 1.2 meters and 2 meters, respectively, and a field of view of 90 degrees and 120 degrees, respectively. The D435 is a more powerful RealSense camera that is ideal for demanding applications such as robotics and augmented reality. It has a range of 10 meters and a field of view of 87 degrees.

The D415, D455, and D435 RealSense cameras use a custom Intel processor called the Vision Processor D4. The Vision Processor D4 is a high-performance processor that is specifically designed for depth sensing and computer vision applications. It can perform up to 36.6 million pixels per second of depth processing.

B. The working Principle of the RealSense cameras

The depth pixel value, which represents an object's distance, is established relative to a plane parallel to the camera's recording position rather than the actual physical distance of the object from the camera. This approach utilizes the camera's internal reference point as the baseline for estimating depth, allowing it to gauge the disparity between the captured images and infer the relative distances of objects within the scene. As a result, the depth measurements are determined based on the positional relationship between the camera and the objects in view, providing a perspective that aids in understanding the spatial layout of the environment.   

[9] The thermal solution guidance is crucial for enhancing obstacle detection in Intel RealSense Depth Cameras like D415, D455, and D435. Maintaining optimal temperatures for components ensures efficient operation in diverse environments. Passive cooling measures, such as using suitable materials for mounting brackets and heat sinks, contribute to stable performance, reducing the risk of overheating during prolonged use. This effective thermal management ensures accurate and reliable depth sensing, improving obstacle detection in real-world applications.                                      

The depth camera, based on infrared projection, plays a fundamental role in capturing depth information. This technology operates by projecting a pattern of infrared light onto the scene, which is then captured by the camera's infrared sensor. By analyzing the distortion of this pattern as it interacts with objects in the environment, the camera calculates the depth value for each pixel, generating a comprehensive depth map that accurately represents the 3D structure of the scene. In the D415 model, this infrared projection system works in conjunction with a precise global shutter, enabling the camera to capture synchronized depth and color frames with minimal motion blur. Similarly, the D435 and D455 incorporate sophisticated infrared projection mechanisms that allow for robust and accurate depth sensing, enhancing the overall depth perception and enabling the cameras to effectively capture intricate details of the surrounding environment [3,8,9].

Table 1. presents the specifications of the RGB frames of                       Table 2 present the specifications of the infrared

Table 3. presents the specifications of the Left and right imagers frames of the Intel RealSense cameras

IV. REVIEW OF THE EXPERIMENT

In this section, we present the preliminary results of our review based on research papers and documents. [13] Choosing Intel RealSense depth cameras over the described monocular vision method enhances inter-vehicle distance accuracy in ADAS by providing direct depth information, mitigating reliance on image-based estimation and improving performance across diverse vehicle types and attitudes.

The RealSense D415, an entry-level model, offers balanced performance with a depth range of 0.3m to 4m, making it suitable for close-range depth sensing tasks [11]. It is an ideal choice for applications not requiring high precision or long-range depth perception. Experimental findings confirm that while the D415 is primarily intended for tracking obstacles, gaming, and gesture recognition, it also serves effectively and economically as a 3D scanner [2]. The D415 excels in short-range tests, demonstrating superior reconstruction quality in capturing and rendering objects within close proximity. However, it exhibits limitations in accurately representing planar surfaces, indicating a potential weakness in handling certain depth-related tasks. [8] The D415 features a narrower field of view (around 64 degrees) and provides higher depth precision.

On the other hand, the RealSense D435 provides an extended depth range from 0.1m to 10m and supports a wider field of view compared to the D415. This broader perspective makes it more suitable for applications requiring capturing a larger scene in a single frame, such as obstacle detection in UGVs.

The D435 also offers higher depth resolution and accuracy than the D415, making it a preferred choice for tasks demanding precise depth sensing and 3D mapping [11]. Integrating the Intel RealSense D435 camera into a compact hexapod walking robot enhances autonomous exploration with accurate localization and lower computational demands. However, the study lacks qualitative evaluations for exploration deployments [17]. This research utilizes Intel RealSense D435 cameras for autonomous navigation of a 4-wheel vehicle in challenging environments. The integration enables accurate navigation and obstacle avoidance, overcoming GPS limitations. However, a drawback is the fixed position of the D435 cameras [16].  [8] The D435 offers a wider field of view (around 86 degrees) but may exhibit more depth noise.

The RealSense D455 represents the latest advancement in the series, offering more robust features for obstacle detection in UGVs [10]. With a maximum range of 10 meters and enhanced depth accuracy and resolution compared to its predecessors, the D455 is well-suited for complex outdoor environments where accurate obstacle detection at longer distances is crucial for UGV navigation and safety. The D455 exhibits a proficient ability to represent planar surfaces, showcasing strength in capturing and delineating flat objects and scenes with higher precision. 18] The study employs budget-friendly Intel RealSense D455 3D cameras in a mobile mapping system for on-street parking data collection. However, using these cameras outdoors presents challenges due to tricky lighting conditions and limited measurement ranges. [19] The RealSense D455 is noted for producing higher-quality depth images compared to the D400 series, making it suitable for discerning objects and obstacles. The article highlights the applications of Intel® RealSenseTM D400 series cameras in diverse fields, from robotics to surveillance and virtual reality. [1] A new method uses a stereo camera like the Intel RealSense camera D455 to detect road obstacles. The approach combines depth and video analysis, enhancing accuracy through a fusion algorithm that merges depth and visual data.

In terms of compatibility, all three RealSense models support various platforms, including Windows, Linux, and Android, making them versatile options for integration into different UGV setups. They come with a software development kit (SDK) that enables developers to access the camera's depth data and develop custom applications tailored to specific obstacle detection requirements.

Moreover, the RealSense D435 and D455 offer hardware synchronization capabilities, allowing multiple cameras to work together seamlessly for more comprehensive and accurate obstacle detection in UGVs [10]. This feature proves beneficial for applications demanding a higher level of precision and robustness, such as those involving complex terrains or dynamic environments where real-time obstacle detection is critical for safe navigation.

[3] The study investigates the utility of Intel RealSense depth cameras, specifically D415 and D435, for obstacle detection in robotics. D415 outperformed D435 in terms of depth image accuracy. The D435 presented challenges because of its wider field of view, including noise and resolution issues.

The D415 outperforms the D455, particularly in scenarios where transparency or translucency influence depth estimates [4]. However, it shows relatively weaker performance in short-range reconstruction quality compared to the D415. Error reduction is possible with the use of the Intel RealSense Depth Cameras D455 and D415, demonstrating low error rates [2].

[1] The D455 has the drawbacks of limited detection ranges and challenges with background noise. Resolving false positives is a critical component of reliable detection systems.

After a comprehensive analysis of the Intel RealSense D415, D435, and D455, it can be asserted that the RealSense D455 stands out as the most suitable option for obstacle detection in UGVs [10]. Its superior depth-sensing capabilities, extended range, and enhanced accuracy make it an optimal choice for navigating challenging terrains and environments. The D455's wider field of view and advanced synchronization capabilities enable more reliable obstacle detection, ensuring the UGV's safe and precise navigation, even in complex and dynamic surroundings.

V. ACKNOWLEDGEMENTS

I would like to express my heartfelt gratitude to all those who have taken the time to read and review this paper. Your valuable insights and feedback have significantly contributed to its quality and completeness. Thank you for your time and support.

Conclusion

In this comprehensive review, we thoroughly examined the characteristics of Intel RealSense cameras, specifically the D415, D435, and D455 models, aiming to enhance obstacle detection capabilities for Unmanned Ground Vehicles (UGVs). The experiment\'s results unveiled distinctive features of each model, guiding our conclusion. While the D415 demonstrated versatility, excelling in short-range tasks and showcasing adaptability beyond obstacle detection, it encountered limitations in accurately representing planar surfaces. The RealSense D435, with its extended depth range and higher resolution, emerged as a strong candidate for applications demanding precise depth sensing and 3D mapping, particularly in UGVs. However, the pinnacle of our evaluation led us to affirm that the RealSense D455 stands out as the most suitable choice for obstacle detection in UGVs. Boasting superior depth sensing capabilities, an extended range of 10 meters, and enhanced accuracy, the D455 excels in navigating complex outdoor environments where precise obstacle detection at longer distances is paramount for UGV safety. Its wider field of view and advanced synchronization capabilities further contribute to more reliable obstacle detection, ensuring the UGV\'s secure and precise navigation, even in intricate and dynamic surroundings. The D455\'s proficiency in capturing planar surfaces adds an extra layer of precision, solidifying its position as the optimal solution for obstacle detection challenges in UGV applications.

References

[1] Perezyabov, O., Gavrilenkov, M., & Afanasyev, I. (2022, April). Depth and image fusion for road obstacle detection using stereo camera. In 2022 31st Conference of Open Innovations Association (FRUCT) (pp. 237-244). IEEE. [2] Servi, M., Mussi, E., Profili, A., Furferi, R., Volpe, Y., Governi, L., & Buonamici, F. (2021). Metrological characterization and comparison of d415, d455, l515 realsense devices in the close range. Sensors, 21(22), 7770. [3] Tadic, V., Odry, A., Kecskes, I., Burkus, E., Kiraly, Z., & Odry, P. (2019). Application of Intel realsense cameras for depth image generation in robotics. WSEAS Transac. Comput, 18, 2224-2872. [4] Curto, E., & Araujo, H. (2022). An experimental assessment of depth estimation in transparent and translucent scenes for Intel RealSense D415, SR305 and L515. Sensors, 22(19), 7378. [5] Singh, Y., & Kaur, L. (2017). Obstacle detection techniques in outdoor environment: process, study and analysis. International Journal of Image, Graphics and Signal Processing, 9(5), 35. [6] Sivaraman, S., & Trivedi, M. M. (2013, June). A review of recent developments in vision-based vehicle detection. In 2013 IEEE intelligent vehicles symposium (IV) (pp. 310-315). IEEE. [7] Memon, Q., & Khan, S. (2001). Camera calibration and three-dimensional world reconstruction of stereo-vision using neural networks. International Journal of systems science, 32(9), 1155-1159. [8] Grunnet-Jepsen, A., Sweetser, J. N., & Woodfill, J. (2018). Best-known-methods for tuning intel® realsense™ d400 depth cameras for best performance. Intel Corporation: Satan Clara, CA, USA, 1. [9] RealSense, I. (2019). Intel RealSense TMD400 Series Product Family. Technical Report. [10] Andriyanov, N. (2022, May). Estimating object coordinates using convolutional neural networks and intel real sense D415/D455 depth maps. In 2022 VIII International Conference on Information Technology and Nanotechnology (ITNT) (pp. 1-4). IEEE. [11] Carfagni, M., Furferi, R., Governi, L., Santarelli, C., Servi, M., Uccheddu, F., & Volpe, Y. (2019). Metrological and critical characterization of the Intel D415 stereo depth camera. Sensors, 19(3), 489. [12] Lee, B., Daniilidis, K., & Lee, D. D. (2015, May). Online self-supervised monocular visual odometry for ground vehicles. In 2015 IEEE International Conference on Robotics and Automation (ICRA) (pp. 5232-5238). IEEE. [13] Huang, L., Zhe, T., Wu, J., Wu, Q., Pei, C., & Chen, D. (2019). Robust inter-vehicle distance estimation method based on monocular vision. IEEE Access, 7, 46059-46070. [14] Se, S., & Jasiobedzki, P. (2008). Stereo-vision based 3D modeling and localization for unmanned vehicles. International Journal of Intelligent Control and Systems, 13(1), 47-58. [15] Bernini, N., Bertozzi, M., Castangia, L., Patander, M., & Sabbatelli, M. (2014, October). Real-time obstacle detection using stereo vision for autonomous ground vehicles: A survey. In 17th International IEEE Conference on Intelligent Transportation Systems (ITSC) (pp. 873-878). IEEE. [16] Kim, H. (2022). Ground Vehicle Navigation with Depth Camera and Tracking Camera. [17] Bayer, J., & Faigl, J. (2019, September). On autonomous spatial exploration with small hexapod walking robot using tracking camera intel realsense t265. In 2019 European Conference on Mobile Robots (ECMR) (pp. 1-6). IEEE. [18] Nebiker, S., Meyer, J., Blaser, S., Ammann, M., & Rhyner, S. (2021). Outdoor mobile mapping and AI-based 3D object detection with low-cost RGB-D cameras: The use case of on-street parking statistics. Remote Sensing, 13(16), 3099. [19] BEYAZ, A. ACCURACY DETECTION OF INTEL® REALSENSE D455 DEPTH CAMERA FOR AGRICULTURAL APPLICATIONS. BOOK OF, 185.

Copyright

Copyright © 2024 Mamlesh VA, Rahul L. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.