A combinatorial RGB and depth images CNN-based model for oil palm fruit bunch detection and heatmap localisation for a visual SLAM system

Abstract

The harvesting job of cutting and collecting fruit bunches in oil palm plantations remains the most labour-intensive job in the oil palm processing cycle. The introduction of an autonomous vehicle to assist workers in the harvesting job promises better productivity. Such a driverless vehicle requires a software module known as simultaneous localisation and mapping (SLAM) to guide the vehicle to navigate autonomously. This work proposes a visual SLAM system with a distinctive capability of detecting and localising oil palm loose fresh fruit bunches (FFB) on the ground using intelligent image processing. This vehicle is equipped with a depth camera capable of capturing RGB images and depth images concurrently. Two VGG16-based convolutional neural network (CNN) models are trained using the acquired RGB and depth images dataset of loose FFBs on the ground. The output from the combinatorial FFB detection model is then fed into a visual SLAM system called RTAB-Map. By combining the FFB detection model and the visual SLAM system, the vehicle can plan for autonomous navigation safely, perform bunch pick-up tasks, and avoid collision with fruit bunches on the ground. The experiment results show that the proposed CNN model can detect and localise loose FFBs with significant accuracy in various lighting conditions.