Research Focus: Automated dataset creation and improvement of the object recognition model in a customised environment

Deep learning models have become increasingly popular in recent years, particularly in robotics and autonomous systems [1]. These models have achieved good performance on tasks such as object detection (OD) and segmentation using image data. However, OD models usually require a large amount of labelled (or annotated) image data for training. The manual labelling process for data from custom environments is time-consuming and requires significant human resources [2]. Although methods such as semi-supervised learning (SSL) and active learning exist to predict labels from a smaller amount of labelled training data, they still require human assistance to examine the generated dataset manually due to the model’s false positive detections.

Objectives:

1. To improve the performance of the object detection model using custom environmental data
2. Remove false positive detections using a clustering-based methodology to automate the process

Methodology
To improve the performance of object detection (OD) models, it is crucial to remove false positives (FPs) from the training dataset; however, this becomes challenging to do manually when dealing with large datasets. To address this, a clustering-based method is proposed for the automatic removal of FPs. The environmental data is first collected using a camera and a 2D LiDAR sensor and stored in a Multi-Channel Audio Pack (MCAP) file. Object detection is performed on the image data using a YOLO model. By combining image-based bounding boxes with corresponding LiDAR points, the 3D positions of detected objects are estimated. To eliminate false positives, the DBSCAN clustering algorithm is applied to these positions, on the assumption that the largest cluster corresponds to the actual object, whilst outlier points are discarded. This filtered and refined dataset is then used to train the OD model, resulting in more accurate and robust detection performance.

Experimental Validation
To illustrate the approach, a service robot called ‘Double 3’ from Double Robotics is considered to be driving in the given environment, and the object of interest for detection is another Double 3 robot. The YOLOv5n object detection model was chosen for training due to its efficiency on small devices. Initially, the robot collects data—including position, orientation and video—during each driving session (referred to as a ‘Job’), which is saved with timestamps. The data recorded after the first driving session is manually annotated and used for training the OD model; once the model is trained, it can be used to extract the datasets from the next driving job. The approach described determines the object’s true location in the environment by checking for its presence in consecutive frames. In other words, to confirm the object’s presence, one can count the number of detections at the estimated location. To explain more clearly, the YOLOv5n network is used to detect the object (Double 3) and attempt to estimate the object’s position using the raw yaw and pitch angles of the observed robot relative to the detected object, and by multiplying the distance by the observed object’s position. These estimated points can be clustered using the DBSCAN density-based clustering algorithm. Finally, the points with the highest clustering scores are considered to be the true positions of the observed object.

The approach presented has been successfully implemented and adapted to the given environment by autonomously removing false positives from the extracted datasets after a few jobs or iterations. After each iteration, the trained model is deployed on the embedded device to validate its detection performance. However, there are still some challenges remaining in the proposed approach. Currently, the approach has only been validated on stationary objects in the environment; when tested on moving objects, the methodology fails to capture some correct predictions due to the clustering density. To overcome this challenge, an object tracking methodology has been proposed, and research in this area is still ongoing. Furthermore, this research could be useful for federated learning to train object detection (OD) models using data from different environments and to assess their overall performance.

About me
: Shivakrishna Karnati began his Master’s degree in Applied Mathematics in 2021, during which he developed a strong interest in interpretable models, deep learning and computer vision. Under the supervision of Prof. Dr.-Ing. Falk Langer, he recently completed his Master’s thesis at IAV GmbH. He is currently broadening his research focus to include Large Language Models (LLMs) and their practical applications in industry. Outside of his academic work, he enjoys playing basketball and is an active member of the Mittweida Friday Club.