DyViR: Dynamic Virtual Reality Dataset for Aerial Threat Object Detection

Presentation at the SPIE Defense + Commercial Sensing Conference

1. Presented By: Gregory Ditzler(1,2) (1)Department of Electrical & Computer Engineering (2)Machine Artificial Intelligence and Virtual Reality Center (MAVRC) Rowan University Glassboro, NJ DyViR: Dynamic Virtual Reality Dataset for Aerial Threat Object Detection Garrett Williams, George D. Lecakes Jr., Amanda Almon, Nikolas Koutsoubis, Kyle Naddeo, Thomas Kiel, Gregory Ditzler, and Nidhal C. Bouaynaya
2. Outline • UAV Threat/Object Detection using ML • Motivating Data Augmentation using Synthetic Data • Real-Time Rendering Technology • What is DyViR and what does it produce? ◦ YOLOv7-tiny Training and Validation ◦ Results from YOLOv7-tiny ML models • Conclusions and Future Directions
3. The Bigger Picture • Design & develop Virtual and Augmented Reality environments of armored and tactical vehicles, and threat scenarios. • Develop AI/ML-based systems to enhance turret operations and to interrogate potential threats. • Build situational awareness systems to provide secure, untethered, high-speed wireless communication between the vehicle and the soldier. Support by a grant from the Army Research Office W15QKN-21-C-0077.
4. The Bigger Picture • Design & develop Virtual and Augmented Reality environments of armored and tactical vehicles, and threat scenarios. • Develop AI/ML-based systems to enhance turret operations and to interrogate potential threats. • Build situational awareness systems to provide secure, untethered, high-speed wireless communication between the vehicle and the soldier Support by a grant from the Army Research Office W15QKN-21-C-0077. Sensors Human Factors VR/AR/MR AI/ML Sys. Int.
5. UAV Object Detection using ML • Unmanned Aerial Vehicles (UAVs) pose a major threat on the modern combat landscape ◦ These objects tend to be small (e.g., contained on a small number of pixels), and ◦ There are limited datasets available to learn an object detector. ▪ Number of samples ▪ Types of UAVs ▪ Biomes where data are collected ▪ Etc. • Machine Learning (ML) is a potential solution in identifying and classifying friendly and hostile UAVs in large swarms; however, limited data will limit the performance of any ML model.
6. Motivating Synthetic Data Augmentation • The lack of datasets with UAVs in combat landscapes due to safety concerns and the emergence of new UAV technologies • Synthetic training datasets are able to augment real datasets, allowing for training on large amounts of balanced data
7. Real-Time Rendering Technology • Advancements in computer graphics allow for the realistic renderings of 3D digital environments produced in real-time (30 frames per second +) • Imagery of custom 3D assets in a simulated 3D environment is created by software such as the Unity Real-Time Development Platform (Unity) ◦ The developer of the datasets are no longer limited by the biome, types of drones, etc. ◦ This allows for users to develop more diverse datasets • Real-time rendering allows for the rapid production of synthetic datasets
8. What is DyViR and what does it produce? • DyViR creates synthetic training datasets for UAV object detection built on the Unity Engine • The 2D bounds of the synthetic UAVs is calculated within the engine • Imagery and bounding box data is linked to produce training datasets in a compatible format for ML models Outputs images from Flaticon.com Compatible with YOLO formats
9. DyViR Produced Datasets
10. YOLOv7-tiny Training and Validation • The You Only Look Once (YOLO) model is designed for object detection • The YOLOv7-tiny model is a lightweight variant suitable for deployment on edge devices ◦ YOLOv7-tiny classifies objects and places a bounding box identifying the extents of the object • The Mean Average Precision (mAP) score is a metric for determine the performance of object detection
11. • A real-world and DyViR model were trained, the former exclusively on real-world data and the latter trained first on synthetic DyViR data and fine-tuned on real-world Results from YOLOv7-tiny ML models Real-World Model images from Flaticon.com DyViR Model Train Fine-Tune Train
12. Results from YOLOv7-tiny ML models
13. • Over 100 Epochs, the DyViR model saw a 60.4% increase in the mAP 0.5:0.95 score over its real-world counterpart (0.653 to 0.407) Results from YOLOv7-tiny ML models
14. • Access to real-world data to build a robust and trustworthy machine learning model can be challenging when real-world data are scarce ◦ Real-world data can be challenging or near impossible to collect ◦ Collecting data that covers all p(x) can be challenging regarding ▪ Different biomes, UAVs, weather, etc. • DyViR allows for rapid development of new datasets for aerial object detection. • Future Work: DyViR intends to produce datasets that have more realistic flight patterns, allowing for object detection-based on temporal changes ◦ Develop software to superimpose VR drones over real-scenes ◦ Investigate curriculum learning methods to boost performance Conclusion
15. Thank you! Questions? This work was supported by grants from the Army Research Office W15QKN-21-C-0077, and National Science Foundation CAREER #1943552. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the sponsors' views. Data https://github.com/RowanMAVRC/DyViR-For-Unity-SPIE-2023