Upgrade to Pro — share decks privately, control downloads, hide ads and more …

DyViR: Dynamic Virtual Reality Dataset for Aeri...

DyViR: Dynamic Virtual Reality Dataset for Aerial Threat Object Detection

Presentation at the SPIE Defense + Commercial Sensing Conference

Gregory Ditzler

June 07, 2023
Tweet

More Decks by Gregory Ditzler

Other Decks in Research

Transcript

  1. Presented By: Gregory Ditzler(1,2) (1)Department of Electrical & Computer Engineering

    (2)Machine Artificial Intelligence and Virtual Reality Center (MAVRC) Rowan University Glassboro, NJ DyViR: Dynamic Virtual Reality Dataset for Aerial Threat Object Detection Garrett Williams, George D. Lecakes Jr., Amanda Almon, Nikolas Koutsoubis, Kyle Naddeo, Thomas Kiel, Gregory Ditzler, and Nidhal C. Bouaynaya
  2. Outline • UAV Threat/Object Detection using ML • Motivating Data

    Augmentation using Synthetic Data • Real-Time Rendering Technology • What is DyViR and what does it produce? ◦ YOLOv7-tiny Training and Validation ◦ Results from YOLOv7-tiny ML models • Conclusions and Future Directions
  3. The Bigger Picture • Design & develop Virtual and Augmented

    Reality environments of armored and tactical vehicles, and threat scenarios. • Develop AI/ML-based systems to enhance turret operations and to interrogate potential threats. • Build situational awareness systems to provide secure, untethered, high-speed wireless communication between the vehicle and the soldier. Support by a grant from the Army Research Office W15QKN-21-C-0077.
  4. The Bigger Picture • Design & develop Virtual and Augmented

    Reality environments of armored and tactical vehicles, and threat scenarios. • Develop AI/ML-based systems to enhance turret operations and to interrogate potential threats. • Build situational awareness systems to provide secure, untethered, high-speed wireless communication between the vehicle and the soldier Support by a grant from the Army Research Office W15QKN-21-C-0077. Sensors Human Factors VR/AR/MR AI/ML Sys. Int.
  5. UAV Object Detection using ML • Unmanned Aerial Vehicles (UAVs)

    pose a major threat on the modern combat landscape ◦ These objects tend to be small (e.g., contained on a small number of pixels), and ◦ There are limited datasets available to learn an object detector. ▪ Number of samples ▪ Types of UAVs ▪ Biomes where data are collected ▪ Etc. • Machine Learning (ML) is a potential solution in identifying and classifying friendly and hostile UAVs in large swarms; however, limited data will limit the performance of any ML model.
  6. Motivating Synthetic Data Augmentation • The lack of datasets with

    UAVs in combat landscapes due to safety concerns and the emergence of new UAV technologies • Synthetic training datasets are able to augment real datasets, allowing for training on large amounts of balanced data
  7. Real-Time Rendering Technology • Advancements in computer graphics allow for

    the realistic renderings of 3D digital environments produced in real-time (30 frames per second +) • Imagery of custom 3D assets in a simulated 3D environment is created by software such as the Unity Real-Time Development Platform (Unity) ◦ The developer of the datasets are no longer limited by the biome, types of drones, etc. ◦ This allows for users to develop more diverse datasets • Real-time rendering allows for the rapid production of synthetic datasets
  8. What is DyViR and what does it produce? • DyViR

    creates synthetic training datasets for UAV object detection built on the Unity Engine • The 2D bounds of the synthetic UAVs is calculated within the engine • Imagery and bounding box data is linked to produce training datasets in a compatible format for ML models Outputs images from Flaticon.com Compatible with YOLO formats
  9. YOLOv7-tiny Training and Validation • The You Only Look Once

    (YOLO) model is designed for object detection • The YOLOv7-tiny model is a lightweight variant suitable for deployment on edge devices ◦ YOLOv7-tiny classifies objects and places a bounding box identifying the extents of the object • The Mean Average Precision (mAP) score is a metric for determine the performance of object detection
  10. • A real-world and DyViR model were trained, the former

    exclusively on real-world data and the latter trained first on synthetic DyViR data and fine-tuned on real-world Results from YOLOv7-tiny ML models Real-World Model images from Flaticon.com DyViR Model Train Fine-Tune Train
  11. • Over 100 Epochs, the DyViR model saw a 60.4%

    increase in the mAP 0.5:0.95 score over its real-world counterpart (0.653 to 0.407) Results from YOLOv7-tiny ML models
  12. • Access to real-world data to build a robust and

    trustworthy machine learning model can be challenging when real-world data are scarce ◦ Real-world data can be challenging or near impossible to collect ◦ Collecting data that covers all p(x) can be challenging regarding ▪ Different biomes, UAVs, weather, etc. • DyViR allows for rapid development of new datasets for aerial object detection. • Future Work: DyViR intends to produce datasets that have more realistic flight patterns, allowing for object detection-based on temporal changes ◦ Develop software to superimpose VR drones over real-scenes ◦ Investigate curriculum learning methods to boost performance Conclusion
  13. Thank you! Questions? This work was supported by grants from

    the Army Research Office W15QKN-21-C-0077, and National Science Foundation CAREER #1943552. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the sponsors' views. Data https://github.com/RowanMAVRC/DyViR-For-Unity-SPIE-2023