Introduction to Camtrap DP: A frictionless data exchange format for camera trapping data

Talk at the Exploring camera-trap data GBIF webinar - November 9, 2022.

Transcript

1. Introduction to Camtrap DP A frictionless data exchange format for camera trapping data Exploring camera-trap data GBIF webinar, 9 November 2022 Peter Desmet 0000-0002-8442-8025
2. - Wildlife monitoring technique - Non-invasive - Well-established - Enables study of animal abundance, distribution, behaviour - Data-intensive: lots of images or videos Camera trapping
3. - Data management platforms - Upload and manage data - Annotate with species identifications (often using image recognition) - Limited data exchange between platforms - Limited data publication from platforms Data are well-managed, not shared Agouti Wildlife Insights TRAP PER eMam mal
4. - Does not capture full scope - Project setup - Camera setup - Blank, vehicle, unknown sequences of images - Star schema too limited to capture all relationships - Camera trap researchers do not recognize data model Darwin Core (Archive)? Agouti Wildlife Insights TRAP PER eMam mal Darwin Core (Archive)?
5. - “Camera Trap Data Package” - Designed to capture all essential data and metadata of a single camera trap study - Model to exchange camera trapping data - Format to exchange camera trapping data Camtrap DP
6. - Metadata about project Camtrap DP model Project / Study media 1 heron media 2 heron media 3 blank event 1 moorhen event 1 coot gbif.org/occurrence/3045046810 gbif.org/occurrence/3045043163 - Deployments: start/end date, location, camera info - Media: file path/url, timestamp, event (sequence) - Observations: blank, or animal of certain species, count, sex, ...
7. - Metadata as datapackage.json - Project metadata - Package structure - Deployments as csv - Media as csv - Observations as csv - Either at event or media level* Camtrap DP format datapackage.json eventID eventObservations .csv mediaObservations .csv deployments.csv media.csv mediaID deploymentID *Upcoming version
8. - Developed by Frictionless Data - Set of open specifications (JSON schemas) that can be combined - Data Package for datasets - Data Resource for data files - Table Schema for table fields - Simple, machine-usable & extensible Using Frictionless Standards specs.frictionlessdata.io
9. - Open, versioned, collaborative - Camtrap DP profile (extends Data Package) - Project metadata - Spatial, temporal, taxonomic scope - 3 Table Schemas - Fields, definitions, data types, controlled vocabularies (enum) - Borrowed terms (skos:...) - Relationships between tables Camtrap DP development github.com/tdwg/camtrap-dp
10. Camtrap DP development - Suggestions via GitHub issues and pull requests - Automated tests, against: - Frictionless specification - Example dataset - Automated human-readable documentation tdwg.github.io/camtrap-dp
11. - In your datapackage.json link to a version of: - Camtrap DP Table Schemas (data) - Camtrap DP profile (metadata) How should your dataset reference Camtrap DP? { "resources": [ { "name": "deployments", "path": "deployments.csv", "profile": "tabular-data-resource", "schema": "https://raw.githubusercontent.com/tdwg/camtrap-dp/0.4/ deployments-table-schema.json" }, { "name": "media", "path": "media.csv", "profile": "tabular-data-resource", "schema": "https://raw.githubusercontent.com/tdwg/camtrap-dp/0.4/ media-table-schema.json" }, { "name": "observations", "path": "observations.csv", "profile": "tabular-data-resource", "schema": "https://raw.githubusercontent.com/tdwg/camtrap-dp/0.4/ observations-table-schema.json" } ], "profile": "https://raw.githubusercontent.com/tdwg/camtrap-dp/0.4/ camtrap-dp-profile.json", "id": "1df9727c-dde9-498a-824e-cc4f19436117", "contributors": {}, "project": {}, "spatial": {}, "temporal": {}, "taxonomic": [], } github.com/tdwg/camtrap-dp/ tree/main/example
12. - Agouti - TRAPPER - Integrated Publishing Toolkit (IPT) Software to create
13. - Frictionless Framework (Python) can validate: - Metadata - Structure - Fields - Controlled vocabularies - Relationships frictionless validate datapackage.json # ----- # valid: deployments.csv # ----- # ----- # valid: media.csv # ----- # ----- # valid: observations.csv # ----- Software to validate v4.framework.frictionlessdata.io
14. # devtools::install_github("inbo/camtraptor") library(camtraptor) dataset <- read_package( "https://raw.githubusercontent.com/tdwg/camtrap-dp/main/example/datapa ckage.json" ) #> Please make sure you have the right to access data from this Data Package for your intended use. #> Follow applicable norms or requirements to credit the dataset and its authors. get_species(dataset) # A tibble: 10 × 6 taxonID taxonIDReference scien…¹ verna…² verna…³ taxon…⁴ <chr> <chr> <chr> <chr> <chr> <chr> 1 DGP6 https://www.catalogueo… Anas p… mallard wilde … NA 2 DGPL https://www.catalogueo… Anas s… gadwall krakee… species 3 32FH https://www.catalogueo… Ardea great … reigers genus 4 GCHS https://www.catalogueo… Ardea … grey h… blauwe… species 5 RQPW https://www.catalogueo… Castor… Eurasi… bever species 6 6MB3T https://www.catalogueo… Homo s… human mens species 7 3Y9VW https://www.catalogueo… Martes… beech … steenm… species 8 44QYC https://www.catalogueo… Mustel… Europe… bunzing species 9 4RM67 https://www.catalogueo… Rattus… brown … bruine… species 10 5BSG3 https://www.catalogueo… Vulpes… red fox vos species # … with abbreviated variable names ¹ scientificName, # ² vernacularNames.en, ³ vernacularNames.nl, ⁴ taxonRank - Frictionless Framework (Python) - frictionless (R package) - camtraptor (R package) - read_camtrap_dp() - read_wi(): import Wildlife Insights export as Camtrap DP - write_dwc(): export as DwC - write_eml(): export as EML - Filter, explore, visualize, … Software to read and analyse inbo.github.io/camtraptor
15. - Release Camtrap DP v1.0 ✨ - Submit paper describing format - Update software - Agouti - Trapper - Camtraptor - … - Biodiversa+ funding to continue this work Next steps
16. - Capture essential data and metadata of a camera trap study - Data exchange model and format - Uses Frictionless Standards - Built-in validation - Open, versioned and collaborative - Version 1.0 coming soon Summary tdwg.github.io/camtrap-dp
17. Thank you tdwg.github.io/camtrap-dp Desmet P (2021) Introduction to Camtrap DP: A frictionless data exchange format for camera trapping data. Presentation at the Exploring camera-trap data GBIF webinar. https://bit.ly/camtrap-dp-gbif-2022 Camtrap DP contributors Peter Desmet Jakub Bubnicki Ben Norton Edward Baker Steve Baskauf Jim Casaer Christian Fiderer Tim R. Hofmeester Patrick A. Janssen Yorick Liefting Jürgen Niedballa Nicolas Noé Raphaël Nussbaumer Tim Robertson David Shorthouse Dans Towell John Wieczorek