Defining dataset specifications to communicate data quality characteristics

Talk at the TDWG 2016 annual conference in Santa Clara de San Carlos, Costa Rica - December 6, 2016.

Recording: https://vimeo.com/showcase/4308386/video/196431347

The Darwin Core standard provides a list of community-ratified terms for sharing biodiversity information. Although some terms have strict definitions, most allow users a certain level of freedom in how to interpret these. This degree of freedom has enabled a wide range of biodiversity data to be mapped to Darwin Core, but it complicates automated data aggregation and processing. One way to resolve this are community specific guidelines describing how data should be mapped, but few have been created or adopted. Moreover, these are intended for humans only.

Inspired by existing data validation specifications in other fields, we propose the usage of a specification file, describing the constraints to which the data should comply. Its syntax is human- and machine-readable, so it can be used to communicate expected data quality/conformity and to validate data automatically. The scope of the set of rules can be specific to a dataset, publisher or community, which allows bottom-up and top-down adoption.

In this talk, we will present a prototype format for these specifications, where the rules are defined on the level of individual terms and expressed as a YAML file. We also present prototype software to validate data with these specifications. We hope it will trigger a discussion on how to express data specifications and mapping guidelines.

Transcript

1. Defining Dataset specifica-ons to communicate data quality Peter Desmet, S-jn Van Hoey, Dimitri Brosens
2. Darwin Core offers a lot of (necessary) freedom
3. But how do you express more rigorous requirements?
4. We need documenta-on
7. Does my dataset comply?
8. We need machine-readable documenta-on
9. YAML Human & machine-readable
11. Demo
12. Dataset
13. Run data-validator
14. Report
15. Improved dataset
16. Rerun data-validator
17. Specifica-ons for datasets
18. Specifica-ons for data publishers
19. Specifica-ons for data users
20. Specifica-ons for communi-es
21. Integra-on in data publica-on workflows
23. Proof of concept github.com/inbo/data-validator Examples used in this presenta-on: bit.ly/2h352c8
24. Thanks! @peterdesmet @s-jnvanhoey @dimibro bit.ly/2h0cDLU Desmet P, Van Hoey S & Brosens D (2016) Defining dataset specifica-ons to communicate data quality. hbp://bit.ly/2h0cDLU