What do I look for in a survey dataset as a (crime survey) data user?

Presentation given to Survey Practice Forum: Challenges and Opportunities for Social Survey Data Collection in Scotland, 5 December 2024

Transcript

1. What do I look for in a survey dataset as a (crime survey) data user? Survey Practice Forum: Challenges and Opportunities for Social Survey Data Collection in Scotland Ben Matthews | University of Stirling
2. Overview • Why use surveys to measure crime? • Constructing a victimization time trend • Sample size and the crime drop • Sampling frames and victimization
3. Why surveys? Photo by Scott Blake on Unsplash
4. Why crime surveys? • How much crime is there? • Actually quite hard to know • We can just rely on data from the criminal justice system: we need triangulation across data sources due to the ‘dark figure’ of crime • Identifying the gap between survey reports of victimization and police recorded crime was the key (and radical) innovation of crime surveys Buil-Gil, D., Brunton-Smith, I., Pina-Sánchez, J., & Cernat, A. (2022). Comparing measurements of violent crime in local communities: a case study in Islington, London. Police Practice and Research, 23(4), 489–506. https://doi.org/10.1080/156 14263.2022.2047047
5. Surveys don’t just happen • Crime data triangulation is not a big problem “in countries where alternative forms of data beyond official criminal justice statistics are obtainable, such as victim surveys” • But is “formidable particularly outside consolidated western democracies where the range of available data is often limited.” Kıvanç Atak, Beyond the western crime drop: Violence, property offences, and the state in Turkey 1990–2016, International Journal of Law, Crime and Justice, Volume 60, 2020, https://doi.org/10.1016/j.ijlcj .2019.100373.
6. Surveys as knowledge infrastructure • Knowing that data will be archived and available years from now (and that data from years ago is archived) is key to exploring change over time • Established social surveys are part of the ‘knowledge infrastructure’ in a way that ‘big data’ or administrative data sources aren’t always
7. Constructing a time trend Photo by Anne Nygård on Unsplash
8. Getting the most from established surveys • Established surveys let you: • Examine change over time • Analyse small population groups after pooling sweeps together to increase sample size • Ideally, both • But there are practical challenges in using these data • And open questions about how best to analyse them
16. ? ‘Complete pooling’ ‘No pooling’ ‘Partial pooling’ ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Treats people from every sweep as if they are the same Treats people from every sweep as if they are completely different
17. Sample size and the crime drop Photo by Daniele Levis Pelusi on Unsplash
19. Sample size and the crime drop 2008/09 2021/22 Change Response rate 70.9% 47.3% -23.6 %pt Core achieved sample 16,003 5,516 -65% Victimization prevalence 20.4% 10.0% -51% Number of victims interviewed* 2,786 532 - 81% Number of repeat victims interviewed* 1,083 164 - 85% Bell et al. (2024, January). Scottish Crime and Justice Survey 2021/22 Technical Report. https://www.gov.scot/binari es/content/documents/govs cot/publications/statistics/2 023/11/scottish-crime- justice-survey-2021-22- main- findings/documents/technic al-report/technical- report/govscot%3Adocume nt/Crime%2BSurvey%2B- %2B2021-22%2B- %2BTechnical%2BReport.p df * Own calculations
20. Sample size and the crime drop • There is an irony here: adults in general don’t have much of an experience of crime – victimization prevalence as estimated by the SCJS was ~10% in the most recent sweeps • So general population victimization surveys are less able to explore victims’ experiences of crime unless they increase sample size over time • This is even more true for repeat victims Scottish Women’s Aid. (2021, December). Response to the consultation on the Scottish Crime and Justice Survey (SCJS), December 2021. https://womensaid.scot/wp- content/uploads/2022/04/S WA-response-to-Scottish- Crime-and-Justice-Survey- consultation.pdf
21. Small numbers means more volatility Sylvia Walby, Jude Towers, Brian Francis, Is Violent Crime Increasing or Decreasing? a New Methodology to Measure Repeat Attacks Making Visible the Significance of Gender and Domestic Relations, The British Journal of Criminology, Volume 56, Issue 6, November 2016, Pages 1203–1234, https://doi.org/10.1093/bjc/azv131
22. The crime drop and survey design • There is a tension between the aims of providing ‘a valid and reliable measure of adults’ experiences of crime’ and ‘examin[ing] trends over time in the number and nature of crimes in Scotland’ • These are not specific to SCJS but maybe more acute than with e.g. Crime Survey for England and Wales due to smaller sample size
23. Sampling frame Photo by Angèle Kamp on Unsplash
24. The ‘average’ person • Nationally representative sample surveys are designed to reflect the experience of the average person or household • “The SCJS is a random probability sample survey of private households” using the Royal Mail Postcode Address File (PAF) • SCJS aims to “provide a valid and reliable measure of adults' experience of crime”… Saunders, A., Caldwell, K., Magnani, L., Cook, B., Page, L., Rose, J., Martin, C., & Hockaday, C. (2021). Scottish Crime and Justice Survey 2019/20: Technical Report. Scottish Government. p.13
25. Sampling frames and victimization • … but if we want a measure of crime for all adults, not just all adults in private households, we need to assume that “the subset of the adult population not captured in the SCJS experience the same level of victimisation as adults in the household resident population”1 • But this excludes homeless people, students, people in prison, nomadic/itinerant populations and those living in refuges 2 1 Scottish Women’s Aid. (2021, December). Response to the consultation on the Scottish Crime and Justice Survey (SCJS), December 2021. https://womensaid.scot/wp- content/uploads/2022/04/S WA-response-to-Scottish- Crime-and-Justice-Survey- consultation.pdf 2 Carr-Hill, R. (2013). Missing Millions and Measuring Development Progress. World Development, 46, 30–44. https://doi.org/10.1016/j.wo rlddev.2012.12.017
26. Sampling frames and victimization • If the outcome you’re interested in (victimization) affects selection into the survey – like in our examples – this is not a ‘problem’ that can be solved’ other than some specific scenarios • Adults in private households and all adults are just different target populations – neither is necessarily right or wrong for a given application Schuessler, J., & Selb, P. (2023). Graphical Causal Models for Survey Inference. Sociological Methods & Research, 00491241231176851. https://doi.org/10.1177/004 91241231176851
27. Thank you Survey Practice Forum: Challenges and Opportunities for Social Survey Data Collection in Scotland ben.matthews@stir.ac.uk