Diagnostic modeling for the Dynamic Learning Maps assessment

Transcript

1. Diagnostic modeling for the Dynamic Learning Maps assessment Presentation to St.Gallen University of Teacher Education 4 May 2026 1
2. DIAGNOSTIC CLASSIFICATION MODELS 2
3. Diagnostic classification models • Psychometric models that support reporting mastery on a set of defined attributes • Attributes: Identified skills of interest • Attributes are categorical: master/nonmaster, proficient/not proficient, presence/absence • Assessment items are mapped to attributes through a Q-matrix 3
4. Profile Att. 1 Att. 2 Att. 3 1 0 0 0 2 1 0 0 3 0 1 0 4 0 0 1 5 1 1 0 6 1 0 1 7 0 1 1 8 1 1 1 Attribute classes • DCMs are confirmatory latent class models • Classes are attribute profiles • With binary attributes there are 2A possible profiles • The number of classes increases exponentially with the number of attributes 4
5. Respondent results • Item responses drive the likelihood of each profile • Balanced by structural parameters that define the prevalence of each profile • Respondent results are the probabilities of each profile • Usually more useful to report mastery for each attribute 5 Profile Att. 1 Att. 2 Att. 3 Prob. 1 0 0 0 .01 2 1 0 0 .05 3 0 1 0 .01 4 0 0 1 .06 5 1 1 0 .04 6 1 0 1 .42 7 0 1 1 .08 8 1 1 1 .33 Profile Att. 1 Att. 2 Att. 3 Prob. 1 0 0 0 .01 2 1 0 0 .05 3 0 1 0 .01 4 0 0 1 .06 5 1 1 0 .04 6 1 0 1 .42 7 0 1 1 .08 8 1 1 1 .33 .84
6. THE DYNAMIC LEARNING MAPS ALTERNATE ASSESSMENT SYSTEM 6
7. The U.S. educational context 7 • Federal law requires all students to be assessed annually, with no opt-outs for disability status • Alternate assessments • Taken by ~1% of students with the most significant cognitive disabilities for whom grade-level content is inaccessible, even with accommodations • Alternate content standards that represent the essence of grade-level expectations at reduced depth, bread, and complexity • Results count in school accountability calculations
8. Dynamic Learning Maps • Alternate assessment administered to over 100,000 students annually in 25 states • Assessments are administered throughout the year • Results are available on-demand to support instructional decision- making • Academic content is represented as a large network of fine-grained learning maps • Nodes range from foundational skills to grade-level targets, providing all students access to academic content 8
10. ELA.EE.RI.7.2 10
11. DLM PSYCHOMETRICS 11
12. Estimating mastery within a learning map • Not feasible to estimate the entire learning map • Psychometrically, far too many nodes and parameters given the number of student who complete the test • Practically, the amount of testing that would be required by students to gather the data needed to support model would not be accepted • Two strategies: 1. Zoom out on the map to identify critical junctures that become the assessment targets 2. Use the underlying map structure to reduce the number of possible profiles 12
13. Linkage levels are the critical junctures 13
14. Linkage levels as the unit of reporting 14
15. Attribute hierarchies for DLM • Within each standard there are 5 attributes, and 32 possible attribute profiles • Given the underlying map structure, we know that earlier skills are prerequisite to the acquisition of other skills • Eliminate the impossible profiles 15 Profile Att. 1 Att. 2 Att. 3 1 0 0 0 2 1 0 0 3 0 1 0 4 0 0 1 5 1 1 0 6 1 0 1 7 0 1 1 8 1 1 1
16. Attribute hierarchies in a DCM context • Multiple methods for enforcing hierarchies • Implemented by putting constraints on the DCM structural model (i.e., the expected prevalence of each profile) • Hard constraints: Only the theoretically possible profiles are allowed (HDCM; Templin & Bradshaw, 2014) • Soft constraints: All profiles remain possible, but respondents are pushed toward the theorized profiles (BayesNet; Hu & Templin, 2020) 16
17. DLM attribute hierarchies • Linkage levels follow a linear hierarchy within each content standard • Reduces the number of profiles from 25 = 32 to only 6 • Greatly reduces the number of item parameters needed as well • Testable! Fit the full model and hierarchical model to compare fit 17 Thompson and Nash (2022)
18. Limitations of the current approach • Each standard estimated independently • Mastery of linkage levels on one standard does not directly inform mastery on other standards • Indirect evidence for the structure of the learning map • Linkage levels are not 1:1 with nodes in the learning map • Empirical support for the ordering of linkage levels only indicates general agreement with the ordering of nodes within the map 18
19. wjakethompson.com wjakethompson@ku.edu 0000-0001-7339-0300 in/wjakethompson @wjakethompson @wjakethompson.com @wjakethompson@fosstodon.com @wjakethompson Thank you! Slides