Investigating Content Multidimensionality in a Large-scale Science Assessment: A Mixed Methods Approach
Loading...
Date
2024-08-07
Authors
Malcom, Cassandra
Journal Title
Journal ISSN
Volume Title
Publisher
University of Oregon
Abstract
Science, Technology, Engineering, and Math (STEM) skills are increasingly required of students to be successful in higher education and the workforce. Therefore, modeling assessment outcomes accurately, often using more types of student data to get a complete picture of student learning, is increasingly relevant. The Program for International Student Assessment (PISA) is promoted as a summative assessment opportunity that includes a science framework. As with many science assessments, the framework includes Life, Physical, and Earth science, which alone seems to imply multidimensionality, and also there are other sources of dimensionality that seem to be described conceptually in the framework. Using data from the 2015 PISA science assessment, a multidimensional item response theory (MIRT) model was fit to see how a multidimensional model operates with the data. Before developing the MIRT model, a qualitative review of the framework for multidimensionality took place and exploratory analyses were implemented for the quantitative data, including a data science technique to explore multidimensionality and some factor analysis techniques. After fitting the MIRT model, it was compared to several unidimensional IRT (UIRT) models to determine the model that explains the most variation. The qualitative analyses generated evidence of multidimensional science content domains in the 2015 PISA science framework, which should require a MIRT model, but quantitative analyses indicate a unidimensional model is more practically significant. Once quantitative results were triangulated with the qualitative review of the framework for multidimensionality, the implications on equity and history of harm with regards to science assessments were discussed. Findings from the qualitative and quantitative aspects of the study were used to generate recommendations for different stakeholders.
Description
Keywords
Item Response Theory, Large-scale Assessment, Multidimensionality, Qualitative Framework Review, STEM Education, Summative Assessment