Assessing Academic and Disciplinary Literacies: Rubric validation to measure argumentation, comparison and source-based writing skills

Abstract

Introduction: Over the last two decades, research has suggested that academic and disciplinary literacies (ADLs) are key to integrating content and language, as language is used to express content knowledge, often through Cognitive Discourse Functions, discourse patterns that respond to cognitive actions in formal education contexts. Nevertheless, systematic assessment tools are required to assess student production of ADLs.Purpose: This validation study focuses on developing an analytic rubric to measure three ADL skills, consisting of three dimensions. Two of these dimensions are CDFs: students’ skills to argue and compare. A third rubric measures an additional academic skill, students’ ability to write from sources. The rubrics are designed to capture cross-disciplinary and multilingual productions of these skills, therefore being applicable for various disciplines (history, science, mathematics, among others...) and languages. Method: A five-step validation process based on expert judgement was used, involving 13 international experts. They quantitatively and qualitatively evaluated the proposed rubrics based on pertinence, conceptual clarity, coherence, and relevance across two iterative rounds. Quantitative descriptive statistics and agreement indices were used, in addition to thematic analysis of qualitative feedback.Results: The quality of the rubric showed clear progression between validation rounds. In the first version, several issues were raised by the experts–most criteria showed weak agreement and low means. After revisions, there was a substantial improvement in the second version, with 83 % of the criteria reaching strong or acceptable agreement. Qualitative feedback highlighted the need for precise and multidimensional operationalisations in each ADL dimension. To illustrate the application of the rubrics, multilingual student samples are provided.Conclusion: The rubrics offer the first steps towards systematising ADL assessment by combining qualitative and quantitative feedback from 13 experts, underscoring the importance of expert input in advancing assessment practices. The study has theoretical and practical contributions: it highlights the multidimensional nature of ADLs and provides an adaptable rubric that can be used across disciplines, languages and educational contexts. This study focused on the validation process; therefore, empirical use of the rubric is still required. Future research should apply the rubrics to large-scale corpora and complement the expert-based validation with psychometric approaches