Identifying Text-Task-Reader Interactions Related to Item and Block Difficulty in the National Assessment for Educational Progress Reading Assessment

This study examines the nature of text-task-reader interactions in relation to student performance on the National Assessment of Educational Progress (NAEP) reading assessment in order to suggest strategies that NAEP might use to select passages, develop items, and interpret data to more closely approximate the complex nature of reading comprehension. More specifically, the study uses data from the 2009 NAEP reading administration to address two questions: (1) which qualitative and quantitative characteristics designed to examine text-task-reader interactions are related to comprehension difficulty at Grades 4, 8, and 12? and (2) what differences among these characteristics, if any, exist across blocks, grade levels, genres, item types, and/or cognitive targets?

The authors developed two types of text-task-reader characteristics using existing research findings to identify characteristics of texts and tasks that had been shown to be associated with comprehension performance. The first type, Text-Task-Reader Comprehension (TTR-C) characteristics, focuses on the interaction among the text, item stem, and correct response. The second type, Text-Task-Reader Distractor characteristics (TTR-D), focuses on the characteristics of the distractors used in multiple-choice items in relation to the text, item stem, and correct response.
 

Key Findings

Results suggested that NAEP reporting could profit from using TTR-C and TTR-D characteristics to augment the current item descriptions and achievement level descriptors to provide a more nuanced description of what students are able to do at different achievement levels. For example, the 2015 multiple choice item (cognitive target “integrate/interpret) used to illustrate fourth-grade Basic, could be described as "recognize main purpose of an informational text when it is explicitly stated in the text and the key is unambiguously the best choice," rather than simply "Recognize main purpose of an informational text."  

In addition, both the TTR-C and TTR-D characteristics and associated rubrics may prove useful in block and item development for NAEP by providing a more detailed framework for understanding and assessing comprehension. For example, developers might

• use the TTR-C characteristics as building blocks for items and to ensure that the level of complexity in selected texts is appropriate.
• use the TTR-D characteristics to construct appropriate distractors for various achievement levels.
• use the TTR-C and TTR-D characteristics to inform cognitive target classification of items.