When students take a statewide or a national test, they will often get results of how their individual scores align with others and how it fits within a scoring scale, such as on the composite or overall test scores for the ACT. But who determines what is considered a “passing” score? This process is called standard setting, or the methodology by which performance standards on tests of proficiency are measured.
Despite its name, the process is far from standard.
MSU’s Mark Reckase has authored a book that could help define a new way of thinking—and change how analyzing (and setting) of test scores is done.
“The Psychometrics of Standard Setting: Connecting Policy and Test Scores” was published by CRC Press in January 2022 and is the culmination of years of research and practice for Reckase, a Michigan State University Distinguished Professor Emeritus.
“The book helps test creators and evaluators create and sustain standards for tests, and also helps with evaluating how the standards work upon dissemination,” said Reckase.
Throughout its pages, Reckase explores a theoretical framework for setting, and adhering, to performance standards.
Though much has been published about setting of performance standards—including several journal articles from Reckase himself— “there has been little effort to specify a formal theory to support the standard setting process,” says the opening chapter of the book. In other words, the book helps to define how to set standards systemically and sustainably, removing any concern of arbitrary or ad hoc efforts.
“In a sense, the practice of standard setting is much like the practice of constructing buildings before the science of strength of materials was developed,” continues Reckase in chapter one. “Master carpenters learned through an apprenticeship how to build sound structures … [A]s structures became larger and more complex, experience and rules of [building] them were no longer sufficient to ensure that structures were safe and would remain standing … Similarly, the detailed procedures for implementing standard setting processes are typically learned through activities like an apprenticeship … based on the experience of the persons asked to perform the task. Little theory is called upon to guide the work.”
“Psychometrics” provides an applicable order to theory, examining policy calling for the existence of performance standards, how test makers can convert policy into test reporting scales. Moreover, the book explores how a standard approach to testing methodology for individual tests can also be used to evaluate and improve test creation and analytics moving forward.
ASSESSMENT INNOVATOR
Reckase is a leading scholar in standard setting and testing. He joined the College of Education in 1998 after 17 years with ACT, including as assistant vice president for the assessment innovations area. As part of his work, he contributed to a team who developed performance standards on the reporting score scales for the National Assessment of Educational Progress (NAEP)—a project, begun in 1992, that Reckase credits as being an initial impetus for the creation of the book. Reckase’s credits also include having served as editor of Applied Psychological Measurement and the Journal of Educational Measurement and as president of both the National Council on Measurement in Education and the International Association for Computerized Adaptive Testing.
And though Reckase’s career has primarily examined testing within educational spaces, the practical usefulness of the book extends to a wide array of tests and applications, such as workplace entrance or preparedness exams, psychological assessments, speed limits, nutritional facts and more.
The latter examples may be surprising, but that is the point, says Reckase.
Standards are incredibly prevalent in our lives, including to the point where most may not notice they exist. Consider, for example, clothing sizes. When ordering or buying a pair of pants, buyers may have the option of sizes in “short,” “regular,” and “tall.” An individual or a group decided what constitutes a short or a regular or a tall pant for that company, but it is not the same across brands or sometimes across offerings within the same store, which can be challenging for the buyer.
Reckase posits that his book could help decision-makers follow a framework to make things more streamlined.
“The Psychometrics of Standard Setting: Connecting Policy and Test Scores” is available for purchase now.