Alumnus talks about improving college preparation with tougher test system (in Michigan)
It’s at least one variable in the calculation for young people predicting their post-graduation success: the standardized test.
In Michigan, education officials recently decided the assessment administered to high school students — or at least what it takes to pass it — does not accurately indicate whether students are prepared for college and careers in today’s workforce.
They made the controversial move to raise cut scores, otherwise known as the levels of achievement students must reach to be considered proficient, for the 11th grade Michigan Merit Exam (MME) as well as all grades and subject areas tested by the Michigan Educational Assessment Program (MEAP).
One of the lead architects behind the change, Joseph Martineau, is a College of Education graduate.
Martineau became the first full-time psychometrician hired by the Michigan Department of Education in 2004, the same year he received his Ph.D. in Measurement and Quantitative Methods at MSU.
He has helped grow the department’s in-house capability to manage the design and validity of assessments and now serves as director of the Bureau of Assessment & Accountability, overseeing all aspects of educational testing and evaluation for the state.
Martineau sat down with the New Educator to explain the rationale for raising cut scores, a major turning point for Michigan and, possibly, a case study for making the link between standardized test results and college success more realistic nationwide.
Question: Why did the Michigan Department of Education recommend increasing cut scores?
Answer: We thought it was important to move toward more rigorous cut scores based on the changing nature of the economy in Michigan. We had cut scores that were based on being able to succeed in an old-line manufacturing economy. When the recession happened, it became clear that kind of economy was going away in Michigan. We could no longer indicate that students were ready for post-high school work with only a basic set of skills; almost any job that is going to have a reasonable wage is going to require much more. We need to focus on preparing students for postsecondary technical training or college.
Q: Does this assessment reform align with changes to curriculum or standards?
A: Yes. We had new high school content expectations in 2006. Those were based on what students need to know and be able to do to succeed in college. (So) we also set the new cut scores to be reflective of college readiness. We actually benchmarked our cut scores against success in college – so that being proficient on the Michigan Merit Exam gives us a good indication if a student is going to perform well in college.
Q: What methods were used to identify the new cut scores?
A: We used something called signal detection theory. The purpose of signal detection theory, at least in this instance, was to maximize consistent classifications from high school to college. So we wanted to maximize the consistency of being considered proficient with achieving a B or better in the first credit-bearing course.
Essentially, we looked at the score on the MME that would maximally consistently classify students as both proficient and getting a B or better in their first credit-bearing course OR not being proficient and not getting a B or better. Essentially you can interpret the new cut scores as the score on the MME that gives you a 50 percent probability of a B or better in your first credit-bearing course. So it’s very likely that they will pass the course.
That’s what we did for transitioning from high school to college. We did the same kind of thing… We mapped backward from high school down to 3rd grade using signal detection theory as well so that if you are proficient in 8th grade, you also are maximally likely to be proficient in 11th grade… and so on down the line.
No one else has used signal detection theory to do this kind of work. Other people have used something called logistic regression. The purpose is to arrive at a specific score with a given probability of success later on. We thought it was more important to do maximally consistent classification so that you have consistency from one grade to the next and from high school to college.
Q: How will the change affect achievement levels for Michigan students?
A: There will be a significant drop in the proficiency rates, but we do anticipate that educators will rise to the occasion. We’ve seen it in the past. The MEAP has been around since 1971, and cut scores have been set many, many times. Educators simply rise to the occasion.
We have had relatively rigorous cuts scores in the high school writing test since 2007, and we’ve seen educators and students really improving on that assessment. So we anticipate that we will see, at first, a dip – a significant dip – in the proficiency rates, but then we will see that rise over time.
Q: How will it affect our performance under No Child Left Behind?
A: It will likely throw many more schools into not making Adequate Yearly Progress, unless we get the waiver that we are going for. Under No Child Left Behind, all schools have to have 100 percent of students proficient by 2014. That’s… quite a task. We are looking for a waiver to extend the time period by at least 10 years and give schools a target to get to that is much more reasonable.
Q: What are other states doing?
A: Tennessee and New York recently did something similar; they raised their cut scores to represent college and career readiness. There are other states that raised their cut scores to represent proficiency on the NAEP (National Assessment of Educational Progress). We chose to peg them to career and college readiness because that is an outcome that really matters in students’ lives. Whether or not you achieve proficiency on the NAEP is not tied to some external success factor.
We are likely in the top tier of states in terms of the rigor of our cut scores now, whereas we were in the middle to bottom in the past. We have higher expectations, and I think we are being more transparent about whether students are prepared for the next level of education. If their test scores in high school show they are proficient then there is a strong probability of success in college, and if they are proficient in a lower grade then there is a strong possibility that they will be proficient in the next grade.
Q: You are the father of five students attending public schools. How do you feel, personally, about the decision to raise cut scores in Michigan?
A: It took a lot of courage for the State Board of Education to do it, because this will have an impact on a lot of students and a lot of schools. It would have been really easy just to stay with the status quo and not have to worry about people asking about a student, “Well, he used to be considered proficient, so why isn’t he now?”
But we had heard from enough parents saying, “Why did my student have to take remedial math or receive instruction in remedial reading when the state told me they were proficient on the MME?” I think we simply needed to be more transparent in saying, this is what students need to succeed today and that should be reflected on state tests.
Q: How did your education at MSU prepare you for your job?
A: There were two things that were really critical. One is the quantitative methods I learned; you have to have them to be in this type of position. The second is I was encouraged by my advisor (Mark Reckase) and Ken Frank to spend a lot of time working with substantive researchers and not just methodological researchers. I did a lot of work with Nell Duke, Victoria Purcell-Gates, Yong Zhao and other faculty members, in actually looking at literacy or technology in education. That got me to a place where I could talk beyond the methods. That advice was critical to helping me survive in an environment where there is as much policy as there is psychometrics.