Indicators and the Difficulty With Them

Evaluators that are working in education contexts are often required to use externally created criteria and standards, such as GPA targets, graduation rates, and other such metrics when evaluating program success. These standardized goals create a problem that program directors and their evaluators should be on the lookout for. It is called goal displacement, which occurs when one chases a target indicator at the expense of the other parts of a larger mission (Cronin and Gugimoto, 2014). An example of goal displacement was provided in a recent blog post by Bernard Marr (https://www.linkedin.com/today/post/article/20140324073422-64875646-caution-when-kpis-turn-to-poison?trk=mp-author-card).

“Another classic example comes from a Russian nail factory. When the government centrally planned the economy it created targets of output for the factory, measured in weight. The result was that the factory produced a small number of very heavy nails. Obviously, people in Russia didn’t just need massively big nails so the target was changed to the amount of nails the factory had to produce. As a consequence, the nail factory produced a massive amount of only tiny nails.”

The lesson here is that we have to understand that indicators are not truth, they are pointers to truth. As such, it is bad assessment practice to only use a single indicator in assessment and evaluation. In the Russian nail factory example ,suppose what you were really trying to measure was success of the factory in meeting the country’s needs for nails. Clearly, even though the factory was able to meet the targets for the weight or quantity indicators, it failed at its ultimate target, which was meeting the need for the right kind of nails.

I was moved to write about this issue when thinking about a real-world evaluation of an education program that has to meet federally mandated performance indicators, such as percentage of students who meet a certain GPA. The program works with students who tend towards low academic performance and who have little role modeling for success. In order to fully understand the program’s value, it was important to look at not only the number of people who met the federal target, but also statistics related to how students with different initial GPAs and different levels of parental support performed over time. This trend data showed the real story: Even those students who were not meeting the uniform federal target were still improving. More often, the students with less educated role models started with lower GPAs and increased those GPAs over time in the program, while students who had more educated role models, tended to start off better, but did not improve as much. This means that through mentoring, the program was having immense impact on the most needy students (low initial performers), whether or not they met the full federal standard. Although the program still needs to make improvements to reach the federal standards, we now know an important leverage point that can help the students improve even further – increased mentoring to compensate for a lack of educated role models in their personal lives. Thus we were able to look past just the indicator, and found what was really important to the program’s success!

Wouters, P. (2014). 3 The Citation: From Culture to Infrastructure. Beyond Bibliometrics: Harnessing Multidimensional Indicators of Scholarly Impact, 47.