Evaluating Impact: How I Moved From Pipeline to Interstate

John Sener of Sener Knowledge LLC, an external evaluator for over a dozen ATE and other NSF grants over the past eight years, discusses how to move beyond the “pipeline” metaphor for describing technician education and adopt the more useful “interstate” metaphor instead.

Evaluate an NSF ATE center or project long enough, and eventually you’ll hear the word pipeline—as in, “closing gaps in the education pipeline,” or “increasing the cybersecurity workforce pipeline.” The pipeline metaphor describes the ATE technician education process pretty well to some extent:

A) Students proceed through an NSF-funded ATE program “pipeline” in well-defined cohorts on a relatively uniform timetable.

B) The pipeline feeds program graduates to employers, often in a relatively limited number of local or regional employers in the related field.

C) Evaluators aim to document that B is the result of A.

However, I have long been dissatisfied with the pipeline metaphor because it traps its users into limited thinking, making it easy to overlook important sources of project impact. As noted in my book, The Seven Futures of American Education, many students exhibit characteristics that reflect a greater degree of choice than the pipeline metaphor implies: “stopouts” who drop in and out of school, “swirlers” who attend multiple institutions, “stay-longers” who exceed the prescribed time period for program completion. I’ve found that an interstate highway is a much more useful metaphor to understand learners who:

• Choose among multiple entry and exit points rather than following a prescribed path;
• Travel at different speeds and on different schedules through their education programs;
• Sometimes seek alternate routes and multiple destinations during their educational journeys.

Here are some ways I use the interstate metaphor to find sources of project impact.

Pipelines move their cargo from Point A to Point B, while interstates support two-way traffic; look in both directions for indications of project impact, for instance:

• Career changers with bachelor’s or graduate degrees returning to community colleges for additional training or certification
• Four-year students mentoring two-year students to prepare for student competitions
• Community college students mentoring high school students or serving as judges for high school student competitions
• Business and industry practitioners getting involved on community college curriculum advisory boards or forming three-way partnerships with faculty and students to enhance the learning/assessment experience or create knowledge collaboratively

Expand the realm of acceptable outcomes. ATE projects have significant impact beyond program completion or employment placement. Students sometimes find jobs before completing a program. Alternatively, intermediate outcomes—such as certificate or multiple course completion—may also indicate progress, especially for students returning to college to enhance their existing professional prospects.

Look in other places for impact. Extracurricular activities, such as student competitions, clubs, or organizations, are one good place to look. I sometimes think of such places as “toll booths”—activities where impact can be measured more easily as students pass through them.