Mistakes Made and Lessons Learned, Part I – Working with Your Evaluator

When I assumed the PI-ship of the Scenario-Based Learning Project in 2006, I had worked closely with the prior PI as the project’s instructional designer, knew I enjoyed the ATE community, shared their vision of an innovative 21st century technician workforce, and had management and teaching experience. What more was there to know? A lot, as it turned out. It quickly became apparent to me that the world of ATE projects and centers was a different place than any I had worked in before.

When I took over as PI on the second grant proposal, the former PI suggested we use the evaluators from a previous non-ATE grant she had led. Big mistake. Those evaluators reported to my National Visiting Committee (NVC) during our initial committee meeting that they didn’t have any results to report because they did not plan to collect data until the end of the project year. My NVC was not happy. I was not happy. My stakeholders were not happy.

How did this happen? In my naivety, I didn’t even discuss the evaluation with the evaluators beyond an initial outline of a plan involving questions to be answered by the evaluation—I thought the evaluators knew what they were doing because they were evaluators. I didn’t understand the complex nature of the profession of evaluation. Since then I have joined the American Evaluation Association, attended their annual conference, and regularly attend EvaluATE’s webinars. I made a mistake, learned from it, and the project improved.

I quickly learned that some evaluators and funders are all about the summative report. The project said they would do A, here is the data to show they did or did not do A. End of report. In contrast, the ATE program is interested in how we are doing as we progress through our work. Formative reports from the evaluator serve as a check-in on where you are in your work plan and outcomes. Your evaluator needs to be a critical friend—an advisor who keeps a distance and is critical where needed yet still supportive with ideas, solutions, and contacts.

Choose an evaluator with ATE experience and expertise in collecting and analyzing the kind of data you will need. Confirm who will collect the data, how it will be collected, from whom, and when early in your discussions with your evaluator and project team. Confirm that your evaluator is willing to provide mentoring to your data collection team as needed if you decide to collect the data yourselves and have the evaluator do the analysis (saves money but requires time).

You might need interim reports/summaries from your evaluator for meetings with stakeholders, your NVC, advisory boards, the ATE annual survey, and your annual and final reports. It is a good idea to align your data collection with your reporting needs to best use your resources.

Learn more about the process of evaluation every chance you get. Choose an evaluator with the expertise appropriate to your project or center. Think of your evaluation as a resource and your evaluator as an ally to help you and your team to create the best project or center possible.