Assessing the Sustainability of Advanced Technological Education (ATE) Projects and Centers

Download: Checklist_2013_ATESustainability (1)

A Checklist for Assessing the Sustainability of Advanced Technological Education (ATE) Projects and Centers

A product of NSF Targeted Research Grant # 1132099, Wayne W. Welch, PI

Introduction

A checklist is a list of factors, elements, components, or dimensions that should be considered when performing a certain task, for example, when doing a program evaluation.1 They are mnemonic devices, that is, a list of reminders of things to consider when making judgments about an object, for example, the sustainability of an ATE grant.

In this report, I describe a checklist that can be used to assess the sustainability of work carried out during the implementation of the Advanced Technological Education program funded by the National Science Foundation (NSF). The checklist comprises three sections; an introduction, the checklist itself, and an interpretive table. It is designed to help identify areas where sustainability has occurred, improve sustainability success, provide accountability evidence, and determine how a specific grant compares with the sustainability of other ATE projects and centers.

Sustainability is one of the goals of the Advanced Technological Education program. The program solicitation (National Science Foundation, 2011) states:

A project or center is expected to communicate a realistic vision for sustainability and a plan to achieve it. It is expected that at least some aspects of both centers and projects will be sustained or institutionalized past the period of award funding. Being sustainable means that a product or service has been developed that the host institution, its partners, and its target audiences want continued. To be sustainable is to ensure a center or project’s products and services have a life beyond ATE funding. (Introduction, para. 5)

The checklist presented here is based on work carried out during an ATE Targeted Research project that studied the impact and sustainability of the ATE program (Welch, 2012). One of the products of this research was a scale that measures ATE sustainability. The scale, which consists of a set of Likert-type statements, has been shown to meet generally accepted standards for validity, reliability, and usability.

The items for the scale, and for the checklist, consist of statements about sustainability provided by nearly 50 ATE PIs and others knowledgeable of the ATE program. These included NSF program officers, members of the projects’ advisory panel and several researchers interested in the sustainability of NSF-supported programs. These people were asked to identify the products and activities of ATE grants that will be or were in place three years after the ATE grant ended. Approximately 70 statements were obtained from this process and were used to map a domain of content for ATE sustainability. These statements became the elements or the dimensions of sustainability. They are the things to consider when making judgments about continuation of ATE grants.

The main headings of the elements are:

• Programs (content and delivery mechanisms)
• Collaborations
• Educational materials
• Faculty
• Facilities
• Students
• Institution (administration, change)
• Revenue

Because of response burden limitations, we could only use about two dozen items to sample the sustainability domain. That amounted to about three items for each main heading. ATE PIs and test development experts reviewed the initial statements, and 23 of them were selected for the final checklist.

We followed generally accepted item development procedures in developing the checklist (Worthen & Sanders, 1987). Selection criteria included clarity of the statement and fit to the dimensions of the domain of content. Both positively and negatively worded items were included.

The statements are printed as quotes from ATE peers. Responders are asked to indicate if they agree with the statement, disagree, are uncertain, or if the statement is not applicable to their situation. These statements and a place to record responses follow as Part II of this report.

Although this checklist was created and normed for use by principal investigators, other groups can use it as well. Staff members, collaborators, participants, and evaluators each have their own perspectives. The checklist will aide your understanding of those perspectives and give you a more comprehensive understanding of sustainability as it pertains to your grant work.

The checklist can be used in several ways. It can serve as a reminder of the many ways that ATE sustainability can occur. For example, it may not be obvious that better prepared students are a possible product of a sustained grant. They are part of the legacy of an ATE project or center and will continue to be involved in advanced technological education after a grant ends. Similarly, a collaboration that a project has established with a local industry that continues after the grant ends is another example of ATE sustainability.

The ATE checklist also can be used in a formative sense to monitor progress toward achieving your sustainability goals during the implementation of a grant. This will help identify strength areas as well as places where more work needs to be done before the project is completed. In addition, it can also be used in a summative way to determine whether the sustainability goals were achieved.

Another way to interpret individual responses is to use the Interpretative Table that follows the checklist. (See pages 5 and 6.) The table displays the percent of responders that replied agree, uncertain, disagree, or not applicable to each statement. This information was obtained in the spring of 2010 from 216 PIs of active and ended ATE grants.

Checklist responders can use these results to determine how their sustainability findings compare to those of their peers. The table also includes a place to identify the areas of success or possible limitations in continuing a project or center’s efforts. It also calls for the development of a plan to address sustainability concerns, that is, how to maintain strength areas and to improve problem areas.

References

National Science Foundation. (2011). Advanced Technological Education (ATE). Retrieved from Program Soliciation NSF 11- 692.

Scriven, M. (2007). The logic and methodology of checklists. Retrieved from Evalu_ATE.

Welch, W. W. (2012). Measuring the sustainability of the advanced technological education (ATE) program. Retrieved from Evalu-ATE.

Worthen, B. R., & Sanders, J. R. (1987). Educational Evaluation: Alternative Approaches and Practical Guidelines. New York: Longman, Inc.

1 This definition is based on the writings of Michael Scriven. See (Scriven, 2007)