Click here to login if you're an NAE Member
Recover Your Account Information
Author: Lisa R. Lattuca, Patrick T. Terenzini, J. Fredericks Volkwein, and George D. Peterson
ABET EC2000 has radically changed the evaluation of undergraduate engineering programs.
Most engineers today are likely to recall their undergraduate years as a period of intense mathematical and theoretical study. Until the 1940s, however, the engineering curriculum at most colleges and universities was mostly practical, with the emphasis on engineering design rather than on engineering science and mathematical applications. The field, and undergraduate education programs, changed dramatically with increased federal support for university research after World War II. As engineering schools hired faculty to teach and conduct scientific research, the number of faculty members with industry experience declined. Eventually, courses in advanced mathematics and theory replaced practical courses in machining, surveying, and drawing (Prados et al., 2005).
The strong emphasis on technical knowledge and skills, and resulting improvements in defense-related technologies and applications, served the United States well during the war and postwar years, but, by the 1980s, economic shifts from defense to commercial applications left engineering employers dissatisfied. New graduates were technically well prepared but lacked the professional skills for success in a competitive, innovative, global marketplace. Employers complained that new hires had poor communication and teamwork skills and did not appreciate the social and nontechnical influences on engineering solutions and quality processes (McMasters, 2004; Todd et al., 1993). Several national reports recommending changes in engineering education appeared (e.g., ASEE 1987; NRC, 1985; NSB, 1986; NSF, 1989).
EC2000 emphasizes learning
outcomes and accountability.
Despite a significant investment by the National Science Foundation (NSF) in engineering education, engineering schools were slow to respond to these criticisms, and deans of many colleges of engineering argued that rigid accreditation standards focused on curricular requirements and credit hours were hindering their efforts to make necessary changes. In 1992, a representative of two groups of deans of major engineering schools met with the leadership of ABET, then the Accreditation Board for Engineering and Technology, to express their concerns. Some deans suggested that a different accreditation organization was needed.
Inside ABET, members of the agency’s Industry Advisory Council echoed the criticisms of members of the engineering community, and ABET’s leadership responded. In 1992, the outgoing and incoming presidents of ABET jointly established an Accreditation Process Review Committee (APRC) to advise the organization on making accreditation criteria more flexible without compromising educational quality. Members of APRC were drawn from ABET and its Engineering Accreditation Commission, as well as from academia and industry. APRC identified three major barriers to change: criteria, process, and participation.
In 1994, with funding from NSF and industries represented on the ABET Industry Advisory Council, ABET sponsored three consensus-building workshops to address these concerns. More than 125 individuals, including university presidents, deans, and faculty members; industry leaders; private practitioners; professional and technical society liaisons and executive directors; state registration board members; and government researchers and regulators in technical fields participated in the workshops with ABET leaders, commissioners, and board members. Recommendations from the workshops, published in A Vision for Change in early 1995, became catalysts for the development of new accreditation criteria, which were circulated to the engineering community a few months later. Following a period of public comment, the ABET Board of Directors approved Engineering Criteria 2000 (EC2000) in 1996.
The new criteria radically altered the evaluation of undergraduate engineering programs, shifting the emphasis from curricular specifications to student learning outcomes and accountability. Under EC2000, engineering programs must define program objectives to meet their constituents’ needs. Rather than taking a cookie-cutter approach, the new criteria accommodate differences and innovations in programs. To ensure accountability, each engineering program is required to implement a structured, documented system for continuous improvement that actively and formally engages all of its constituents in the development, assessment, and improvement of academic offerings. Programs must publish specific goals for student learning and measure their achievement to demonstrate how well these objectives are being met. ABET was one of the first accrediting bodies to implement such a radical change in its accreditation philosophy.
Engineering Education Then and Now
To evaluate the impact of the new outcomes-based criteria on engineering graduates, ABET commissioned the Center for the Study of Higher Education at Pennsylvania State University to conduct an assessment. The study addressed two issues: (1) the impact (if any) of EC2000 on engineering students’ preparation for entering their professions and (2) whether engineering programs had changed their organizational and educational policies and practices in ways consistent with the EC2000 specifications (Lattuca et al., 2006).
Following the conceptual model developed for the study (Figure 1 - see PDF version for all figures), if EC2000 had the desired effects, one would expect to see a variety of changes rippling through the engineering educational process. The most immediate changes would be in engineering programs, as faculty members revised their curricula and instructional practices to promote the learning outcomes specified in EC2000 Criterion 3.a–k (Figure 2). One might also expect to see a shift in faculty culture toward greater involvement in learner-centered activities, such as outcomes assessment and professional development to improve teaching. Similarly, one might anticipate changes in administrative policies and practices, such as more emphasis on teaching and learning in hiring, promotion, and tenure decisions.
If all of the program changes were under way, the logic of the conceptual model suggested that improvements would be apparent in students’ engineering education-related experiences. Like the changes in programs, the improvements would be consistent with achieving the learning outcomes specified by EC2000. Finally, such a transformation in engineering education over the past decade would be reflected in higher knowledge and skill levels among recent graduates, compared to those of their predecessors, on measures of the EC2000 outcomes.
A Pr?cis of the Study Design and Methods
The study design for Engineering Change (Lattuca et al., 2006) was based on information from graduates (both pre- and post-EC2000), faculty members, program chairs, deans, and employers; the goal was to obtain a 360-degree view of changes in engineering education since the implementation of EC2000. A two-stage, disproportionate, stratified, random-sampling design ensured that sample sizes were adequate and that program disciplines, accreditation review schedules, and participation in an NSF-funded coalition during the 1990s were all taken into account. Adjustments were made to include institutions that serve historically underrepresented populations.
Data were collected from 40 colleges or schools of engineering that offer more than 200 engineering programs in aerospace, chemical, civil, computer, electrical, industrial, and mechanical engineering. Analyses were based on survey information from 1,243 faculty members (42 percent response rate), 147 program chairs (72 percent), 5,494 graduates in the class of 1994 (42 percent), 4,330 graduates of the class of 2004 (34 percent), 39 deans (98 percent), and 1,622 employers (population size and, thus, response rate unknown). By social science standards, these response rates were highly respectable, and the numbers of respondents were more than adequate for valid analyses. Statistical adjustments were made to correct for response bias by sex, NSF coalition participation, and discipline among faculty members, as well as for sex and discipline among graduates. Adjustments were also made for varying response rates among institutions. For a detailed description of the study design and methods, see Lattuca et al. (2006). The major findings are summarized below.
Changes in Programs
Curriculum and Instruction
According to program chairs and faculty members, engineering program curricula have changed considerably “over the past seven to 10 years.” Although few programs have relaxed their emphasis on foundational skills in mathematics, science, and engineering science, both program chairs and faculty members reported increased emphasis on nearly all of the professional skills and knowledge sets associated with EC2000 Criterion 3.a–k. For example, 75 percent or more of program chairs reported “some” or “significant” increases in emphasis on communication, teamwork, use of modern engineering tools, technical writing, lifelong learning, and engineering design. Similarly, more than half of the faculty respondents reported “some” or “significant” increases in their emphasis on the use of modern engineering tools, teamwork, and engineering design in courses they teach regularly.
Teaching methods also changed substantially. One-half to two-thirds of faculty respondents said they increased “some” or “significantly” their use of active learning approaches, such as group work, design projects, case studies, and application exercises. They also relied less on lecturing and textbook problems (Figure 3).
Faculty members reported increased involvement in the past 10 years in assessing student learning and using assessment results to guide program improvement. Program chairs, moreover, reported high levels of faculty support for assessments and related decision-support practices. More than 75 percent of program chairs estimated that either “more than half” or “almost all” of their faculty members supported efforts to ensure continuous quality improvement, and more than 60 percent of program chairs reported “moderate” to “strong” support among their colleagues for assessing student learning. Faculty members corroborated that finding. Nearly 90 percent reported being personally involved in outcomes assessment, and more than half reported “moderate” to “a great deal” of personal effort in this area. Nearly 70 percent thought their level of effort was “about right.”
Another change in faculty culture included more engagement in professional development activities focused on teaching and learning. More than two-thirds of faculty respondents reported that, compared to seven to 10 years ago, they were reading more about teaching, and about half reported more frequent involvement in professional development activities, such as workshops on teaching, learning, and assessment and projects to improve engineering education. Depending on the activity, one-fifth to one-quarter of faculty members said they had increased their teaching-and-learning-related professional development efforts in the past five years.
Administrative Practices and Policies
The institutional reward system may be the single most important influence on how much time and energy faculty members devote to their teaching, research, and service. Thus, it is important to know if faculty members believe their efforts to improve teaching and undergraduate education are important considerations in decisions about promotion, tenure, and merit-based salary increases.
About half of the program chairs and faculty surveyed said they saw no change in the reward system over the past decade. Roughly 25 percent of faculty respondents said there was less emphasis on teaching since the mid-1990s; another 25 percent, however, said that the emphasis on teaching in the reward system had increased “some” or “significantly” over the same period. Faculty perceptions varied by academic rank. Senior faculty members were more likely to report an increase in emphasis on teaching in promotion and tenure decisions; untenured faculty were more likely to report less emphasis. About one-third of program chairs said the emphasis on teaching in promotion, tenure, and salary and merit decisions had increased over the past decade.
Changes in Student Experiences
According to the logic of the study’s conceptual framework, changes relating to curriculum and instruction, faculty culture, and program practices and policies should be reflected in the experiences of graduates. Students who graduated in 2004 differed significantly from their predecessors in eight of 10 experiences inside and outside the classroom. The 2004 graduates reported more active engagement in their learning, more interaction with instructors, more faculty feedback on their work, more time spent studying abroad, more international travel, more involvement in engineering design competitions, and more openness in their programs to new ideas and people. Although the differences tended to be small, they persisted even after adjustments were made for an array of graduate and institutional characteristics.1
The 2004 graduates, however, reported experiencing a less welcoming diversity climate than their predecessors. This finding may be attributable to several factors: differences in the sex and/or racial/ethnic mix of the two cohorts, differences in awareness of diversity issues, and/or differences in the willingness to discuss and challenge prejudices or discrimination. The evidence provides no basis for evaluating these possible explanations. The 1994 and 2004 groups gave the same evaluations of their instructors’ teaching skills and the number of hours spent in cooperative or internship activities.
Changes in Learning Outcomes
Given all of these changes, are recent graduates better prepared to enter the profession than their counterparts were a decade earlier? Figures 4, 5, and 6 contrast the achievement levels graduates reported on scales reflecting the Criterion 3.a–k learning outcomes.2 For nine different measures,3 the differences were consistent with the assumption that EC2000 is a force for change in engineering education. All differences were statistically significant (p < .001), ranging from +.07 to +.80 of a standard deviation (sd) (mean effect size = +.36 sd).4 Five of the nine effect sizes exceeded .3 sd, which would be characterized as “moderate” in the social sciences.
The largest differences over the past decade were in five areas: awareness of societal and global issues (effect size = +.80 sd), awareness of ethics and professionalism (+.46 sd), group skills (+.47 sd), and applying engineering skills (+.47 sd). The smallest increases were in the ability to apply mathematics and sciences (+.07 sd). This last difference, although small, is noteworthy because some faculty members and others had expressed concerns that focusing on the development of the professional skills specified in EC2000 might detract from the teaching of the science, math, and engineering science skills that are the foundations of engineering. This finding not only indicates no decline, but shows a slight rise in knowledge and skills in these areas.
Linking Changes and Learning Outcomes
Multivariate statistical analyses provide moderate to strong evidence that changes in programs and student experiences were empirically linked to differences in learning. The evidence for this remained, moreover, even after controlling for a battery of students’ precollege characteristics and for the characteristics of the institutions they attended. Fourteen of 16 program and faculty changes had statistically significant effects on at least one, and as many as five, student experiences, even after controlling for other factors. The most frequent and influential programmatic changes were an increase in emphasis on foundational knowledge and project skills in program curricula, less reliance on traditional pedagogies, an increase in active and collaborative pedagogies, and an increase in programmatic emphasis on assessment for improvement. Although student experiences varied, a consistent pattern emerged. The observed shifts over the past decade in program curricula, practices and policies, and faculty activities and culture related positively, at statistically significant, if sometimes small or moderate, levels, even after other factors were taken into account.
Finally, undergraduate program experiences, both inside and outside the classroom, were clearly linked to what and how much students learned. Nine of 10 measures had statistically significant, positive, and sometimes substantial influences on all nine of the measures of the EC2000 learning outcomes. The clarity of instruction, the amount of interaction with and feedback from instructors, and engagement in active and collaborative pedagogies had the strongest influence on learning.
Out-of-class experiences, however, also influenced student learning in important ways. The most important non-classroom activities were internships and cooperative education experiences, participation in design competitions, and active participation in student chapters of professional societies or associations. These non-classroom activities significantly and positively affected learning in six or more of the nine skill areas measured, although the magnitude of the effects were smaller than those of in-class experiences.
Employers’ Views of Change
The 1,622 employer respondents in the study represented a wide range of geographic locations, industry types, company size, educational attainment, and engineer-evaluation experience. The employer survey asked three primary questions.