In This Issue
Winter Issue of The Bridge on Frontiers of Engineering
December 25, 2021 Volume 51 Issue 4
The NAE’s Frontiers of Engineering symposium series forged ahead despite the challenges of the pandemic, with virtual and hybrid events in 2021. This issue features selected papers from early-career engineers reporting on new developments in a variety of areas.

Are We Making a Difference? A Case Study of Assessment in Innovation Training

Monday, January 3, 2022

Author: Lyn Denend, Paul G. Yock, Josh Makower, Dan E. Azagury, and James K. Wall

We are working to develop systematic, comprehensive assessment instruments for our engineering innovation and entrepreneurship training program.

Over the past several decades, universities in the United States and abroad have seen exponential growth in training programs in innovation and entrepreneurship. Although substantial talent and resources have been invested in creating a variety of offerings, there is no consensus on the best means of measuring the impact of these programs on the careers of the trainees (Lyons et al. 2015). We present a case study that tracks how we developed and deployed assessment methods for a program in health technology innovation over a 20-year period.

Background

Stanford Biodesign was launched in 2000 as an inter­disciplinary postgraduate training experience in the medical technology (medtech) space.[1] The design of the program was shaped by several major trends in ­industry and academia. In industry, medtech was experi­encing a remarkable period of growth, with robust startup ­activity in California’s Silicon Valley and other technology hubs. Students from engineering, medicine, and business were increasingly attracted to careers in this dynamic sector. At the same time, there was a developing awareness on university campuses that innovation could be approached as a discipline, with design thinking and entrepreneurship blended into popular new course offerings at both the undergraduate and graduate levels (Kelley and Kelley 2013).

In the context of this emerging landscape, Stanford Biodesign chose an initial focus on postgraduate training. There appeared to be an educational “white space” for engineers and physicians at this level who wanted to develop careers in medtech entrepreneurship or intrapreneurship (innovation-based careers in existing companies).

At the time, courses in innovation and design at the undergrad and graduate levels did not offer sufficient depth in critically important areas such as intellectual property, regulation, reimbursement, and business planning. Engineers or physicians who directly entered industry careers were typically channeled in their ­early years into focused technical roles and did not gain enough breadth from their experiences to feel confident in launching entrepreneurial careers. And although business schools were increasingly offering courses on innovation and entrepreneurship, there typically were not specific offerings in the medtech domain.

The Biodesign Innovation Fellowship

The initial structure of the Biodesign Innovation Fellow­ship was unique in several ways. We recruited engineers and physicians who had finished their graduate training. Some of the engineers had industry experience; physicians generally had completed their residency. Fellows were assembled in teams of four, combining physicians and engineers from different backgrounds. Fellows received a stipend on a postdoctoral-level scale for this 1-year experience, which resulted in a certificate of completion.

The training curriculum (dubbed the “biodesign innovation process”) was a medtech-specific version of needs-based design thinking (Yock et al. 2015).[2] The fellows learned and practiced the process through a project-based educational experience intended to equip them to

  1. identify and prioritize important unmet health-­related needs,
  2. invent and vet new technologies to address the most compelling of these opportunities, and
  3. develop detailed implementation plans to bring the concepts into further development and commercialization (see figure 1).

The fellowship drew heavily from the local medical technology ecosystem for experts, mentors, and ­coaches with real-world experience to help train and coach the fellows as they advanced their project through this process.

Early Program Evaluation

Participant Feedback

For the first several years, program evaluation was primarily limited to a year-end feedback session with the outgoing fellows. Program leaders did not perceive a need for significant change, since fellows completing the training were successfully launching medtech innovation careers in industry and academia. In some cases, fellows were successful in founding and leading their own startups based on technologies they originated during their training.

Denend et al figure 1.gif

After approximately 10 years of gradual program evolution, faculty and core advisors recognized the need to provide a more systematic evaluation of career outcomes, as well as an assessment of the elements of the program that could be influencing those outcomes. But determining what to measure proved difficult.

Retrospective Analysis

We did a retrospective analysis that focused primarily on the startup activity of the individuals trained in our innovation fellowship as well as a graduate course and two global partner programs[3] that offered similar curricula. The results (Brinton et al. 2013) showed a solid tally of entrepreneurial successes. By this point 26 new companies (of which 15 had been launched by participants in our innovation fellowship) had originated directly from the combined training programs, $200 million in funding had been raised, and, most importantly, the new technologies had been used to help 150,000 patients.

Beyond startup formation, alumni who chose other career paths appeared to be placing well in existing companies or in university faculty positions. A few of the academic alumni had launched and were leading similar medtech-oriented programs at other universities. 

These data, though encouraging at a macro level, did not directly address the actual learning outcomes of the fellowship program. Specifically, they did not indicate to what extent the skills acquired in the program equipped our fellows to assume leadership positions as innovators in startups, medtech companies, universities, medical centers, and organizations. 

Approach to More Rigorous Evaluation

In 2014 we initiated a three-pronged approach to more systematically evaluate and improve the fellowship program:

  1. conduct an in-depth qualitative evaluation of our educational inputs;
  2. survey program alumni with respect to learning objectives, skills acquisition, and career productivity; and
  3. track and assess the career pathways of our alumni and compare them to a matched group of individuals who did not complete the fellowship.

Assessment of Educational Inputs

For this qualitative assessment, we sought to identify unmet program needs through alumni focus groups, in-depth interviews with current fellows, and interviews with program mentors from the medtech industry. We sorted, coded, and summarized the information into major themes for improvement, reported below.

Survey of Alumni Learning Objectives,
Skills Acquisition, and Career Productivity

To assess how well we met learning objectives, building relevant skills, and positioning the fellows for productive medtech careers, we conducted a detailed survey to probe alumni opinions about how beneficial the fellow­ship program had been for their careers, how well it achieved specific educational goals, and how effective it was in teaching select skills.

Additionally, we sought to capture self-reported data on important productivity metrics, such as number of issued patents, patents licensed for human testing or commercial development, peer-reviewed publications, board seats, roles as clinical investigators for health technologies, and roles teaching/training others on the biodesign innovation process.

In an effort not to overburden alumni, we decided to conduct this survey every 3–5 years; it was first conducted in 2016 (Wall et al. 2017), and then again in 2020.[4] With fellowship leaders’ personalized outreach we achieved a response rate over 90 percent for both surveys.

Comparative Career Tracking

To obtain more specific information about alumni career pathways, we initiated an effort to track their career outcomes. For those in industry, we gathered job titles and company names from public sources such as LinkedIn and company websites. For alumni on academic paths, we captured data from university websites.

For comparison purposes, we also collected similar data from public sources for finalists who interviewed for the fellowship but were not selected (Wall et al. 2017).[5] As background: Each year we interview a finalist group that is approximately twice the size of our final cohort (e.g., 24 finalists for 12 fellows chosen). These finalists are selected from an applicant pool of approximately 150 individuals. While clearly these two groups (fellows and finalists) are not matched in any statistically significant sense, the intent of this analysis was to explore any large differences in individual career trajectories that might lead to a better understanding of candidate selection and/or impacts of the fellowship program.

In parallel, we continued collecting data on the companies launched based on technologies conceived by the fellows during the training program.[6] Company metrics are tracked by an annual survey of fellowship alumni who founded medtech startup companies based on their fellowship projects and reflect all stages of funding until acquisition or closure. The data included number of companies founded, total funding raised, number of employees hired, status/stage of the company, and total number of patients helped by the technologies as self-reported by the founder(s) (based, for example, on their records of patients enrolled in clinical trials and sales data).

Findings and Implications

Educational Inputs: Categories for Improvement

Our initial interviews and the 2016 survey indicated that the fellowship was generally equipping fellows to be successful in a variety of roles in medtech innovation. However, program alumni and external mentors sent a clear message that the curriculum was not evolving at the same pace as the rapidly changing environment we were preparing our fellows to enter. Opportunities for improvement fell into five main categories:

  • Provide specific training and mentoring on health economic and commercial value as an increasingly important determinant of the commercial viability of a new technology.
  • Reorient the needs identification experience and the innovation process we teach toward health outside of the hospital—in clinics, ambulatory care centers, community facilities, home care, and rehabilitation centers.
  • More deeply address engineering fundamentals, including more advanced prototyping and technical ­derisking.
  • Provide mentoring “beyond the pitch” to support more in-depth training around commercial translation (team building, business planning, fundraising).
  • Increase the focus on individualized career placement and mentoring to help alumni succeed after completing the fellowship.

In response to this feedback, we created advisory teams composed of faculty and outside experts in each opportunity area to coordinate the development and implementation of relevant new content in the fellowship curriculum.

Denend et al figure 2.gif
FIGURE 2 Success rate of the Stanford Biodesign Fellowship in achieving learning goals (number/percentage of 2020 survey respondents). Source: Stanford Biodesign 2020 Innovation Fellowship Alumni Survey.

Progress against these improvement objectives was tracked as part of our qualitative survey work, allowing for further iteration. For instance, figure 2 from the 2020 survey shows that we need to do more to explicitly prepare our alumni for leadership roles. Accordingly, we recently expanded our career placement and mentoring workstream to include curriculum enhancements related to professional leadership development.

Learning Objectives and Skill Acquisition

Despite efforts to strengthen our educational input described above, the 2020 survey reinforced that we still have room to improve when it comes to the areas of engineering fundamentals, economic value, and commercial translation, as well as team building and conflict management. We identified permanent internal leads with specialized knowledge and expertise in these areas, and they further expanded the curriculum and established specific deliverables to enhance skill acquisition and the educational experience in these zones.

These workstreams were further improved through the creation of “bootcamps.” During each fellowship year the fellows receive 2–3 days of intensive didactic and project-based instruction in engineering, value, and commercial translation at the respective points in the process when these skills are most needed to advance their projects.

To address team building and conflict management, we expanded our use of a psychologist specializing in team and interpersonal dynamics to meet with and guide the fellowship teams on a biweekly basis.

Career Productivity

The measures of impact we consider most important are the numbers of patients helped by technologies invented by the fellows and alumni, followed by certain business metrics. By the time of the 2020 survey, 3.35 million patients had been helped by technologies initiated by fellows during the program, and an additional 1.4 million patients were helped by technologies subsequently invented by alumni (table 1). In aggregate, companies formed by fellows either during or after the fellowship have raised nearly $1.9 billion in funding, and account for 1850 new full-time positions.

Several other measures of career productivity are worth highlighting. Among the 137 who responded to the 2020 survey of 149 alumni, they hold a total of 93 board seats in health technology companies. Among responding clinician alumni, 1 in 2 have led clinical trials of a new technology. Among all responding alumni, 2 out of 3 have at least one US patent. And 91 percent of the responding fellows have formally or informally trained or coached others on aspects of the biodesign innovation process, with an average of 154 trainees per alumna/-us.

Denend et al table 1.gif

Finally, as an overall qualitative measure, 94 percent of alumni responding to the 2020 survey said they found the fellowship beneficial to their career development (2 percent said it was only slightly beneficial). And 91 percent reported that the fellowship was influential on their chosen career trajectory (5 percent indicated that it was only slightly or not influential on their choices).

Comparative Career Tracking

The comparison of alumni with the group of finalists (as described above) indicates that our alumni choose careers in the health technology field in greater numbers than candidates who interviewed for the fellowship but were not selected (figure 3). They also hold leader­ship positions at a higher rate than the comparison group (figure 4).

Limitations

We recognize that there are significant limitations in our approach to assessing the three domains of our program. For instance, data gathered from public sources are prone to error; self-reported input is inherently subjective; and our use of a comparison group that is not formally matched has major shortcomings. On this basis, our findings described here must be considered directional in nature.

An overriding challenge is the lack of validated assessment instruments in the area of engineering innovation and entrepreneurship training (as opposed to core engineering education).

An analysis of the wide variety of published approaches to assessment yielded a taxonomy of the instruments used (Purzer et al. 2016). Like the large majority of the 50-plus assessment approaches reviewed in that study, our analysis is primarily “summative” (designed to assess the program itself), “local” (addressing only one program), and survey-based. The authors of the 2016 analysis raise the concern that few studies provide any validation of the assessment instrument used and suggest that a group comparison is one important method where feasible. Our study at least attempted a group comparison, albeit with the significant shortcomings mentioned.

Future Opportunities

Despite the limitations outlined above, our approach to gathering more expansive data about the effectiveness of our fellowship has been useful in several respects.

We now understand that tracking only the technologies/companies founded out of our fellowship—the easiest data for us to “count”—was leading us to be complacent about the design and conduct of our training program. Through detailed surveys and focused interviews, we uncovered areas in our curriculum that needed enhanced attention. Similarly, we realized that we were missing and/or undervaluing substantial alumni career activities other than startup formation.

This work also has motivated us to explore a different category of skills assessment. As explained, our fellows learn and practice an innovation process that begins with identifying important unmet health needs. We teach the use of specific tools for needs characterization, including the creation of a need statement and development of need criteria that specify (in advance) the features that a successful invention will require. In conjunction with providing training on how to use these tools, we teach how to perform observations and conduct stakeholder interviews to augment secondary research and provide the background for needs characterization.

Although we have evolved our teaching approaches in these areas, we have not yet developed objective assessment techniques for evaluating the quality of observations and interviews or the resulting need statements and criteria. To date, this type of evaluation has been subjective and dependent on the perspective of the instructor or coach who is providing feedback. We now have an opportunity to develop and test a more objective approach to assessing work quality in these critically important areas.

Denend et al figure 3.gif
FIGURE 3 Career type by primary role, Stanford Biodesign alumni and comparison group (see text for explanation). Source: Stanford Biodesign 2020 Innovation Fellowship Alumni Survey.

Denend et al figure 4.gif
FIGURE 4 Leadership positions, Stanford Biodesign alumni and comparison group (see text for explanation). Source: Stanford Biodesign 2020 Innovation Fellowship Alumni Survey.

Conclusion

Designing, implementing, and sustaining assessment activities requires a significant investment of resources. Over the past 2 decades, we have refined our approach from an ad hoc, year-end review to a systematic tracking of career data of our alumni, resulting in major ­changes to our program. Next, we will continue to develop and validate instruments to better perform knowledge and skills assessment relevant to our specific training approach.

Our understanding of assessment in the areas of innovation and entrepreneurship training is still evolving. But we are encouraged that the evidence shows that this type of education does make a difference in the career choices, trajectories, and ­productivity of the participants.

References

Augustin DA, Yock CA, Wall J, Lucian L, Krummel T, Pietzsch JB, Azagury DA. 2020. Stanford’s Biodesign innovation program: Teaching opportunities for value-driven innovation in surgery. Surgery 167(3):535–39.

Brinton TJ, Kurihara CQ, Camarillo DB, Pietzsch JB, Gorodsky J, Zenios SA, Doshi R, Shen C, Kumar UN, Mairal A, and 6 others. 2013. Outcomes from a postgraduate biomedical technology innovation training program: The first 12 years of Stanford Biodesign. Annual Review of Biomedical Engineering 41(9):1803–10.

Kelley D, Kelley T. 2013. Creative Confidence: Unleashing the Creative Potential within Us All. New York: Crown Business.

Lyons R, Lynn T, Mac an Bhaird C. 2015. Individual level assessment in entrepreneurship education: An investigation of theories and techniques. Journal of Entrepreneurship Education 18(1):136–56.

Nimgaonkar A, Yock PG, Brinton TJ, Krummel T, Pasricha PJ. 2013. Gastroenterology and biodesign: Contributing to the future of our specialty. Gastroenterology 144(2):258–62.

Purzer S, Fila N, Nataraja JP. 2016. Evaluation of current assessment methods in engineering entrepreneurship education. Advances in Engineering Education 5(1).

Schwartz JG, Kumar UN, Azagury DE, Brinton TJ, Yock PG. 2016. Needs-based innovation in cardiovascular medicine: The Stanford Biodesign process. JACC Basic to Translational Science 1(6):541–47.

Wall J, Hellman E, Denend L, Rait D, Venook R, Lucian L, Azagury D, Yock PG, Brinton TJ. 2017. The impact of postgraduate health technology innovation training: Outcomes of the Stanford Biodesign fellowship. Annual Review of Biomedical Engineering 45(5):1163–71.

Yock PG, Zenios S, Makower J, Brinton TJ, Kumar UN, ­Watkins FT, Denend L. 2015. Biodesign: The Process of Innovating Medical Technologies, 2nd ed. Cambridge; New York: Cambridge University Press.

 


[1]  Stanford Biodesign became the Byers Center for Biodesign in mid-2016.

[2]  Over time the program evolved from roots in cardiovascular medicine to all disciplines of health care (Augustin et al. 2020; Nimgaonkar et al. 2013; Schwartz et al. 2016).

[3]  In 2007 we launched a joint fellowship program with India’s department of biotechnology and the All India Institute for Medical Sciences in Delhi, followed in 2010 by a joint fellowship with Singapore’s Economic Development Board and Agency for Science, Technology and Research (A*STAR).

[4]  The Stanford Institutional Review Board determined that a survey for program improvement did not meet the definition of human subjects research.

[5]  The names of finalists who interviewed for the fellowship but were not selected were not available for the academic years ending in 2002, 2003, 2004, or 2009.

[6]  ​​In this context, we define a company as an entity that has raised at least $250,000 in external grant and/or private capital funding, licensed intellectual property from Stanford or other relevant owner (if applicable), and completed incorporation.

About the Author:Lyn Denend is director for academic programs at the Stanford Byers Center for Biodesign and a lecturer in the Stanford School of Medicine. Paul Yock (NAE) is the founder of Stanford Biodesign, the Martha Meier Weiland Professor of Medicine, and founding cochair of and professor in Stanford’s Department of Bioengineering. Josh Makower (NAE) is cofounder and director of Stanford Biodesign. Dan Azagury is director for education and James Wall is director for program development for the Biodesign innovation fellowship program; both are also associate professors of surgery with Stanford Medicine.