In This Issue
The Bridge: 50th Anniversary Issue
January 7, 2021 Volume 50 Issue S
This special issue celebrates the 50th year of publication of the NAE’s flagship quarterly with 50 essays looking forward to the next 50 years of innovation in engineering. How will engineering contribute in areas as diverse as space travel, fashion, lasers, solar energy, peace, vaccine development, and equity? The diverse authors and topics give readers much to think about! We are posting selected articles each week to give readers time to savor the array of thoughtful and thought-provoking essays in this very special issue. Check the website every Monday!

Inclusive Human-Centered Machine Intelligence

Monday, March 1, 2021

Author: Shrikanth S.Narayanan and Asad M. Madni

As engineering strives to better people’s lives, human-centered technologies—enabled by converging engineering advances in sensing, computing, machine learning, data communication—will draw on machine intelligence[1] (MI) to help understand, support, and enhance the human experience. The challenge is to create technologies that work for everyone while enabling tools that can illuminate the source of variability or difference of interest.

Consider, for example, speech and language technologies for children to use conversational MI systems. Automatically recognizing and understanding child speech is far more challenging than adult speech because of variability and differences due to developmental changes along physical, cognitive, and socioemotional dimensions (Narayanan and Potamianos 2002). Health conditions that impair communication ability (e.g., autism spectrum disorder) present further challenges.

The ability to measure, analyze, interpret, and act on children’s speech data can enable broad inclusive access for them. It can also offer valuable tools and insights to scientists in discovery and clinicians in individualized treatment planning (Bone et al. 2016, 2017).

In addition, MI can create technologies for diversity and inclusion awareness by, for example, shining light on representations along dimensions of gender, race, age, appearance, and ability in entertainment and advertisement media (Guha et al. 2015; Martinez et al. 2019;  Ramakrishna et al. 2017).

An Engineering Lens into the Human Condition

The dynamics of the human state, behavior, and actions result from complex brain-body mechanisms and interactions with the world and are influenced by both individual and contextual variability (figure 1). And many aspects of human physical and psychological traits, states, and behaviors are not directly observable or accessible.

Narayanan figure 1.gif

Human behavioral signals available aurally and visually in speech, body language, and movement offer a window into decoding not just what people are doing but how they are thinking and feeling, their intent and emotions. People infer these using their sensory perception and judgment, often relying on observations of verbal and nonverbal behavioral expressions and appearance. There is tremendous variability across people and contexts, in both human expression and human processing of behavior cues (which can be subjective and idiosyncratic).

MI approaches can help analyze human trait (e.g., age), state (e.g., emotion), and behavior (e.g., speech) dynamics (Narayanan and Georgiou 2013) to enable contextually rich sensing, create computational measures, and design models to understand complex mechanisms (e.g., the human ability to be resilient) and to predict behavior change, all while tackling challenges ranging from noisy measurements to uncertainty in human-centered representations. At a simple level, this could entail determining who is talking to whom about what and how, using automated audio and video analysis of verbal and nonverbal behavior.

MI can also be used to recognize and assess higher-level states like emotions. Behavioral, physiological (e.g., heart rate, respiration, skin conductance), and environmental signals (e.g., location, soundscape, light, temperature, air quality) together offer possibilities for understanding dynamic cognitive, affective, and physical human states in context. Furthermore, MI could help detect and analyze deviation from what is deemed typical.

These MI techniques can in turn facilitate or enhance decision making by humans—and by autonomous -systems—in the context of a given application. In mental health, for example, MI can contribute to novel screening, diagnostics, and treatment support including just-in-time implementation and response monitoring (Bone et al. 2017).

Engineering Challenges in Enabling Inclusive Human-Centered Machine Intelligence

Several factors affect the ability of an engineered MI system to serve the needs of all users. The dimensions of individual variability—physical, cognitive, affective, and social—are multifaceted and interconnected. The right data and an engineering approach that adequately accounts for this variability are paramount.

Research to ensure algorithmic fairness in machine intelligence has gained much ground recently (Chouldechova and Roth 2020). Engineering models should perform at the same level regardless of variability not central to the intended task or experience, through either appropriate coverage of the data that informs the design or, more critically, the algorithmic ability to adapt or compensate for sources of variability.

For example, systems that are inclusive of children should consider behavioral variability due to individual differences in age, gender, sociocognitive level, and language ability. Additional contextual factors that need to be accounted for include the nature of interactions—how many people are involved, who they are (e.g., peers, parents, teachers), how free or structured the interaction is—as well as the interaction environment (indoor/outdoor, home/school/clinic) and data capture constraints (wearable or environmental sensors, frequency and quality of sensing).

Opportunities for Creating and Using Inclusive Machine Intelligence

Some of the UN Sustainable Development Goals[2]—good health and wellbeing, quality education, gender equality, reduced inequalities—indicate opportunities where inclusive machine intelligence can, and should, contribute. The following examples illustrate the breadth of needs and possibilities as well as applications addressing the NAE’s Grand Challenges for Engineering.[3]

Personalized Learning

Child-centered MI can help engineering systems personalize learning. Children make up over a quarter of the world’s population, but their physical, cognitive, emotional, and social developmental differences and changes challenge the design of inclusive MI technologies (e.g., enabling broad interactivity through -automatic speech, language, and vision technologies).

But an engineering system that attempts to personalize learning needs to also understand sources of individual differences, such as cognitive (confusion) and emo-tional (frustration) states. Additional factors such as health state and neurocognitive differences (e.g., attention deficit) may add further challenges.

Health Informatics

MI engineering can help advance informatics for behavioral and mental health. With over 10 percent of the world’s population affected by mental health challenges (Ritchie and Roser 2018), and with clinical research and practice heavily dependent on (relatively scarce) human expertise in diagnosing, managing, and treating conditions, opportunities for engineering to offer access at scale and tools to support care are immense.

For example, to determine whether a child is on the autism spectrum, a clinician would engage and observe the child in a series of interactive activities targeting relevant cognitive, communicative, and socio-emotional aspects, observe the resulting behavior cues, and codify specific patterns of interest (e.g., vocal intonation, facial expressions, joint attention behavior) (Bone et al. 2017; Guha et al. 2016). MI advances in both processing speech, language, and visual data and combining them with other clinical data enable novel and objective ways of supporting and scaling up these diagnostics.

The right data and an engineering
approach that adequately accounts
for individual variability are paramount.

Likewise, tracking psychotherapy can enhance understanding of the quality of care and of causal factors of outcomes. Engineering systems can automate analysis of a psychotherapy session through computing quality assurance measures that, for example, rate a therapist’s expressed empathy (Xiao et al. 2015). And technology can go beyond clinics to patients in their natural settings. For example, remote sensing of biobehavioral cues and secure computing can enable new ways to screen and track response to treatment and offer just-in-time support (Arevian et al. 2020).

Looking Forward

Inclusive machine intelligence holds great promise for many human-centered societal realms. But there are many challenges. Research and development are needed to

  • enable the collection of relevant data;
  • design algorithms that handle human aspects such as data variability, heterogeneity, and uncertainty;
  • specify and derive targets for modeling that reflects diverse human perspectives and subjectivity and affords interpretability; and
  • create an ecosystem of trusted partnerships between the designers and users of engineered MI systems, including addressing data provenance, integrity, and, not least, the ethical use of technology.


Arevian A, Bone D, Malandrakis M, Martinez V, Wells K, Miklowitz D, Narayanan S. 2020. Clinical state tracking in serious mental illness through computational analysis of speech. PLoS One 15(1):e0225695.

Bone D, Bishop S, Black M, Goodwin M, Lord C, Narayanan S. 2016. Use of machine learning to improve autism screening and diagnostic instruments: Effectiveness, efficiency, and multi-instrument fusion. Journal of Child Psychology and Psychiatry 57(8):927–37.

Bone D, Lee CC, Chaspari T, Gibson J, Narayanan S. 2017. Signal processing and machine learning for mental health research and clinical applications [Perspectives]. IEEE -Signal Processing 34(5):196–95.

Chouldechova A, Roth A. 2020. A snapshot of the frontiers of fairness in machine learning. Communications of the ACM 63(5):82–89.

Guha T, Huang C, Kumar N, Zhu Y, Narayanan S. 2015. -Gender representation in cinematic content: A multimodal approach. Proceedings, ACM International Conf on Multimodal Interaction, Nov 9–13, Seattle.

Guha T, Yang Z, Grossman R, Narayanan S. 2016. A computational study of expressive facial dynamics in children with autism. IEEE Transactions on Affective Computing 9(1):14–20.

Martinez V, Somandepalli K, Singla K, Ramakrishna A, Uhls Y, Narayanan S. 2019. Violence rating prediction from movie scripts. Proceedings, AAAI Conf on Artificial Intelligence 33:671–78.

Narayanan S, Georgiou P. 2013. Behavioral signal processing: Deriving human behavioral informatics from speech and language. Proceedings of the IEEE 101(5):1203–33.

Narayanan S, Potamianos A. 2002. Creating conversational interfaces for children. IEEE Transactions on Speech and Audio Processing 10(2):65–78.

Ramakrishna A, Malandrakis N, Staruk E, Narayanan S. 2015. A quantitative analysis of gender differences in -movies using psycholinguistic normatives. Proceedings, Conf on Empirical Methods in Natural Language Processing, Sep 17–21, Lisbon.

Ritchie H, Roser M. 2018. Mental health. Our World in Data, online at

Xiao B, Imel Z, Georgiou P, Atkins D, Narayanan S. 2015. “Rate my therapist”: Automated detection of empathy in drug and alcohol counseling via speech and language processing. PLoS One 10(12):e0143055.


[1]  This overarching term encompasses artificial intelligence, machine learning, and other engineering capabilities and technologies (e.g., adaptive sensing, communication, interfaces) that enable intelligent engineering systems.

[2] -development-goals/


About the Author:Shrikanth (Shri) Narayanan is University Professor and Niki and Max Nikias Chair in Engineering at the University of Southern California. Asad Madni (NAE) is a distinguished adjunct professor and distinguished scientist of electrical and computer engineering at UCLA and faculty fellow, Institute of Transportation Studies.