NAE Regional Meeting at Amazon and the University of Washington: Human-Robot Interaction

On May 19, Amazon and the University of Washington co-hosted the 5th NAE Pacific Northwest Regional Meeting, this one on the topic of Human-Robot Interaction.

Originally planned for spring 2020 but postponed twice due to Covid, the meeting had nearly 150 attendees: roughly 60 students, 40 faculty members, 40 corporate engineers, and 10 NAE leaders and staff members – including 30 Members of NAE and 10 Frontiers of Engineering alums.

Sidd Srinivasa, dual-hatted as the Boeing Endowed Professor at the University of Washington’s Paul G. Allen School of Computer Science & Engineering and as Director of Robotics AI at Amazon, began the day with an overview of robotics, AI, and the fulfillment process at Amazon: how the pieces fit together to enable any of 100s of millions of items to make their way from a fulfillment center to your door in two days (as well as any of 10s of millions of items in only one day, and any of several million items within hours).

Amazon warehouse.jpgRussell Allgor (NAE), Chief Scientist of Amazon WW Operations, described the changes that have been required over time to meet ever-increasing customer expectations for speed, selection, and cost. Optimizing any two of the three is not that hard. Cost and selection, ignoring speed: one massive full-inventory fulfillment center in the middle of the country. Speed and selection, ignoring cost: a large number of large full-inventory fulfillment centers scattered across the land. Speed and cost, ignoring selection: a large number of small limited-inventory fulfillment centers, similarly scattered. Simultaneously optimizing all three, though, is a challenge. Amazon’s fulfillment network has evolved to consist of several hundred inbound cross-dock and fulfillment centers, middle mile sort hubs and trailers, airplanes and air hubs, and last mile delivery stations and vehicles.

The introduction of robotics has been accompanied by major changes in the design and operation of fulfillment centers. Previously, inventory was stored in library shelving or bin shelving, which pickers could access on one face from aisles. The introduction of Amazon Robotics autonomous mobile drive units (AR Drives) enabled inventory to be stored in moveable pods, with any of the 4 faces of the pod accessible at pick and stow stations on the periphery of the fulfillment center. Importantly, the sides of the pods need not be accessible when the pods are “parked” – aisles are not necessary, which dramatically increases the capacity of the fulfillment center. (Think of a valet carpark.)

Nia Jetter, Senior Principal Technologist in Amazon Robotics AI, focused on the safety aspects of autonomous mobile robots in fulfillment centers. The fleets of AR Drives operate within theAutonomous vehicles.jpg confines of enormous “safety cages” where humans venture only rarely, and then only with multiple levels of protective systems. Next-generation mobile robots will share space with humans.

Mike Wolf, Principal Applied Scientist in Amazon Robotics AI, addressed the challenges of integrating robotics at enormous scale, designing processes where humans and robots each do what they do best. He discussed multi-robot task assignment and motion planning (fleets of AR Drives moving inventory pods to pick and stow stations), and manipulation challenges in robotic picking (100s of millions of unique items, no 3D models, non-rigid items).

In a bit of a deviation from the day’s theme – no humans involved – Byron Boots, dual-hatted as a professor in UW’s Allen School and as Principal Research Scientist at the NVIDIA Seattle Robotics Research Lab, described recent progress in high-speed off-road autonomous vehicles under the DARPA RACER (Robotic Autonomy in Complex Environments with Resiliency) program. The off-road environment – unprepared terrain, no maps, no GPS, sensor perturbations – is dramatically more challenging than urban or highway environments, and requires rethinking perception, planning and control.

And in an orthogonal deviation – no physical robots involved – Gérard Medioni, Vice President and Distinguished Scientist at Amazon, described Amazon’s “Just Walk Out” technology and the “Amazon One” approach to customer identification, both of which rely on remarkable recent progress in computer vision, which Gérard also described. A key message was the need to invent on behalf of the customer: customers identify the pain-point (e.g., “Waiting in line to check out is really annoying”), but not the solution.
Amazon One Customer ID.jpg

The day concluded with a panel on human-AI interaction – broader than human-robot interaction – with Maya Cakmak and Jamie Morgenstern from UW’s Allen School, Ryan Calo from the UW School of Law, and Meredith Morris, Director and Principal Scientist, People + AI Research at Google Research, moderated by Ed Lazowska (NAE) from the Allen School. The advances in AI are remarkable, with tremendous positive impacts, but they also create a number of challenges including the ethics of autonomous decisions, the impact on jobs and the economy, privacy, security, the safety and robustness of autonomous systems, and issues of fairness, bias, and transparency.


The meeting was organized by Ed Lazowska and Sidd Srinivasa. Russell Allgor assisted with the program, and Kelly Warren from Amazon expertly handled the logistics.

Reception Amazon Spheres.jpg