Raphael Thys
December 16, 2021 – Palo Alto Research Center Incorporated (PARC), a Xerox company that provides custom R&D services, technology, expertise, best practices and intellectual property to global companies, has today announced that it has been awarded a USD $5.8 million contract by the Defense Advanced Research Projects Agency (DARPA).
The contract will see PARC work with the University of California at Santa Barbara, the University of Rostock and Patched Reality—an augmented reality (AR) company that focuses on product strategy, UX and UI design, and technical development—to develop an artificial intelligence (AI) system that will guide users through complex physical tasks. According to PARC, the system will convert text and video-based manuals to a format that can be processed by a computer and guide users to complete tasks using AR guidance while also monitoring task execution.
PARC will be the lead contractor on the project, known as ‘Autonomous Multimodal Ingestion for Goal-Oriented Support’ (AMIGOS) for the Perceptually-enabled Task Guidance Program. The project will introduce new methods to extract procedural knowledge from content, perceive the environment, reason about physical tasks, and enable conversational and AR guidance that is personalized to an individual user’s level of expertise. PARC stated that the goal is to enable mechanics, medics and other specialists to perform tasks within and beyond their skillsets by providing just-in-time feedback and instructions for physical tasks.
“Augmented reality, computer vision, language processing, dialogue processing and reasoning are all AI technologies that have disrupted a variety of industries individually but never in such a coordinated and synergistic fashion,” said Dr. Charles Ortiz, the principal investigator for AMIGOS. “By leveraging existing instructional materials to create new AR guidance, the AMIGOS project stands to accelerate this movement, making real-time task guidance and feedback available on-demand.”
An example of PARC’s on-site AI and AR problem solving solutions.
The project will deliver two major systems to DARPA. The first will be an offline component that can learn from multiple modalities, such as language and vision, to extract the steps from unlabeled content required to complete a task, including text instructions from manuals, illustrations and instructional videos. The second will be an online component that uses a hybrid AI approach combining both symbolic and neural AI elements to create interactive AR guidance based on the information extracted by the offline component. The guidance delivered by the system will be personalized to the individual user’s abilities and sensitive to the user’s emotional state during performance.
Xerox and PARC are continuing to make investments in AR-enabled field service. Earlier this year, Xerox announced the acquisition of CareAR and subsequent formation of CareAR Holdings (a consolidation of CareAR, DocuShare and XMPie), an augmented reality support platform that provides real-time access to expertise for customers, employees, and field workers. PARC added that the expertise gained from the AMIGOS project will help inform future product development for CareAR.
For more information on PARC and Xerox’ augmented reality solutions for field services, please click here.
Image credit: Xerox