The variety of tasks, associated tensions, and sheer amount of information that medical staff face in the OR is enormous, and these factors significantly affect the performance of OR teams. Modern AI approaches can provide an answer to the challenge. The DIGITAL SCRUBS project was founded to reduce the fatigue and cognitive load of OR teams and significantly improve the quality of medical interventions. The interdisciplinary consortium of semvox GmbH, abat+ GmbH, ZeMA GmbH and nexus Institut für Kooperationsmanagement und interdisziplinäre Forschung GmbH, aims to research, develop, and evaluate human-technology interaction in operating rooms with a neuroergonomic attention assistance system.

Based on semvox's geni:OS platform, DIGITAL SCRUBS will be permanently installed in a dedicated, networked operating room as a technology demonstrator with a neuroergonomic evaluation platform. Based on 2 OP use cases from the vascular and neurosurgery departments, the research hypothesis posits that this attention assistance system can reduce cognitive load/fatigue, which can be confirmed by objective, psychophysiological, and psychometric measurement data from the OR team. This result is expected to be accompanied by a reduction of OR errors/duration and an improvement of OR quality.

Here, this personalized, interactive AI assistance system leverages the augmented reality of three sensory modalities (vision, hearing, and touch) for all members of the surgical team. Through a generic interface, it provides the user with a familiar human-machine-interface for a variety of networked OR systems. Depending on the task/activity at hand, as well as the individual cognitive and emotional state, relevant information for optimizing the surgical procedure is provided autonomously and context-sensitively via systems worn on the body for the multimodal and cross modal transfer of information.

The AI assistance system itself can be interacted with directly via speech and gesture control with haptic feedback, in addition to indirect control via medical technology systems in the operating room. The introduction of interpersonal AI assistance will lead to a change in dynamics within the surgical team. In order to take this aspect into account or already optimize it during development, DIGITAL SCRUBS is accompanied by participatory social science technology research.

In the consortium, semvox’s research department will apply competencies in the areas of dialog management and affective interaction, using rule based and statistically based methods established in its own geni:OS interaction platform and the speech technologies (ASR, TTS) from semvox’s partner Cerence. The dialogic processing logic for affective interaction in operating rooms will be realized by using the geni:OS platform, resulting in this specialized version as geni:OS for Operation Rooms (geni:OS4OP). semvox will also contribute its expertise in the area of intention recognition for MMI systems and both user and situation adaptive dialog systems.

Further Information: https://www.interaktive-technologien.de/projekte/digital_scrubs

The goal of the KARLI project is to develop an adaptive, responsive and level-compliant interaction in the vehicles of the future. To this end, customer-relevant AI functions are being developed in KARLI that detect driver states and design interactions for different automation levels.

These AI functions are developed in KARLI from empirical and synthetically generated data. The data will be collected and used in KARLI in such a way that the project results are scalable to Big Data from production vehicles available in the future.

In the KARLI project, the research department of semvox will apply competencies in the area of dialog management and affective interaction with the use of rule- and statistically-based methods on the basis of its own geni:OS interaction platform and speech technologies (ASR,TTS).

Go to KARLI-Website

The joint project "MITHOS" (Mixed-Reality Interactive Training for Dealing with Heterogeneity and Conflict Situations in Schools) researches and develops a mixed reality system for the acquisition of socio-emotional competencies. The training is based on a novel combination of VR and sensor technologies and methods for simulating interactive virtual agents. It aims to equip school teaching staff to deal with challenging social situations.

In an exemplary scenario, a teacher in a virtual environment faces students who are simulated by the system and represented as virtual agents. They may have different linguistic and cultural backgrounds, or special needs in the areas of language and/or emotional development. The teacher practices dealing with different (simulated) challenges, such as aggression towards classmates, withdrawal from the classroom, rejection of the work assignment, or attention seeking. The system evaluates the teacher's conflict management based on language and other signals and is able to provide guidance on how to handle the situation via a virtual agent (tutor).

Technically, the demonstrator should combine several communicative modalities: The virtual characters (student and tutor) show non-verbal behavior (gestures, facial expressions) and speak to the teacher, who in turn can express himself linguistically and whose physical behavior is tracked with suitable sensor technology. The usable language should be as free as possible - without using rigid grammar.

Go to MITHOS-Website

Do you know Paul, the assistance robot? He is able to make out if there is a human being calling for his attention.

The ASARob (Attention Sensitive Assistance Robot) project was set up as a cooperation between Bremen university, Fraunhofer Institute, Mojin Robotics, Unity Robotics and semvox. Aiming at developing an assistant robot for the use in domestic care, the Care-O-Bot® is not only featured with kinematic abilities but also captures and directs user attention. By interpreting behavior, actions and language context, Paul enables a natural human-machine-interaction.

Using its semvox ODP S3 platform,  semvox assists in language processing and the development of new dialog interactions between man and machine. In the realization process, learning modes from Artificial Intelligence are applied. The ASARob is equipped with adaptive behavior which underlies continuous enhancement in the course of the project.

More about the ASARob project: ASARob