Intelligent AI-Assistance in networked Operating Rooms
Digital Scrubs
Supporting Surgeons with their Attention
During a medical operation the risk of failure increases with fatigue and cognitive load, leading to inattention.
DIGITAL SCRUBS is an AI powered attention assistance system, that monitors the surgeon’s cognitive and emotional state. In addition to indirect control via medical technology systems in the operating room, the system itself allows direct interaction via speech and gesture control with haptic feedback.
Digital Scrubs
Multimodal Information Transfer
Depending on the specific task/activity and the individual cognitive and emotional state, relevant information for optimizing the surgical procedure is provided independently and context-sensitively via body-worn systems for multi- and cross-modal information transfer.
Digital Scrubs
Improve Surgical Quality
The research hypothesis is that in two surgical use cases from vascular and neurosurgery, a reduction in cognitive load/fatigue through the attention assistance system can be demonstrated using objective, psychophysiological and psychometric measurement data from the surgical team, and that this is associated with a reduction in surgical errors/duration or an improvement in surgical quality.
semvox in Digital Scrubs
In the DIGITAL SCRUBS project, a multidisciplinary consortium was researching, developing and evaluating for the first time an interpersonal and multimodal attention assistance system for context-sensitive, neuroergonomic human-machine interaction in networked operating rooms.
semvox leveraged its expertise in dialogue management and affective interaction, employing rule- and statistically based methods on its own geni:OS interaction platform and the speech technologies (ASR, TTS) of its partner Cerence.
The geni:OS platform enables the implementation of dialogic processing logic for affective interaction in the operating room, resulting in this specialized version known as geni:OS for Operation Room (geni:OS4OR).
semvox also contributed its expertise in intention recognition for MMI systems and user- and situation-adaptive dialogue systems.
The research and development of geni:OS4OR was carried out in close cooperation with partners abat+ and ZeMA (through further development of the basic geni:OS technology). In particular, the integration process in the demonstrator was implemented jointly with partner abat+, adhering to the highest IT security standards in terms of both software and hardware.
The AI system geni:OS4OR had to be integrated into the existing operating room infrastructure with varying degrees of digitization and heterogeneous interfaces via touchpoints (see AP 4). Furthermore, the DIGITAL SCRUBS in Clinical Mode (CM) (see Fig. 1, right) provides the necessary neurotechnology for the minimally intrusive assessment of the surgical team’s psychophysiological response, among other things as a basis for the empathic assistance provided by geni:OS4OR.
Digital Scrubs
Augmented Sensory Modalities
This personalized, interactive AI assistance system utilizes the augmented reality of three sensory modalities (sight, hearing, and touch) of all members of the surgical team. Through a generic interface, it provides the user with a familiar human-machine interface for a variety of networked surgical systems.