With its foundations as a spin-off from the German Research Centre for Artificial Intelligence (DFKI) semvox has ever since been pioneering innovative solutions. We have a close partner network gathering universities, research centres and laboratories around the globe. Our aim: facilitating life and work by supporting people with our smart speech interaction solutions. Day by day.

Current research projects

DIGITAL SCRUBS – AI-System to Reduce the Cognitive Load in Operating Rooms

The variety of tasks, associated tensions, and sheer amount of information that medical staff face in the OR is enormous, and these factors significantly affect the performance of OR teams. Modern AI approaches can provide an answer to the challenge. The DIGITAL SCRUBS project was founded to reduce the fatigue and cognitive load of OR teams and significantly improve the quality of medical interventions. The interdisciplinary consortium of semvox GmbH, abat+ GmbH, ZeMA GmbH and nexus Institut für Kooperationsmanagement und interdisziplinäre Forschung GmbH, aims to research, develop, and evaluate human-technology interaction in operating rooms with a neuroergonomic attention assistance system.

Based on semvox's geni:OS platform, DIGITAL SCRUBS will be permanently installed in a dedicated, networked operating room as a technology demonstrator with a neuroergonomic evaluation platform. Based on 2 OP use cases from the vascular and neurosurgery departments, the research hypothesis posits that this attention assistance system can reduce cognitive load/fatigue, which can be confirmed by objective, psychophysiological, and psychometric measurement data from the OR team. This result is expected to be accompanied by a reduction of OR errors/duration and an improvement of OR quality.

Here, this personalized, interactive AI assistance system leverages the augmented reality of three sensory modalities (vision, hearing, and touch) for all members of the surgical team. Through a generic interface, it provides the user with a familiar human-machine-interface for a variety of networked OR systems. Depending on the task/activity at hand, as well as the individual cognitive and emotional state, relevant information for optimizing the surgical procedure is provided autonomously and context-sensitively via systems worn on the body for the multimodal and cross modal transfer of information.

The AI assistance system itself can be interacted with directly via speech and gesture control with haptic feedback, in addition to indirect control via medical technology systems in the operating room. The introduction of interpersonal AI assistance will lead to a change in dynamics within the surgical team. In order to take this aspect into account or already optimize it during development, DIGITAL SCRUBS is accompanied by participatory social science technology research.

In the consortium, semvox’s research department will apply competencies in the areas of dialog management and affective interaction, using rule based and statistically based methods established in its own geni:OS interaction platform and the speech technologies (ASR, TTS) from semvox’s partner Cerence. The dialogic processing logic for affective interaction in operating rooms will be realized by using the geni:OS platform, resulting in this specialized version as geni:OS for Operation Rooms (geni:OS4OP). semvox will also contribute its expertise in the area of intention recognition for MMI systems and both user and situation adaptive dialog systems.

Further Information:


The goal of the KARLI project is to develop an adaptive, responsive and level-compliant interaction in the vehicles of the future. To this end, customer-relevant AI functions are being developed in KARLI that detect driver states and design interactions for different automation levels.

These AI functions are developed in KARLI from empirical and synthetically generated data. The data will be collected and used in KARLI in such a way that the project results are scalable to Big Data from production vehicles available in the future.

In the KARLI project, the research department of semvox will apply competencies in the area of dialog management and affective interaction with the use of rule- and statistically-based methods on the basis of its own geni:OS interaction platform and speech technologies (ASR,TTS).

Go to KARLI-Website


The joint project "MITHOS" (Mixed-Reality Interactive Training for Dealing with Heterogeneity and Conflict Situations in Schools) researches and develops a mixed reality system for the acquisition of socio-emotional competencies. The training is based on a novel combination of VR and sensor technologies and methods for simulating interactive virtual agents. It aims to equip school teaching staff to deal with challenging social situations.

In an exemplary scenario, a teacher in a virtual environment faces students who are simulated by the system and represented as virtual agents. They may have different linguistic and cultural backgrounds, or special needs in the areas of language and/or emotional development. The teacher practices dealing with different (simulated) challenges, such as aggression towards classmates, withdrawal from the classroom, rejection of the work assignment, or attention seeking. The system evaluates the teacher's conflict management based on language and other signals and is able to provide guidance on how to handle the situation via a virtual agent (tutor).

Technically, the demonstrator should combine several communicative modalities: The virtual characters (student and tutor) show non-verbal behavior (gestures, facial expressions) and speak to the teacher, who in turn can express himself linguistically and whose physical behavior is tracked with suitable sensor technology. The usable language should be as free as possible - without using rigid grammar.

Go to MITHOS-Website

Past research projects

EmmA - the emotional mobile assistant

Psychic disorders are the main reason for employees to drop out from work. That's why the German law obligates companies to introduce a risk assessment for the mental health of employees and advance reactive measures in case of mental illness.

Our research project EmmA (Emotional mobile Assistant) aims at the development of a mobile assistance system for smartphones which will be able to assess the merit of employees' mental disorder and consult them individually about preventive or reactive actions.

The core of EmmA is a multimodal real-time sensor analysis. With the help of smartphones, pyhsical and social signals will be evaluated and in a further step, a socio-emotional behavior model, connected to a virtual avatar, will be developed. By this step EmmA gets a face to the world, is able to analyze the situation of people and provide individual help.

In the project, semvox is responsible for the generation of voice dialogs, the investigation of emotional states as well as the examination of a proper adaptive behavior. Together with our partners Charamel GmbH, WIPS GmbH, Deutsches Forschungszentrum für Künstliche Intelligenz and Augsburg University we support mental health prevention by the development of a digital assistant. Without being in need of expertised personnel, the assistant can be applied for the assessment of an employee's mental health and the presentation of measures against psychic problems.

Visit the project website

ASARob - Medical technologies


Do you know Paul, the assistance robot? He is able to make out if there is a human being calling for his attention.

The ASARob (Attention Sensitive Assistance Robot) project was set up as a cooperation between Bremen university, Fraunhofer Institute, Mojin Robotics, Unity Robotics and semvox. Aiming at developing an assistant robot for the use in domestic care, the Care-O-Bot® is not only featured with kinematic abilities but also captures and directs user attention. By interpreting behavior, actions and language context, Paul enables a natural human-machine-interaction.

Using its semvox ODP S3 platform,  semvox assists in language processing and the development of new dialog interactions between man and machine. In the realization process, learning modes from Artificial Intelligence are applied. The ASARob is equipped with adaptive behavior which underlies continuous enhancement in the course of the project.

More about the ASARob project: ASARob

SynDiQuAss – Industry 4.0

Initiated by the German Federal Ministry of Education and Research semvox is project partner for the SynDiQuAss project, which focuses on the digital enhancement of the assembling process, a better quality management and health-related improvements at industry workplaces.
The SynDiQuAss project is basically meant to serve small and mid-scale companies taking a standardized and affordable measure in the automization of assembling processes. Due to the highly competitive nature of the market many industrial companies are not able to invest big amounts in the transition to a more flexible and automatic production. SynDiQuAss will help these companies make their processes more efficient and install safer workplaces by offering standardized best-practice reference models.
Together with industry and research partners semvox is developing a multimodal production assistance system which enables workers to generate context-based user profiles and customized supporting services tailored to their own needs. The worker will be flexible in the choice of commands, being able to switch from voice-operating to gesturing or using touch commands. Further, the system will show how to assemble the parts and offer help when needed. This enables a hands-free and less defective assembling.

PDExergames – Medical Technology

PDExergames is a project that addresses the treatment of physical and cognitive restraints caused by one of the most widespread diseases, the Parkinson's Disease (PD). The symptoms that come along with the disease are often cured using pharmacological concepts.

PDExergames are meant to offer an alternative and playful therapy, making use of voice and gesture-based interaction concepts as well as sensory controllers and concepts for data analyzation. With the help of these games, the mental and motional activity of the persons concerned shall be sustained.

With semvox ODP S3, the long standing know-how of semvox in speech-based dialog interactions contributes to the multimodal support of PD patients. The main goal of the project lies in gaining insights into voice-based solutions in the Health IT sector.

more info about the project: PDExergames


Your contact in research

For further questions regarding research, please get in contact with:

Massimo Romanelli

Head of Research


Alvaro Cepero

Head of Backend


Arnaud Fadja

Head of Machine Learning


Timo Sowa

Dialog Team