The INIT Lab and Ruiz HCI Lab have a joint paper to appear at the upcoming ACM International Conference on Multimodal Interaction (ICMI), which will be held as a virtual event in October of 2020. The work behind this paper was led by former Ph.D. student Dr. Sarah Morrison-Smith, now a teaching and research fellow at Barnard College, who was advised by Dr. Jaime Ruiz. The paper presents a dataset of voice and gesture commands that we hope will enable deeper investigation into multimodal authentication for the future of smart home and smart environment technology. Here is the abstract:

The future of smart environments is likely to involve both passive and active interactions on the part of users. Depending on what sensors are available in the space, users may make use of multimodal interaction modalities such as hand gestures or voice commands. There is a shortage of robust yet controlled multimodal interaction datasets for smart environment applications. One application domain of interest based on current state-of-the-art is authentication for sensitive or private tasks, such as banking and email. We present a novel, large multimodal dataset for authentication interactions in both gesture and voice, collected from 106 volunteers who each performed 10 examples of each of a set of hand gesture and spoken voice commands chosen from prior literature (10,600 gesture samples and 13,780 voice samples). We present the data collection method, raw data and common features extracted, and a case study illustrating how this dataset could be useful to researchers. Our goal is to provide a benchmark  dataset for testing future multimodal authentication solutions, enabling comparison across approaches

Interested readers can find the camera-ready version of the paper here. The dataset will be available to download before the conference here.

We are sad not to be able to visit the beautiful city of Utrecht in the Netherlands to meet other multimodal interaction researchers in person when presenting this work, but given the times we are living in, we are just glad to see that conferences are able to make alternate arrangements for letting researchers share their work with each other. We are hopeful that we will see everyone in person again at ICMI 2021!