This summer, I have been working on a project related to the $-family of gesture recognizers. The $-family is a series of simple, fast, and accurate gesture recognizers designed to be accessible to novice programmers. $1 [1] was created by Wobbrock and colleagues, and INIT Lab director Lisa Anthony contributed to later algorithms, including $N [2] and $P [3]. My goal this summer was to implement my own versions of the $-family algorithms, and then to try them out on a new dataset that was collected from adults and children in a different context than previous datasets collected by the INIT lab.

The first step of my work on this project was to understand how the different algorithms of the $-family work. I examined the advantages and limitations of each recognizer in the $-family by reading the related research papers and playing around with existing implementations of the recognizers. After studying the recognizers, I created my own implementations of $1 and $P in Javascript by making a web application. I faced several challenges when implementing these algorithms. My first challenge was to decide in what form the gestures to be recognized should be taken as input (predefined point arrays or through a canvas where user-defined gestures can be given as input). Using this $1 implementation as a reference, I normalized each of the gestures, then computed the distance between the gestures and performed user-defined gesture recognition through a canvas. While implementing the algorithms, I followed a step by step approach so that I could evaluate whether each function was working before moving forward with recognition. In the process, I learned the importance of debugging the program to help pinpoint errors in my code more efficiently than trying to find the problems manually.

After completing the web applications, my next task was to recognize gestures from a dataset with XML files as input. I created another implementation of the $1 recognizer in Python to learn and explore another programming language. I was initially unsure how to read in the gesture data from XML files so I had to learn how to parse them. I used the pseudo code presented in the original $1 paper [1] as a guide to implement the algorithm. Resampling the points of the gesture before recognition was challenging. Every gesture needed to have the same number of resampled points for recognition. To solve the issues I encountered while preprocessing the gestures, I plotted the gestures using the matplotlib library from Python. Not only did visualising gestures help in that context, it also helped me to understand why some gestures were wrongly recognized, since they looked more like the other gestures than what they actually were. Solving these errors and getting a correct implementation gave me a great sense of achievement. After implementing the recognition algorithms, I learned how to run user-independent recognition experiments where I systematically varied the number of participants included in the training set. Then I ran those experiments to find out the accuracy of the algorithms that I implemented. Now, I am working on analyzing articulation features [4] [5] of a new set of gestures to help quantitatively investigate the difference between adult’s and children’s gestures in a new context.

I am a final year undergraduate computer science student from MIT, Pune, India working with the INIT lab as an REU student this summer, as part of the UF CISE IMHCI REU program. I have greatly enjoyed my time working in the INIT lab. One thing I have really enjoyed while I’ve been here is related to another project I worked on: interacting with an ocean temperature application on the PufferSphere, which is a large interactive spherical display. Through my experience in the INIT lab, I have been able to closely follow the different stages of the research process. I’ve added to my technical knowledge through improved understanding of gesture recognizers, and I’ve also learned the importance of being clear and concise in scientific writing. I am looking forward to continuing my work on this project and understanding new ways to improve children’s gesture interaction experiences.

References:

[1] Wobbrock, Jacob O., Andrew D. Wilson, and Yang Li. “Gestures without libraries, toolkits or training: a $1 recognizer for user interface prototypes.” Proceedings of the 20th annual ACM symposium on User interface software and technology. ACM, 2007.

[2] Anthony, Lisa, and Jacob O. Wobbrock. “A lightweight multistroke recognizer for user interface prototypes.” Proceedings of Graphics Interface 2010. Canadian Information Processing Society, 2010.

[3] Vatavu, Radu-Daniel, Lisa Anthony, and Jacob O. Wobbrock. “Gestures as point clouds: a $ P recognizer for user interface prototypes.” Proceedings of the 14th ACM international conference on Multimodal interaction. ACM, 2012.

[4] Anthony, Lisa, Radu-Daniel Vatavu, and Jacob O. Wobbrock. “Understanding the consistency of users’ pen and finger stroke gesture articulation.” Proceedings of Graphics Interface 2013. Canadian Information Processing Society, 2013.

[5] Shaw, Alex, and Lisa Anthony. “Analyzing the articulation features of children’s touchscreen gestures.” Proceedings of the 18th ACM International Conference on Multimodal Interaction. ACM, 2016.