$-Family of Gesture Recognizers

Quick recognition of touchscreen or pen gestures doesn’t require a lot of training — of you, or of the recognizer.

Details: We have been working on developing, evaluating and extending a suite of simple, fast, accurate pen and finger gesture recognizers, which we call the “$-family” (more below). These recognizers aim to provide simple, fast, and reasonably accurate gesture classification that can be easily incorporated into rapid prototyping processes. The $-family began with Wobbrock et al’s $1 algorithm, fast but one-stroke. Our $N extended $1 to multi-stroke gestures, supporting complex symbols like letters and numbers. $P and $Q both improved the efficiency of $N in recognizing multi-stroke gestures, as well as accuracy and robustness.

The INIT Lab is proud that the impact of this work has been recognized: $P received the Best Paper Award when it was published at ICMI’2012, and $Q was nominated for Best Paper at MobileHCI’2018.

Team Members

Faculty

Dr. Lisa Anthony

We also worked with Jacob Wobbrock at UW and Radu-Daniel Vatavu at University Stefan Cel Mare of Suceava.

Current Students

project closed

Former Students

Alex Shaw
Qingchuan Zhao “Bruce”

Read more about our toolkits for understanding how people make gestures here.

Publications

Refereed Conference Papers

Vatavu, R.-D., Anthony, L. and Wobbrock, J.O. 2018. $Q: A super-quick, articulation-invariant stroke-gesture recognizer for low-resource devices. Proceedings of the ACM Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI ’18). Barcelona, Spain (September 3-6, 2018). New York: ACM Press.
Vatavu, R., Anthony, L., and Wobbrock, J.O. 2012. Gestures as Point Clouds: A $P Recognizer for User Interface Prototypes. Proceedings of ACM International Conference on Multimodal Interaction (ICMI’2012), Santa Monica, CA, p.273-280.
Anthony, L. and Wobbrock, J.O. 2012. $N-Protractor: A Fast and Accurate Multistroke Recognizer. Proceedings of Graphics Interface (GI’2012), Toronto, Canada, p.117-120.
Anthony, L. and Wobbrock, J.O. 2010. A Lightweight Multistroke Recognizer for User Interface Prototypes. Proceedings of Graphics Interface (GI’2010), Ottawa, Canada, 02 Jun 2010, p.245-252.

More of the Story…

The $-family is a suite of loosely related geometric template-based recognition algorithms for pen and finger touchscreen gestures.

The $-family began with Wobbrock et al’s $1 unistroke gesture recognition algorithm. The main goal of $1 was to implement a simple, fast, reasonably accurate recognizer that could be easily incorporated into rapid prototyping processes, to give application designers the power of trying out different gesture design schemes without having to spend a lot of effort getting the recognition up and running.

$N extended $1 from recognizing only single stroke gestures — where the pen or finger does not lift during the gesture — to recognizing multi-stroke symbols — letters and numbers like X and +. Try out an online demo of $N in JavaScript, and check out the detailed pseudocode listing. $N is a great example of the connection between user and system: while it has some limitations, it performs well enough to support prototyping and simple development use cases without requiring machine learning expertise.

$P was the next member of the $-family, and it was faster and more robust to differences in how users actually make (articulate) gestures. $P can recognize gestures made with any number of strokes in any order or direction with low storage cost, whereas $N can have high storage costs due to how it internally represented all the potential variations (which $P’s matching algorithm can handle seamlessly). Using a point-cloud representation allowed for non-linear matching processes and this improved accuracy as well as efficiency. Try out an online demo of $P in JavaScript, and check out its detailed pseudocode listing. $P received the Best Paper Award when it was published at ICMI’2012.

The most recent incarnation of the $-family that the INIT Lab contributed to is $Q, a “super-quick” version of $P that uses a few straightforward algorithmic optimizations to reduce the calculation costs of the $P matching process. This means it is particularly suited to extremely low-resource devices, but it uses features like look-up tables that may be less accessible to novice programmers. In these cases, $P is sufficient. $Q was nominated for Best Paper when it was published at MobileHCI’2018. Try out an online demo of $Q in JavaScript, and check out its detailed pseudocode listing.

For help in choosing the best member of the $-family for your needs, check out the “cheat sheet” which we include in the $P paper. Also, we provide more detailed information, and C# reference implementations, on our full project websites for $N, $P, and $Q!

Funding

This work was partially supported by National Science Foundation Grant Awards #IIS-1218395 / IIS-1433228 and IIS-1218664. Any opinions, findings, and conclusions or recommendations expressed in this paper are those of the authors and do not necessarily reflect these agencies’ views.

[Ancient] News and Updates

back to home