Supporting, rather than undermining, human attention and cognition is critical in high-stakes environments.
Details: ENKIx is the acronym for Enabling Knowledgeable Task Guidance In the Extremes. This DARPA-funded effort targeted augmenting human cognition by developing an assistive system powered by AI that can guide users through procedural tasks. The essential capability of the system to understand the task, when to provide guidance, and what guidance to provide is just one of the many research challenges we are studying on this project. “Extreme” in the title refers to the extremes of human task performance, including high hazard and risky operations under harsh and high-demand environments. The system is designed for augmented reality (AR) headset technology. Dr. Jaime Ruiz of the Ruiz HCI Lab is the lead investigator heading up a large team (see below).
Team Members
Faculty
Besides PI Jaime Ruiz, we collaborated on this project with many other faculty members:
Dr. James Fairbanks (UF CISE), Dr. David Kaber (UF ISE), Dr. Taskin Padir (Northeastern University), Dr. Maryam Zahabi (Texas A&M University), Dr. Evan Patterson (Topos Institute), Dr. Isaac Wang (James Madison University)
Current Students
project inactive
Former Students
Alexander Barquero
Rodrigo Calvo
Niriksha Regmi
Dinank Bista
Andrew Tompkins
The INIT Lab, in conjunction with the Ruiz HCI Lab, have been working on:
- Creating the AR app
- Designing the interface that the user will see, using a user-centered design approach
- Tracking the state of the world (where objects are in the AR)
- Voice and gesture input
Other teams at UF and associate performers are working on:
- Task representation: the state of the task and what steps are not yet completed
- User modeling: determining user’s cognitive and physical stress levels, and knowing their skill level
Publications
Refereed Conference Papers
- Alexander Barquero, Niriksha Regmi, Oluwatomisin Obajemu, Rohith Venkatakrishnan, Christina Boucher, Lisa Anthony, and Jaime Ruiz. 2025. How Hand Constraints Influence User Defined Gestures in Mixed Reality. In Graphics Interface 2025 (GI ’25). Association for Computing Machinery, New York, NY, USA, 10 pages. https://doi.org/10.1145/3769872.3769888
- Alexander Barquero, Rodrigo Luis Calvo, Daniel Alexander Delgado, Isaac Wang, Lisa Anthony, and Jaime Ruiz. 2024. Understanding User Needs for Task Guidance Systems Through the Lens of Cooking. In Proceedings of the 2024 ACM Designing Interactive Systems Conference (DIS ’24). Association for Computing Machinery, New York, NY, USA, 2006–2018. https://doi.org/10.1145/3643834.3661611
Check out some demo videos here.
Funding
This work is partially supported by the Defense Advanced Research Projects Association (DARPA) award #HR0011-22-2-0004. Any opinions, findings, and conclusions or recommendations expressed in this paper are those of the authors and do not necessarily reflect these agencies’ views.
News and Updates
- TIDESS project lessons learned presented at AVI 2022 workshop! #laterpub #latertweet #tidess
- INIT Lab undergrad presents her TIDESS work at UF’s Undergraduate Research Day! #latertweet #tidess
- Two INIT Lab undergraduates complete successful senior theses! #latertweet
- More on what we learned from conducting research during COVID-19, published in Obesity journal! #laterpub #latertweet
- INIT Lab undergrad alum Julia Woodward leads our most recent CHI publication! #laterpub #latertweet
- New collaborative project for the INIT Lab looking at the design of AR-based intelligent assistive agents! #latertweet
- Another TIDESS project paper appears in a science ed journal! #laterpub #latertweet #tidess
- TIDESS project paper on collaboration around multi-touch spherical displays published at CSCW 2021 (plus an Honorable Mention)! #laterpub #latertweet #tidess
- INIT Lab PhD student Nikita Soni successfully defends dissertation! #latertweet #tidess
- INIT Lab collaboration on smart, wearables-based COVID-19 mitigation techniques leads to journal article in Sensors! #laterpub #latertweet