In my previous post, I mentioned that I have been able to observe the process of running user studies with younger children and have started assisting in person with my teammates, Alex Shaw and Ziyang Chen. After observing the process of running the user studies and assisting my teammates in the user studies more than ten times, I ran the user study twice as the secondary experimenter who reads some of the instructions to the children and helps guide them through the experiment process. Actually carrying out the user study was quite different compared to simply observing and taking notes. In this post, I will go into detail about the lessons I have learned from running the user studies.

The following are my take-away messages derived from the notes I took during my observations and the experience of running user studies with young children.

Prior to the user study, we follow the INIT Lab study protocol to ensure good experiment reliability and data quality. The guide includes the following items:

1. Confirm all documents are organized in the experiment binder, including the experiment script, and there are enough copies.
2. Confirm study equipment is fully charged and functional.
3. Confirm backup equipment, batteries, and/or chargers are ready for use if needed.
4. Pre-determine study order with participants’ random identifiers.
5. Confirm study area is properly set up for running the session.

In addition to the above check points, I find it very helpful to memorize the main steps of the study and prepare a personal checklist of to-do items. For the personal checklist, my suggestion is to break down the above general check points in a way that suits the experimenter. My personal checklist for the study is provided as an example:

Moreover, when preparing for the user study, I usually go through the tasks again to make sure that I am familiar with the process. During this process, saying the instructions out loud to myself instead of just reading them in my mind is helpful for noticing which part of the instructions on the screen is actually not easy for me to present smoothly. These preparations are tedious but necessary for me to feel confident when conducting the study.

During the user study, extra care is needed since I have seen that sometimes children do not express their feelings in a way that is easily interpretable, which might lead to inaccurate experiment results due to the children’s fatigue or frustration. For example, there were times when the children said that they don’t need a break, but showed signs of fatigue or frustration in the following task, such as placing their hands on their cheeks with a bored face or moving around in their seats. When fatigue or frustration comes into play, the children might rush through the task without paying attention, such as drawing the gestures carelessly or sorting the cards without careful thought in the card sort task from the NIH Toolbox®, which would affect the quality of our study data. Therefore, in addition to presenting the instructions clearly and smoothly, we also have to pay attention to the mood of the children by observing their facial expressions or changes in their behaviors and asking how they are feeling.

Also, based on my observation, children’s behaviors are relatively unpredictable compared with adults’ behaviors. In the studies that I have been a part of, I learned that some of the children can be very talkative while others are silent most of the time. Some were excited about the small prizes we showed them while others showed little interest in the prizes but liked to chat about their days before starting. There are still others who just need time to adjust to the new environment. Furthermore, I also noticed that children tend to forget the instructions more often than adults. In that case, the patience of the experimenter plays an important role in repeating the instruction while keeping a friendly atmosphere. The unpredictability in user studies with young children requires the experimenter to be patient, flexible, and responsive to keep the children motivated and engaged in the tasks.

I am a first-year Ph.D. student at University of Florida studying Computer Science. The challenging but rewarding experience of running user studies with children has motivated me to interact with children more. Recently I started volunteering in teaching children ages 5 to 11 basic Chinese on the weekends. Since children have always been a special population of users for new interaction technology, my goal is to improve my skills in running experiments with children by gaining more hands-on experience of interacting with this special population of users.