An Interactive App to Investigate Human Decision-Making
Project Overview
As part of my graduate research in the Fear Lab at the California Institute of Technology, I developed a browser-based application for running human decision-making experiments remotely during the COVID-19 pandemic.
Problem Statement
The browser application administered a set of surveys and a video game task to assess the impact of social influences and emotions on risky decision-making. The video game environment needed to maintain ecological validity while still delivering the user an experience sufficiently engaging for them to perform the task for up to an hour. The interface needed to be accessible and intuitive for users across many demographics, with an information architecture providing users with clear task instructions. These studies aid in understanding decision-making processes during cooperation and conflict, knowledge which can enable the design of resilient social systems and organizations.
Tools
My Role
I was the sole investigator, software developer, and designer on this project.
Research Methods
Usability Testing
A sample of external users and fellow lab members were asked to carry out a set of tasks as part of a usability test. Participants were observed carrying out the tasks and various usability metrics such as success rate, error rate, and time per task were recorded.
User Tasks
- Navigate from the Prolific website to the web application.
- Complete the series of surveys administered in the web application.
- Play through one of each type of level in the video game task.
- Complete one instance of the metacognition task.
- Use the URL containing the completion code to navigate back to Prolific from the web application.
User Interviews
Following completion of all tasks, users participated in semi-structured interviews to assess their pain points and subjective satisfaction. I was particularly interested in gaining feedback about the information architecture of the app and the instructions for the video-game task.
Playtesting
Both external users and fellow lab members were asked to playtest the video game task within the app. The task requires users to progress through a finite sequence of repeated levels, so during the playtest, participants were asked to complete one full playthrough. Participants were observed during the test and then asked to provide both open-ended feedback about their experience and responses to survey questions designed to address the following goals:
Goals
- Discover, diagnose, and fix any bugs in the game's codebase.
- Ensure that the game's reward system is sufficiently motivating to keep players engaged for the duration of the task.
- Test new algorithms for human-like behavior of AI-controlled characters.
Generative Modeling of User Behavior
Using domain-specific knowledge from behavioral ecology and the neuroscience of decision-making, I constructed Bayesian statistical models of user behavior which incorporated features of the video-game task as variables. The goal of these models was to develop a quantitative and causal understanding of how different features of the virtual environment impacted user's behavior.
Results
The usability tests highlighted an opportunity to improve the accessibility of the app; for example, some icons in the app interface were only differentiated by color, which presented a problem for colorblind users. Later versions of the app ensured that all icons were differentiated by at least two types of information such as both color and shape. User interviews also provided important suggestions for improving the app's information architecture.
The playtests revealed a major bug in the app's codebase which could only be triggered by a specific sequence of user behavior. Since only a small minority of users exhibited this behavior and the cause of the bug could only be discovered by observing the users in action, it would have been difficult (if not impossible) to diagnose and fix the bug in the absence of playtests.
An essential aspect of the video-game task is that for the purpose of the decision-making, users need to believe that the AI-controlled characters are being controlled by fellow humans. To this end, the playtests were valuable in allowing me to iteratively design algorithms for the game's AI and test how human-like the AI's motions were perceived to be by users. The final algorithm provided a good match between computer and human behavior.
Modeling user behavior allowed me to understand how different parameters of the app, such as the presence or absence of a visual feature or the the movement speed of a game character, impacted a user's behavioral strategy. These models enabled predictions about how alterations to different features of the app would change a user's behavior, allowing me to make more informed design decisions about how app features could be adjusted to elicit more succcessful behavioral strategies from users.
Takeaways
When features of an application can be mapped to variables in generative models of user behavior, those models can be used to predict how changes in features lead to changes in user behavior, instructing further iterations of app design. Playtests are not just useful for debugging, but also for designing new algorithms.