FIRST-TIME & RETURNING USERS
Usability testing & interviews with new and existing users.
Role: Lead Researcher
Company: Act Now Project (project: Covid Act Now)
Timeline: ~1 Month
Research Method: Usability Testing & Interviews
Research Type: Exploratory, Evaluative
CONTEXT
Covid Act Now built local data models and products to help governments, private organizations, and tens of millions of people make informed, science-based decisions on how to stay safe during the COVID pandemic.
As we approached the third wave of COVID-19 and the holiday season near the end of 2020, our team wanted to assess the utility of our website in users' decision making processes and in their understanding of COVID-related metrics.
Some key questions on our mind:
Are we accessible and useful to those populations most at risk of COVID (e.g., older populations, persons of color, etc.)?
The risk level map takes up all the above-the-fold space on our homepage-- do people rely on it enough for this space ot be justified?
GOALS
Generative Goals
Understand user workflow as they search for key pieces of information.
Understand how the experience for first-time users might differ from returning users: why do people use our site? Why don't they?
Design/Product Goals
Identify pain points where users struggle to find or understand data points.
Understand which tools and information users rely on for decision-making.
Assess value of the homepage map as a means of navigation.
METHODS
Recruitment
Returning users (N=7) were recruited via a random sample drawn from our newsletter subscriber list.
First-time users (N=9) were recruited through quota sampling of team members' personal networks, with emphasis on those who did not fit our typical user base at the time, specifically:
Users who may have lower education levels.
Users who are of minority status.
Users who are less comfortable with technology (e.g., older populations).
Semi-structured Interview
Both returning and first-time users underwent a semi-structured interview in which I asked them about their sources for obtaining COVID-related information, their key COVID-related decisions (e.g., travel, mask-wearing, etc.), and their top COVID-related concerns.
Usability Test: First-Time Users
Free navigation:
Show me what you'd do if you landed on this home page. Tell me what you think about what you see.
Specific tasks:
Find the test positivity rate for the county in which you live.
Explain to me what infection rate means.
Usability Test: Returning Users
Free navigation:
Show me what you typically do when you come to this website. Tell me what you think about what you see.
Specific tasks:
Find the test positivity rate for the county in which you live.
Explain to me what infection rate means.
ANALYSIS
Data Preparation
To prepare my data for analysis, I used otter.ai software to transcribe the recordings of the sessions.
Coding
I used this transcription text along with the videos themselves to thematically code the data based on certain actions and where they occured (homepage, location page, etc.), exploring specifically:
Tools and Data used to make decisions:
For how many locations are they referencing COVID-19 data? At what level (county, state)?
Which data points are they referencing (e.g., infection rate, infection rate, vulnerability level, risk level, etc.)?
Where are they accessing these data (e.g., map, location pages, Compare Table, Trends Chart, API, etc.)?
Are they using any other dashboards/tools to supplement? Why/why not?
Navigation and Discoverability:
Did they navigate to their desired location page via the homepage map, via search, via a saved/bookmarked link, or another way?
Did they successfully find the infection rate for their county?
Understanding of COVID Metrics:
Did they accuratly define "infection rate"?
If not, did they look for metric definitions on our site? Did those definitions clear up confusion?
KEY INSIGHTS AND HOW WE ACTED ON THEM
We learned:
For both returning and first-time users, COVID data are difficult to understand, and users are not apt to seek out clarifying information on other pages when they are confused.
Despite our having metric definitions housed in our glossary, users did not click off the location page to find it unless explicitly asked to do so, but once they were prompted to read the glossary, users could accurately define COVID metrics.
For first time users specifically, the initial confusion about certain metrics made them feel as though they were not our intended audience, and some said they would click off the site entirely if they were on their own.
Both first-time and returning users expressed interest in our methodology and data sources, and felt concerned that we did not explicitly state them in any obvious place.
Users expressed feeling satisfied by those data sources and methodology once they were shown the information.
In both cases, we had sufficient content to satisfy their needs, but the users didn't want to hunt for it.
We changed:
We added an “About this data” pop-up module that shares data sources, methodology, and definitions for each metric next to where the data point lives, providing key context and clarification right at the point of confusion and concern.
We learned:
Although users appreciate having so much to choose from on the website, they are overwhelmed by the amount of information on each page, and find it difficult to navigate.
They do not like that they have to scroll far to find what they are looking for.
We changed:
We combined multiple timeseries charts into a single chart with tabs for each metric to reduce the scrolling users must do to connect with key information.
We also drawered longer individual modules to further reduce scroll, giving users agency to explore the parts of the website they desire while making more items on the page discoverable.
We added personalized, auto-loaded state and county location cards on the homepage based on IP addresses below the search bar, keeping relevant locations front-and-center and removing a step for finding information of interest.
Other learnings for our user knolwedgebase:
Our key value-adds include:
The ability to compare metrics across different locations.
A one-stop-shop for covid data and information.
An easy-to-navigate map.
Generally user-friendly design and language use, as compared to competitors.
Thus, we decided that dedicating a large portion of our home page to our interactive map and Compare table makes sense, so we did not make Homepage design changes.
COVID Decision-making:
When it comes to acting on our data, both first-time and returning users rely on many pieces of information to make behavioral decisions related to COVID (e.g., whether to travel, whether to visit family, etc.).
Each has their own process for coming to these conclusions and they vary widely.
