FIRST-TIME & RETURNING USERS
Usability testing, interviews, and analytics to understand differences in experiences between first-time and returning users of Covid Act Now.
Role: Lead Researcher
Company: Act Now Project (project: Covid Act Now)
Timeline: ~1 Month
Research Method: Usability Testing, Interviews, and Analytics
Research Type: Exploratory, Evaluative
CONTEXT
Covid Act Now built local data models and products to help governments, private organizations, and tens of millions of people make informed, science-based decisions on how to stay safe during the COVID pandemic. I led research efforts there, working closely with design, engineering, and data teams to guide questions and insight implementation.
As we approached the third wave of COVID-19 and the holiday season near the end of 2020, our team wanted to assess the utility of our website in users' decision making processes and in their understanding of COVID-related metrics.
Some key questions on our mind:
Are we accessible and useful to those populations most at risk of COVID (e.g., older populations, persons of color, etc.)?
The risk level map takes up all the above-the-fold space on our homepage-- do people rely on it enough for this space ot be justified?
GOALS
Generative Goals
Understand user workflow as they search for key pieces of information.
Understand how the experience for first-time users might differ from returning users: why do people use our site? Why don't they?
Design/Product Goals
Identify pain points where users struggle to find or understand data points.
Understand which tools and information users rely on for decision-making.
Assess value of the homepage map as a means of navigation.
METHODS
Recruitment
Returning users (N=7) were recruited via a random sample drawn from our newsletter subscriber list.
First-time users (N=9) were recruited through quota sampling of team members' personal networks, with emphasis on those who did not fit our typical user base at the time, specifically:
Users who may have lower education levels.
Users who are people of color and/or immigrants to the US.
Users who are less comfortable with technology (e.g., older populations).
Semi-structured Interview
Both returning and first-time users underwent a semi-structured interview in which I asked them about their sources for obtaining COVID-related information, their key COVID-related decisions (e.g., travel, mask-wearing, etc.), and their top COVID-related concerns.
Usability Test: First-Time Users
Free navigation:
Show me what you'd do if you landed on this home page. Tell me what you think about what you see.
Specific tasks:
Find the test positivity rate for the county in which you live.
Explain to me what infection rate means.
Usability Test: Returning Users
Free navigation:
Show me what you typically do when you come to this website. Tell me what you think about what you see.
Specific tasks:
Find the test positivity rate for the county in which you live.
Explain to me what infection rate means.
ANALYSIS
Data Preparation
To prepare my data for analysis, I used otter.ai software to transcribe the recordings of the sessions.
Coding
I used this transcription text along with the session videos themselves to thematically code the data based on certain actions and where they occured (homepage, location page, etc.), exploring specifically:
Tools and Data used to make decisions:
For how many locations are they referencing COVID-19 data? At what level (county, state)?
Which data points are they referencing (e.g., infection rate, test positivity rate, vulnerability level, risk level, etc.)?
Where are they accessing these data on our site (e.g., map, location pages, Compare Table, Trends Chart, API, etc.)?
Are they using any other dashboards/tools to supplement (e.g., New York Times, Johns Hopkins)? Why/why not?
Navigation and Discoverability:
Did they successfully find the infection rate for their county?
Did they navigate to their desired location page via the homepage map, via search, via a saved/bookmarked link, or another way?
Understanding of COVID Metrics:
Did they accuratly define "infection rate"?
If not, did they look for a metric definition somewhere on our site? Did those definitions clear up confusion?
Behavioral Data
I also incorporated behavioral data from thousands of users to get a fuller picture, using FullStory to evaluate common click journeys to analyze:
How many clicks, on average, were users making to land on a location page? What was the maximum?
How long was it taking a user, on average, to arrive at a locaiton page? What was the maximum?
What were users' most common points of entry to a location page?
KEY INSIGHTS AND HOW WE ACTED ON THEM
We learned:
For both returning and first-time users, COVID data are difficult to understand, and users are not apt to seek out clarifying information on other pages when they are confused. Most did not accurately define "infection rate".
Despite our having metric definitions housed in our glossary, users did not click off their location page to find it unless explicitly asked to do so. But once they were prompted to read the glossary, users could accurately define COVID metrics. Click data showed that our glossary was not commonly visited.
For first time users specifically, the initial confusion about what certain metrics mean made them feel as though they were not our intended audience, and some said they would click off the site entirely if they were on their own.
Both first-time and returning users expressed interest in our methodology and data sources, and felt concerned that we did not explicitly state them in any obvious place. This called into question if we were a trustworthy source.
Users expressed feeling satisfied by those data sources and methodology once they were shown the information.
In both of these cases, we had sufficient content to satisfy their needs, but the users didn't want to hunt for it.
We changed:
We added an “About this data” pop-up module that shares data sources, methodology, and definitions for each metric next to where the data point lives, providing key context and clarification right at the point of confusion and concern.
This tooltip received 50% more clicks than our glossary page.
We learned:
Although users appreciate having so much to choose from on the website, they are overwhelmed by the amount of information on each page, and find it difficult to navigate.
They do not like that they have to scroll far to find what they are looking for.
We also observed that when it comes to acting on our data, both first-time and returning users rely on many pieces of information to make behavioral decisions related to COVID (e.g., whether to travel, whether to visit family, etc.).
Each has their own process for coming to these conclusions and they vary widely.
Click data showed that, on average, it took users 4+ clicks and 30 seconds to land on a location's COVID-19 data page.
We changed:
We combined multiple timeseries charts into a single chart with tabs for each metric to reduce the scrolling users must do to connect with key information.
Since people have such different processes for making COVID-related decisions, we did not want to take away any of the tools they could use. We instead chose to drawer longer individual modules (see right) to further reduce scroll, making more items on the page discoverable while still giving users agency to explore the parts of the website they desire.
We added personalized, auto-loaded state and county location cards on the homepage based on IP addresses below the search bar, keeping relevant locations front-and-center and removing a step for finding information of interest.
This is also a 75% decrease in clicks required to see a location's full data page.
Other learnings for our user knolwedgebase:
Our key value-adds include:
The ability to compare COVID metrics across different locations.
A one-stop-shop for covid data and information.
Generally user-friendly design and language use, as compared to competitors.
An easy-to-navigate map. Click data validated that the map is the most common point-of-entry from the homepage to a location page.
Thus, we decided that dedicating a large portion of our home page to our interactive map and Compare table makes sense, so we did not make Homepage design changes.
