First-Time & Returning Users

Usability testing  & interviews with new and existing users.

Context

As we approached the third wave of COVID-19 and the holiday season near the end of 2020, our team wanted to assess the utility of our website in users' decision making processes and in their understanding of COVID-related metrics. 

Goals

Generative Goals

• Understand user workflow as they search for key pieces of information.

• Understand how the experience for first-time users might differ from returning users. Why might some users not return?

Design/Product Goals

• Identify pain points where users struggle to find or understand data points.

• Understand which tools and information users rely on for decision-making.

• Assess value of the homepage map as a means of navigation.

Methods

Recruitment

• Returning users were recruited via a random sample drawn from our newsletter subscriber list.

• First-time users were recruited through quota sampling of team members' personal networks, generating a range of races/ethnicities, ages, political ideologies, education statuses, and language proficiencies. These individuals had never been on our website before.

Interview

• Participants underwent a semi-structured interview in which I asked them about their sources for obtaining COVID-related information, their key COVID-related decisions, and their top COVID-related concerns. These questions were the same for both groups.

Usability Test: First-Time Users

• Free navigation: Show me what you'd do if you landed on this home page. Tell me what you think about what you see.

• Specific tasks:

  • Find the test positivity rate for the county in which you live.

  • Explain to me what infection rate means.

Usability Test: Returning Users

• Free navigation: Show me what you typically do when you come to this website. Tell me what you think about what you see.

• Specific tasks:

  • Find the test positivity rate for the county in which you live.

  • Explain to me what infection rate means.

Key Insights

The key value-adds we offer: the ability to compare metrics across different locations, a one-stop-shop for covid data and information, generally user-friendly design and language use, an easy-to-navigate map.

Both first-time and returning users rely on many pieces of information to make behavioral decisions related to COVID (e.g., whether to travel, whether to visit family, etc.), and each has their own process for coming to conclusions. 

Although users appreciate having so much to choose from on the website, they are overwhelmed by the amount of information on each page, and find it difficult to navigate. They do not like to have to scroll far to find what they are looking for.

Users like information that is specific to them, and do not like to have to complete many steps to find it.

COVID data is difficult to understand, and users are not apt to seek out information on other pages when they are confused. Despite our having metric definitions housed in our glossary, users did not click off the location page to find it unless explicitly asked to do so. Once they were prompted to read the glossary, users could accurately define COVID metrics. For first time users, the initial confusion made them feel as though they were not our intended audience, and some said they would click off the site entirely if they were on their own. 

Both first-time and returning users expressed interest in our methodology and data sources, and felt concerned that we did not explicitly state them in any obvious place.

Implementation