Inbox Analysis
Weekly tracking and trending inquiries from all emails received to our inboxes.
Year: 2020-2021
Role: Lead Researcher
Research Method: Inbox Analytics and Qualitative Analysis
Research Type: Perpetual, Exploratory
Company: Act Now Coalition (projects: Covid Act Now, Rewiring America)
Context
When I joined the Covid Act Now team, Customer Support was a full-time role consisting of treating the symptoms of many problems (e.g., user confusion, data issues) without addressing their root causes. The Customer Support Lead would respond to users and answer their questions, but no one was documenting consistent themes coming into the inboxes. Furthermore, users often were flagging data and product bugs, which were not being triaged.
Goals
Generative Goals
Keep a pulse on key user concerns as the COVID landscape evolves.
Utilize the inboxes as a repository of consistent user insights.
Design/Product Goals
Identify pain points where users struggle to understand information and data.
Identify product and data bugs.
Discover product needs and requests from our users.
Define new KPIs to track after new product launches.
Methods
Recruitment
Since the inbox is an at-will process, concerned users would reach out to us when it suited them. Understanding that this is not a representative sample of our users, we used inbox insights directionally only.
Analysis
Each week, I logged every email received to our three inboxes. I coded each email on various characteristics, including User Type, Email Category, and Product with Issue.
Once all emails were coded, I ran counting functions and proportions for each coded category to see trends in email and user types. I pulled trends within each category by hand (e.g., trends within the product request emails).
Reporting
Each week I would break down the findings from the inbox analysis in a detailed report document that would break down trends in email types and highlight impactful outreach (e.g., partnership requests, notable feedback).
We would receive anywhere from 75 to 150+ emails per week, on average, typically fluctuating alongside COVID waves.
Inbox report not included here, as it contains confidential information that is internal to Act Now Coalition only.
High Level Impact
By the numbers:
36
weekly reports created
3.5k
emails tracked from 2.6K users
862
data bugs flagged
384
product requests surfaced
182
product bugs flagged
1
customer support position eliminated
Over time, the proportion of emails received that were about data bugs decreased by almost 50%, suggesting that our data quality improved as a result of these reports.
Implementation
Here are a couple of examples of how insights from weekly inbox analysis led to new products and updates.
5th Color
At the beginning of the second wave of COVID-19 in the fall of 2020, many users expressed concern that the map was no longer useful-- with nearly all states in the red/high risk category, there was no way for them to tell the difference between states that might be just over the "high" threshold, and states experiencing even more severe outbreaks.
We added another risk level and corresponding map color (dark red) to add more nuance to these differences.
Vaccine Eligibility Alerts
Right after vaccines became available to the general public, our inbox flooded with users confused about when they were eligible and where to sign up when it was their turn.
We developed an email alert system that allowed users to subscribe to states of interest, and we then sent them emails whenever their state entered a new tier of eligibility. To remove another barrier, we included a direct link to appointment booking websites so eligible parties could immediately sign up upon learning of their eligibility.