Designing CallRail’s first self-serve onboarding to improve retention

CallRail 2023 | Drove Adoption & Engagement

Hebe Zheng (Lead Product Designer), Heather Lutz (Senior Content Strategist), Grant Sadowski (Senior Product Manager), and the Experience Engineering team.

Summary

Sep - Oct 2023

Context

In 2023, CallRail's churn jumped from 11.6% to 14%, a 20% increase. Improving new user activation became a company-wide priority. CallRail requires meaningful setup before users experience value, but users are unlikely to invest time without seeing that value first. This created a core onboarding gap.

Outcome

90-day retention improved from 86.4% to 88.6%, a 2.55% YoY improvement. Accounts that completed the checklist were 38% more likely to remain active. Unexpectedly, marketing agencies began using the checklist as a standard client setup process, extending its value beyond first-time users.

My Role

As Lead Product Designer, I designed CallRail’s first self-serve, in-app onboarding experience. I led the work end-to-end, from research and exploration through execution, and facilitated cross-functional workshops to define onboarding strategy and task prioritization.

Problem

A catch-22: users needed to configure their account to see value

CallRail’s value depends on setup, creating tracking numbers, installing tracking scripts, configuring call routing, and activating integrations. This created a catch-22: users needed to complete setup to see value, but without early signs of value, they lacked the motivation to complete setup. This results in slow activation, weak habit formation, and rising churn.

"Have someone walk you through the set up as it can be a little confusing at first. But once you get the hang of it’s pretty easy." - Marketing agency, Feb 2023

Research

To understand where users were getting stuck, I synthesized insights from FullStory, Looker, prior Growth research, and conversations with Sales and Support.

Users signed up with clear intent (call tracking, attribution), but lost direction once inside the product
CallRail's ability to integrate with users' existing tools is a key consideration for potential customers.
Experienced agency users follow the same setup process across all client accounts.

Expand the existing first-run experience: front load the essential setups

Exploration & Tradeoffs

Early concept cross-functional collaboration

Approach 1: Expand the First-Run Wizard

I explored front-loading critical setup steps into the initial onboarding flow to take advantage of users’ motivation at signup.

Installing the tracking code snippet (essential for CallRail's functionality)
Setting up call routing and activating integrations (both strongly correlated with retention and customer value)

Why we didn’t pursue it:

Higher risk of overwhelming new users too early
Trust concerns: users may hesitate to install tracking code or route calls to CallRail right after signup
Stakeholders prioritized getting users into the product quickly

Expand the existing first-run experience: front load the essential setups

Approach 2: Customized onboarding based on user’s intended use

We explored a customized onboarding approach that adapted to each user’s intended use. The idea came from our Sales team. When Sales onboards new customers, they start by asking about goals and how the customer plans to use CallRail, then tailor setup guidance accordingly. We brainstormed a product-led version of that same personalized experience.

Why we didn’t pursue it:

High technical complexity for meaningful personalization
Limited data to support accurate segmentation
Not feasible within timeline and scope

Customized onboarding based on user’s intended use

Design thinking

While a fully customized onboarding experience was directionally compelling, it wasn’t feasible within our timeline or data constraints. I choose a ‘one-size-fits-most’ approach, focusing on the highest-impact tasks across users. This allowed us to move quickly, validate the model, and deliver and improve retention.

Key decisions

Optimized for speed and impact over full customization
Focused onboarding on behaviors that correlate with retention
Designed for momentum to build early user confidence
Identifying the highest impact onboarding task

Responsive variation (tablet and mobile)

Identifying the right onboarding task

To identify the right onboarding tasks, I used three methods:

Observational analysis: I studied the behavior of our marketing agency customers as they set up accounts for their clients, extracting key patterns and best practices.
Internal knowledge mining: I reviewed our Account Executives' guidebook to understand how they help new customers configure their accounts.
Collaborative workshops: I ran action mapping exercises with my PM and Content Strategist, inspired by "Better Onboarding" (A Book Apart). This helped us identify which actions new users need to build similar habits to experienced users.

This helped us prioritize the smallest set of actions that consistently led to long-term value.

Better Onboarding - Action mapping exercise

Onboarding task evaluation

Embedding onboarding into the homepage

We placed the checklist directly on the homepage, the first place users land.

As users complete tasks, the dashboard updates in real time to surface tangible progress and results. The core design challenge was integrating the checklist so it felt native to the homepage, not like an interruption.

Designing for momentum

The checklist was designed to reduce uncertainty and build momentum:

Task prioritization: We sequenced the checklist to balance effort with perceived value, building confidence while steadily revealing CallRail’s value.
Positive reinforcement: I added clear completion feedback to reinforce progress, increase confidence, and sustain motivation throughout onboarding.

Outcomes

The onboarding checklist improved both user outcomes and business metrics. To assess its long-term impact, I compared graduation rates between Q1 2023 and Q1 2024. Graduation rates, the percentage of customers who remain after the first 90 days, increased from 86.4% to 88.6%, a 2.55% improvement.

38% higher retention for accounts that completed the checklist

Unexpectedly, marketing agencies began using the checklist as a setup to-do list when configuring client accounts, extending its value beyond first-time users.

Reflection

Onboarding is inherently challenging for products that require upfront setup. For this project, we intentionally shipped a one-size-fits-most v1, applying the 80/20 rule to move quickly and deliver measurable impact. It was the right tradeoff for the stage we were in.

If revisiting this today, I would push further in two areas:

1. Adaptive onboarding

Our early exploration into customization was directionally right but technically complex at the time. With stronger data infrastructure and AI capabilities, onboarding could evolve based on user behavior and goals, shifting from a static checklist to a more responsive solution.

2. Persistent onboarding across the product

Due to technical constraints, we could only surface the guide on the homepage. Ideally, onboarding would be a persistent layer that’s accessible across the product, supporting users wherever they are in their journey.