When Measuring Performance, Find Lines in the Sand

One of the most useful ideas from the 2013 book Lean Analytics (book summary) is the idea of the ‘line in the sand’ — the book’s name for a specific value for a metric that tells you if you’re doing well or doing badly.

The book illustrates the power of this idea with the story of WP Engine founder Jason Cohen’s search for better retention:

WP Engine is a service company. Its customers rely on WP Engine to provide fast, quality hosting with constant uptime. WP Engine is doing a great job, but customers still cancel. All companies have cancellations (or churn), and it’s one of the most critical metrics to track and understand —not only is it essential for calculating metrics like customer lifetime value, but it’s also an early warning signal that something is going wrong or that a competing solution has emerged.

Having a cancellation number isn’t enough; you need to understand why people are abandoning your product or service. Jason did just that by calling customers who cancelled. “Not everyone wanted to speak with me; some people never responded to my calls,” he recalls. “But enough people were willing to talk, even after they had left WP Engine, that I learned a lot about why they were leaving.” According to Jason, most people leave WP Engine because of factors outside of the company’s control (such as the project ending where hosting was needed), but Jason wanted to dig further.

Having a metric and an understanding of the reasons people were leaving wasn’t enough. Jason went out and found a benchmark for cancellation rate. This is one of the most challenging things for a startup to do: find a relevant number (or line in the sand) against which to compare yourself. Jason researched the hosting space using his investors and advisors. One of WP Engine’s investors is Automattic, the company behind WordPress, which also has a sizeable hosting business.

Jason found that for established hosting companies, there’s a “best case scenario” benchmark for cancellation rate per month, which is 2%. That means every month—for even the best and biggest hosting companies around—you can expect 2% of your customers to leave.On the surface, that looks like a huge number. “When I first saw our churn, which was around 2%, I was very concerned,” Jason says. “But when I found out that 2% is pretty much the lowest churn you’ll get in the hosting business, it changed my perspective a great deal.” Had Jason not known that this is simply a fact of life in the hosting industry, WP Engine might have invested time and money trying to move a metric that wouldn’t budge—money that would have been far better spent elsewhere.

Instead, with a benchmark in hand, Jason was able to focus on other issues and key performance indicators (KPIs), all the while keeping his eye on any fluctuation in cancellation rate. He doesn’t rule out the possibility of trying to break through the 2% cancellation rate at some point (after all, there can be significant value in reducing that churn), but he’s able to prioritise according to what’s going on in his business today, and where the biggest trouble spots lie, all while keeping an eye on the future success of the company.

By searching for an industry benchmark to compare himself against, Cohen discovered that losing 24% of customers every year was the best one could do in the hosting business.

This gave him the peace of mind to pursue other goals.

The lesson here is: until you know what’s normal, you can’t judge if your metrics are good or bad. Thus, as you get more data-driven, it's important to discover the lines-in-the-sand for your specific business.

In this post, we’ll look at ‘lines in the sand’ (LITS) for several broadly used metrics, as presented in Lean Analytics. As always, these recommendations are heuristics — true in the general case, but possibly different when applied to an individual company in a specific sector. Caveat emptor.

LITS: Cost of Customer Acquisition

The rule of thumb for customer acquisition costs is this: your CAC should be less than a third of the total value a customer brings to you over her lifetime. This rule is most applicable for subscription based businesses, like software-as-a-service (SaaS).

Why is this the case? There are a couple of reasons:

1. The lifetime value (LTV) you calculate is probably wrong. It takes a really long time to calculate true LTV, especially if the customer sticks around for a couple of years. You’ll want to build a margin of error into your estimate.
2. The CAC you calculate for a customer is probably wrong as well! In the early years of a business, most companies don’t do a good job at accounting for hard-to-measure up-front costs like onboarding and support, adding additional infrastructure, and the like.
3. All subscription businesses go through what’s called a ‘cash trough’. In the beginning, they’re spending money to acquire customers, in the hopes that they can recoup those costs later once a customer has stuck around long enough, paying sufficient subscription fees as they do so. Keeping CAC to a third of LTV protects you from creating a situation where your cash flow problems become unmanageable.

LITS: Mailing List Effectiveness

There’s a fairly large body of work around what makes for an effective email marketing campaign. A quick summary:

• Email open rates vary by industry. Construction, home and garden and photo emails achieve nearly 30% open rates. Emails related to medicine, politics and music get as little as 14%. (web source)
• Data from Mailchimp suggests the biggest factor that predicts mailing list effectiveness is subject line. A good one gets an open rate of 60-87%, and a bad one 1-14%. The best subject lines include simple, self-explanatory messages that include something about the recipient.
• Also per Mailchimp, targeting mailings by tailoring messages to different segments of the same subscriber base improves clicks and opens by nearly 15%.
• 3pm is when people are most likely to open something.
• Obvious: more links in an email means more clicks, and newer subscribers are more likely to click on a message.
• Experian reports that the word ‘exclusive’ alone in email promotional campaigns increased unique open rates by 14%.
• But bear in mind that open rate is a fundamentally flawed metric, as it depends on the mail client to load a hidden image — which most modern mail apps no longer do by default.

The overall advice that Lean Analytics settles on is that a well-run email campaign should hit between 20-30% open rates, with at least 5% click-through. Though of course these numbers vary according to industry and campaign type.

LITS: Uptime and Reliability

You can’t always control uptime. Sometimes the cloud providers you build your services on go down. Other times underlying Internet infrastructure causes problems.

Lean Analytics suggests that a yearly uptime of 99.95% is prohibitively costly for startups, as it means you can only afford to be down 4.4 hours a year. A more realistic line in the sand is to aim for:

• At least 99.5% uptime if you offer a paid service that users rely on (e.g. email service, or project management app). Note that this means two days and four hours of downtime a year.
• Lower levels of service if your app isn’t as critical. Note that 99% uptime means  three days and 15 hours of downtime a year — this is certainly more applicable to startups that are early in their journey!

LITS: Site Performance

There’s a famous ratio called 30/10/10 that was popularised by Fred Wilson in a 2011 blog post. Wilson writes:

One thing that never ceases to amaze me is how similar some of the metrics are from service to service and company to company. I like to call these the web/mobile laws of physics. One fairly common "law of web/mobile physics" is the ratio of registered users/downloads to monthly actives, daily actives, and max concurrent users (for services that have a real time component to them).

I call this ratio 30/10/10 and so many services that we see exhibit it within a few percentage points here and there. Here's how it works:

30% of the registered users or number of downloads (if its a mobile app) will use the service each month.

10% of the registered users or number of downloads (if its a mobile app) will use the service each day.

The max number of concurrent users of a real-time service will be 10% of the number of daily users

IBM notes a similar ratio for business intelligence applications:

As a general rule, the ratio of named to active to concurrent users for business intelligence applications is about 100:10:1. In other words, for every 1000 named users there are 100 active users and 10 concurrent users.

In this example, a named user is a user that is authorised to use the BI tool, and an active user is a user that is currently using the tool. IBM recommends that customers who deploy Cognos, their BI solution, use this ratio to determine the amount of system resources to provision for the application.

For a web service, Lean Analytics notes that the hard threshold for users to leave is around 15-18 seconds. They argue that first-time visitors should never experience page loads above 5 seconds. These are page load goals, however — in practice, you want to optimise for the ‘first meaningful paint’, which Google defines as ‘the time at which the user feels the primary content of the page is visible’. That metric should be less than 1 second.

Conclusion

These ‘lines in the sand’ are only for the most generic, broadly applicable metrics. For more detailed LITS that are applicable to specific business models, check out our summary of Lean Analytics, where we turn the book’s business-specific benchmarks into an easily-downloadable PDF reference.

Data analytics can seem difficult. From experience, we think the most challenging part is building the data-driven mindset necessary to succeed in today’s data-rich world. The ‘line in the sand’ idea from Lean Analytics shows us one way to think about data analysis: measure data, find lines in the sand, and set goals according to what the lines tell you are good.