Daniel Schulman

The Tyranny of Numbers
Often when reading webmaster forums or speaking with people at conferences, I hear them refer to their conversion rates as if they had absolute certainty in their accuracy. After all their web analyst, market research person, or the pony-tailed dude down in tech, provide them with a daily/weekly/monthly/etc. report that says X percent of visitors to our website contacted us, bought something, or did some other tracked action.

If the report says X percent, conversion is X percent — how can that be any more clear?

To comments like this, I usually start raising some red flags: so are you controlling for multiple referrers? do you credit first or last referrer or something else? how do you do cross-machine cookie syncing? Eyes glaze over, questions get answered with questions, all of a sudden I’m being made to feel stupid because I cannot understand that if a report says conversion is X percent, then conversion is X percent - duh!

In all fairness, much of the difficulty comes from the fact that we are often speaking past each other. Most people who look at conversion numbers are engaged in relatively straightforward eCommerce. They have a website that sells widgets for $39.95, they write an search ad that says “get your best widgets here,” someone searches for widgets, clicks the ad, and buys the widget for $39.95 all in the same session. In this type of relatively straightforward situation, the objections I raise to conversion stats still apply, but they are not so severe.

In the case of long or complex sales cycles or cases where the lead is generated online and the sale is finalized offline, conversion data might be completely meaningless. Basing business decisions on such faulty data can lead to all sorts of problems.

Delayed Conversion Examples

1. Complex Offerings. I’ve been doing online marketing for a law firm that specializes in DUI defense cases. Not overly surprising, when someone who has had no prior contact with the criminal justice system gets taken to jail and charged with a serious crime, they often conduct extensive Internet research on their situation. Also not too surprisingly since I’ve been doing the online marketing for awhile, this firm has a huge presence throughout all of the Internets. Since nothing in this example has been too surprising so far, it shouldn’t be much of a shocker to be told that we often see visitors come and go from the website over the course of several hours arriving from a variety of sources including organic searches, pay-per-clicks, Internet Yellow Pages, niche directories, and partner sites.After their exhaustive research with multiple visits to the website, the person figures out they are in serious trouble and this law firm represents their best chance at minimizing impacts, they fill out the contact form, and I tally up the conversion to the referring source. Oops, there is not a single referrer.
2. Long Sales Cycle. I have been working with a summer camp company helping to generate customers for their business. In this business, from the time a person hears about the program until they consider all of their options and make a deposit, can take upto several months and is almost always at least two weeks. We also see a great deal of website activity on Monday and Tuesday mornings whereas deposits tend to come in Friday afternoons and over the weekend. Based on these patterns, we believe that the modal customer gathers information camp information during the week at the office, but waits until they are at home with the kids to make final purchase decision.With long sales cycles, some basic assumptions built into many conversion stats are nolonger tenable. You cannot assume that the customer will make the purchase (or other action) from the same computer they got the referral from. You cannot even assume that if they are using the same computer, they did not wipe their cookies between referral and purchase. A JupiterResearch study, found that “Nearly 40 percent of Internet users delete cookies from their primary computers on at least a monthly basis.”
3. Offline Final Conversions. Offline sales as a tracking problem tend to get more interest than other the other sources of error. Heightened interest in finding solutions to these problems likely results from the fact that the pattern of online research and offline sales often applies to higher priced items such as cars, TV’s, financial services, and professional services. Certainly my prior examples of summer camps and legal service contacts have the majority of their conversions offline.

Future Posts
In this post, I layed out some of the reasons why you might not want to accept your conversion stats as gospel. In planned follow-ups, I’ll look at issues such as how web analytics packages arrive at conversion stats and how to get a better understanding of which sources lead to conversion.

This entry was posted on Tuesday, March 27th, 2007 at 4:09 pm and is filed under web analytics. You can subscribe via RSS 2.0 feed to this post's comments. You can comment below, or link to this permanent URL from your own site. Your comments will appear immediately, but I reserve the right to delete innapropriate comments.