If you’re determined, you can get your statistics to tell you any story you want. In the metric-obsessed world of online marketing this is particularly dangerous as it’s easy to make the wrong decisions by being biased from preconceptions about what the conclusions should be.
Here are five common pitfalls in stats:
1) Correlation doesn’t imply causation
Anyone who have ever studied a statistical subject or opened a book on the topic will have come across this expression. It basically tells you that just because two events are correlating; it doesn’t prove that the one factor is causing the other to behave in a certain way.
For example: let’s say you are experiencing a correlation in paid search average position and conversion rate. Based on this you might assume that the position is directly impacting the conversion rate. However, by looking beyond the figures, you can see that there are other factors impacting the conversion rate, which brings us to the next point…
2) Looking at stats in isolation
By further interrogating the above example, it could well be discovered that the average campaign position increased due to competitors dropping of in the space. Thus the increase in conversion rate was not due to the average position change, but simply down to the fact that there was less competition.
3) Lack of benchmarking
A 20% increase in site traffic from a particular campaign might sound great at first, but it doesn’t actually tell you much unless you know the point of reference or can benchmark against other activity. Look for industry benchmarks, and historical performance of similar campaigns to better understand what to expect.
4) Small sample sizes
A very common pitfall is that of using small sample sizes and not achieving statistical significance, before judging performance. As the lead in to this article states – you can prove anything with statistics.
For example: It’s easy to get excited about a 20% increase in site conversion rate. However, if that 20% increase is measured by a split test where 100 visitors are leading to 6 sales compared to 100 visitors leading to 5 sales, the results are not statistically significant (in this instance probably have to wait for another 5,000 or so visits…).
5) Graphs don’t lie…
…they are just very easy to misinterpret and/or distort. Consider the three graphs below that are all based on the same data. Chart A and C shows exactly the same amount of data over time, but there’s a vast difference in what story they tell – chart A shows dramatic ups and downs, whereas chart B shows a fairly steady scenario. Chart B is zooming in on the last few data points, and is thus showing a tremendous upswing that would look good in any pitch.
What common misuse of statistics have you come across?
Related posts:
