The Law of Small Number

The Law of Small Numbers
One of the common fallacies in PPC is the law of the small numbers. This term was coined by Kahneman to describe how people exaggerate the degree to which the probability distribution in a small group will closely resemble the probability distribution in the overall population. Kahneman illustrate how people expect close to the same probability distribution of types in small groups as they do in large groups, asking a group of undergraduates the following question:

A town is served by two hospitals. In the larger hospital about 45 babies are born each day, and in the smaller hospital about 15 babies are born each day. As you know, about 50 percent of all babies are boys. However, the exact percentage varies from day to day. For a period of 1 year, each hospital recorded the days on which more than 60 percent of the babies born were boys. Which hospital do you think records more such days?

Only 22% of subjects answered correctly that the smaller hospital would report more such days.

Data Sparsity
Even though we as advertisers are given a tremendous amount of data a number of challenges arise when estimating conversion probabilities. One of the major challenges is data sparsity. The majority of keywords in an advertiser’s portfolio generate only little traffic. For these sparse keywords, taking simple averages of clicks and conversions in the past is insufficient to estimate the CVR as these estimates can be highly inaccurate. As illustrated in this picture*, it shows the proportion of conversions we can expect to observe given a certain sample size for 2 probabilities: 1% and 5%. The figure shows that one needs at least a couple of hundred clicks for the sample average to be a reliable estimator of the true CVR. If a keyword generates little traffic, it may take a while before enough clicks are collected. The true probability may already have changed within this time frame. What’s often not taken into consideration as well is the effect of the CVR itself on the amount of confidence you can put on how likely the sample CVR will resemble the actual CVR. As you can see in the picture, the lower the CVR the more data you need to be confident it will resemble a similar rate. To make it worse, CVR’s tend to be low in a typical PPC-account. Which makes it hard to be confident about the CVR we see on the lowest levels. This often results in pausing ad groups too early or to give too much weight onto yesterday’s winners.

Monte Carlo Model
So now, if you sacrificed your valuable time to read this already quite lengthy post, you may wonder: is there nothing we can do? Fortunately yes, this is not an uncommon problem in other fields either and smart statisticians came up with a method: the Monte Carlo experiment. But that’s a topic for another time