CAMPAING DESIGN at AMAZON.COM Case Study

The Test Mailing

In the initial phase we decided to send the test mails to 12 groups. 12 groups was in our opinion enough to get accurate results, because both ‘promo’ and ‘excerpt’ are tested with each review at least once. Choosing all 24 groups would have given us the most accurate response rates, but we thought it would be too expensive to conduct since the cost of one group (i.e. test cell) is $7000.

However, the marginal cost of e-mails is almost negligible so we decided to send as much e-mails as possible. Although 50 000 was the maximum number, dividing it by 12 gives us an uneven number (4166.666…) so we decided to go with even 4000. It is easier figure for the management to understand. The test mailing table is given below.


The Roll-Out

Before doing any thorough statistical analysis with STATA, we decided to do a simple analysis with Excel. This was only because our team was not so familiar with STATA and this way we were able to get some hints what the ‘actual’ results might look like. So, we calculated simple response rates for our test cells. You can find them in the tables below


So, we found out that the highest response rate for men was in ‘no-promo’-‘no-excerpt’-‘review = 2’ category and for women it was in ‘promo’-‘excerpt’-‘review = 1’ category. We found that Excel analysis was insufficient so we decided to do the analysis with STATA also.

First, we created the dummy variables as instructed in the case. Then we did the logistic regression and check the significant p-values. Everything else except ‘review = 1’ for men was significant. We later eliminated this problem by setting its odds ratio to 1.

After the logistic regression we imported the data to Excel in order to do the odds ratio table. You can find the two versions of tables below. The first one is the original with ‘review =1’ handled as significant and the revised is where it is set to 1.


So, we founded out that the best groups to send the roll out mails were for men ‘no-promo’-‘excerpt’-‘review = 2’ and for women ‘promo’-‘no-excerpt’-‘review = 1’.

For the final roll out we decided to focus on these two groups for two reasons. Firstly, the population is said to be almost evenly divided between men and women so we ‘have to’ send mails also to men. Secondly, fixed cost of sending mails to many groups is expensive and of course it is wise to focus on the most profitable groups. You can find our final roll out mailing strategy below.