Building Performance Marketing Tests

What is the most important thing for young marketers to understand, when formulating performance marketing tests and testing different strategies? How do you approach that and what principles would you suggest they use, to really roll out tests?

Good question. The first thing is, what is your hypothesis? What are you trying to prove here? In this case, the company I worked at, they said, display advertising doesn’t work, because people don’t click on it, they don’t see it, they don’t look. So the hypothesis is, that I formulated, people seeing this banner are more likely to book with EasyToBook than people who haven’t seen it. Then you have to think about, how am I going to test that? I need to have a group of people that have seen it; I need to have a group of people that haven’t seen it. You can do that in various ways. You could, for instance, do a pre/post analysis. At a certain timeframe, you haven’t shown the banner. At a certain timeframe, you show the banner and you see what the difference is in traffic, across those two.

That’s a toughie, especially in travel, because if you have Easter here and no Easter here, travel behavior differs. Then you can think about a product split, like I discussed before. For Barcelona, from certain origin markets, you show the banner; for other origin markets, you don’t show the banner. It’s called a geo split. As long as the behavior of those different geos is similar and you have large enough tests, so you can get a statistical outcome, then that might work.

So you have geo tests, and you have a product split test, as well. So on some products, you could this for both and then flip it around. Then you have a cookie split test, which you might do on retargeting, for instance. On your own users, who are on your website, you would show banners to 50% of them and not show banners to the other 50% and then see, with both groups, how many of those are booking, eventually. But of course, you have to be careful that you randomize how you split the two groups. If you have one group that had already gone all the way to the booking checkout page, and the other group has only hit the home page, different groups; you can’t compare them.

Lastly, the best kind of test that you can do, is a user split test. In this case, you need some help. Let’s start with yourself. You could do a user split test if you look at your email database. If you have a large email database, you can make a split at the email level, then by user and you can do different AB experiments, on that. Of course, if you buy traffic with Google and Facebook, you might not have all of the email addresses of those people. You can do some user splits, by connecting your CRM database to Google and Facebook, but it’s limited to the size of your own database.

Then you need help, from Google and Facebook and you work with them, on a user split. Then Google would actually do the splitting of the two groups, at their end. They would take a look at, who are you trying to reach, what’s your audience and how can we move those into two different groups that are the same in their behavior, but we make sure that whenever you show a video or an ad to the test group, that there is a group that is a control, that does not get that ad. Because Google and Facebook have such a large log-in userbase, they can actually do this and prevent people from ever seeing your ad.

I think that is the most important thing for people to understand is those different types of testing, time based, geo based, product based, cookie based and user based. You also need the basics of statistics. You need to understand how P value works. You need to understand when you can see that a certain experiment is statistically significant and you can say, with 95% certainty, that the outcome is true. But you still have to understand that there is a 5% chance that it’s not. So you need to rinse and repeat a lot of these experiments, to make sure that you are still looking at the right data.

What do you think the biggest mistake is, that young marketers make on running performance marketing tests?