• Frequently asked questions

Asking a question is like testing. Listening to the answer is learning.

What is Test & Learn about, and how does it work?

Excellent question as a starting point! Look at the Home page or at our “concept overview” and “detailed approach” pages to learn everything you need to know about Test & Learn. And if you still have questions, feel free to contact us!

I already use some statistical techniques such as split testing or champion challenger to improve my campaigns. Why would the T&L approach give me some better results?

First of all, congratulations for already using these techniques – they are far from being useless. But there are some big differences between these methodologies and the Test & Learn. Please check our Unique approach page to learn more about that.

That seems a bit complicated... Are we sure that my firm can actually perform this kind of activity?

If you do Direct Marketing, you probably can do Test & Learn. There are some simple requirements that you certainly already meet. Check it on our T&L fits you well page. And stay unfrazzled: Teanler can take care of everything.

What is the minimum number of contacts to run a Test & Learn campaign?

Well, it depends. The rule is to make sure that you will have at least 100 observations (ie. positive “respondents”) per test cell and per customer segment to analyze. This means that 3 factors do impact the minimum number of contacts you need for a specific campaign: the expected redemption rate, the number of test cells, and the number of customer segments for which you want to identify a best offer. For instance, you would need at least 100.000 contatcs for a test campaign with 5 test cells, 1 single customer segment (eg overall population) and a 0,5% expected redemption rate: this would correspond to 20.000 contacts per test cell, which should give you around 100 observations each with a 0,5% redemption rate. This minimum quantity would double if you had 2 different customer segments, or if you had 10 cells instead of 5. Please check out our T&L fits you well page to get more details about this topic.

How many test cells are there in a test, on average?

For each test, there is always a minimum number of test cells which are required to make sure that the statistical models will be able to extrapolate the observed results to all theoretical combinations. This minimum number of test cells depends on the complexity of your test: the more sophisticated your test will be, the more cells you will need.The complexity of a test is driven by 3 main factors: the number of dimensions, the number of values for each dimension, and the presence of cross-effects between 2 or more dimensions. For instance, you will need at least 7 test cells for a 3-dimension test with 3 values for each dimension, and no cross-effects (corresponding to a total of 3x3x3=27 theoretical combinations). Teanler uses some maths formulas to calculate the minimum number of cells for a specific test. Please contact us to know more about that.

How do I choose the dimensions and values that I may test?

You can play with every dimension of your value proposition – from pricing to communication to product features. Anything you can adjust in your value proposition is a potential candidate for Test & Learn, and your only limit is your creativity! Obviously, you will tend to prefer those dimensions that you can easily adjust without (or with very low) investment.

I've never run a Test & Learn campaign before. How can I make sure that it will work for me?

The best way to know if the Test & Learn approach can work for you is to follow the principle itself: just test, you’ll learn a lot! That said, keep in mind that it’s hard to find a company that tried but eventually did not manage to leverage successfully the Test & Learn methodology. But one thing is certain: doing it with Teanler is definitely the best way to maximize your probability of success.

How much time does it take to run a Test & Learn campaign from A to Z?

Short answer: around 1 month of T&L-specific activities + all the time you need to launch the campaign operationally (creativity, printing, sending, …) and to observe the results over time (eg. 3 months for a mailing campaign, 1 month for an emailing campaign, …).

Detailed answer: let’s distinguish 2 things. The technical part of the methodology on the one hand, and the operational part on the other hand. A test campaign is always structured around 4 main steps: define what you want to test, run the models, launch your campaign, and analyze the results. Steps #2 and #4 are “technical” steps, while steps #1 and #2 are “operational” steps. For the technical steps, the average completion time is quite standard and does not vary too much: 1 to 2 weeks for step #2, and 2 to 4 weeks for step #4. The small variation is function of the test complexity: it will be longer to build and analyze a multi-cluster test with cross-effects than a simple test with standard characteristics.For the “operational” steps, it’s all about you. Step #1 should be quite straight-forward if you already have a good idea of what you want to test, and how you want to test it. Let’s consider between 1 and 2 weeks to reach the objective.Step #3 is by far the longest one, but also the most traditional. You have to sum the time you need to build the communication (eg. 1 month for a mailing, 2 weeks for an outbound call campaign, …), plus the time you need to get all the results that can reasonably be associated to the campaign (eg. 2 to 3 months for a mailing, 1 month for an emailing campaign, …).

Will it be possible to register the economic and financial results of the test immediately after the end of the campaign?

Not exactly. The main goal of a test is to identify the best offer(s) that you will be able to launch during the next campaigns (also called roll-out or generalisation phase). Because the test campaign itself is comprised of cells that outperform and others that underperform the standard offer (benchmark), the overall results are generally very close the standard offer (that you would have launched normally). But right from the next campaign onwards, you will be able to actually register the economic and financial results that have been estimated at the end of the test, thanks to the actual launch of the best offer(s).But remember, Test & Learn is a dynamic : you barely launch a test campaign only. Most players perform tens of different tests per year, in order to get closer and closer to the optimal offer around all the possible dimensions. Some tests may fail… but they will be more than compensated by the success of the others.

Which channels are eligible for the Test & Learn approach, and which ones work best?

There is no “incompatibility” regarding channels when it comes to running a test campaign. You can use outbound calls, mailings, emailing, couponning, SMS, … You can even use alternative channels such as a message on your banking receipt at the ATM, or private messages to your Facebook fans…! In terms of efficiency, we highly recommend to prefer those channels which offer the largest room for customization at the lowest cost. Emailing is a good example, but this is probably also true for others channels, according to your specific context. That’s definitely a topic we will discuss together when we’ll launch the first Test & Learn campaign for you!

How do I measure the performance increase that is actually generated by the Test & Learn approach?

This is a key element of the methodology. The Test & Learn approach is a highly quantitative technique and, as such, must deliver some reliable results that can be immediately and easily construed. That’s why each test campaign must include one “standard” cell (also called “benchmark”) which corresponds to the value proposition / communication which would have been launched independently from the test. All the performance gaps will be calculated by comparing the results (redemption rate, revenues, profitability) between this standard cell and all the theoretical combinations of the test (launched or not).The same rule applies to the roll-out phase, where the best offer(s) that you will send to the vast majority of your target will always be compared to the standard offer that you will keep on sending to a limited number of selected customers.

Can I also include a placebo in my test in order to check the actual impact of my DM campaign?

Sure, you can. But keep in mind that the placebo has another goal (ie. identify the impact of the DM campaign vs. no Direct Marketing) and must be considered as independent from the Test & Learn itself. This placebo will tell you if it’s worth running a DM campaign, but will not interfere with the identification of some best offers within the campaign.