A/B Testing is Not Personalization
But they complement each other…
We should think about these tools as complementary rather than one or the other. In general, performance marketing’s goal is to drive higher conversion rates, create new customers/users and increase revenue. We achieve this by leveraging the functionality, strengths and purpose of both toolsets together, feeding each other.
It’s not uncommon to try to use A/B testing tools for personalization, now even more that A/B testing software companies are trying to extend their offerings into the personalization world, while at the same time personalization tools are adding A/B testing functionality. The lines are somehow blurring, but even with that in mind, current A/B testing software are not strong enough to carry out website personalization and contextual marketing programs at scale, which are more intricate and contain more variables to execute a fully successful campaign. At the same time, testing platforms are not meant to deal with personalization, but when personalization is simple and short-term, they can certainly be useful.
A/B testing and Personalization are not independent efforts – once we all understand this, we can get much more out of our conversion rate optimization efforts.
The Differences: Use Cases, Strengths and Purpose
The following table summarizes the main differences between testing and personalization. Mainly, the way I like to summarize as follows:
- Testing is about trying out variations to see if something will work, it’s related to hypothesis-driven approach, reduced scope and reach.
- Personalization is about content production and delivery of customized experiences. Once you have proven out that something works, how do you operationalize it?
| A/B Testing | Personalization | |
|---|---|---|
| Description | – Test and identify best performing page/component/tactic to show all users. – One idea vs. another. – Also, to test for personalization ideas – but not long term. – E.g., deciding between two sign up flows, hero variations, etc. | – Personalization is about tailoring the experience – trying to identify who the user is. – Serve different versions of your site tailored to different contexts, interests, audiences. – More than one version of the site allows for custom targeting. – E.g., showing different hero images, different specials/offers, etc. |
| Use Cases | – One idea vs. the other – Identifying the ‘best’ for all | – Best page or content for each predefined audience – Better performance and audience behavior won’t change over time |
| Scope | – Basic / simple testing – 1 to all (or to few segments) – Unable to scale – Few tests/ideas at the same time – No need for 360 view of users – Past data is not necessary | – 1 to many (or 1 to 1) – One or several ideas at a time per audience – Tighter integrations with CRM and audience manager tools – Improves the more you know about the user – 360 view – Previous data is crucial to success |
| Duration/Length | – Shorter term – Fast cycles | – Ongoing personalization activities – Longer term |
| Tradeoffs | – Simpler management and governance – Keep it simple – One version at a time = simpler | – Time and effort to manage rules and multiple approve versions – Content efforts is bigger (producing more variations), but this means better performance. |
Using A/B Testing for Personalization
When is A/B testing enough? Can I use A/B testing as a proxy for personalization? Do I really need to overcomplicate the process with a personalization strategy beyond testing? If you are asking yourself these type of questions, you are not alone. As mentioned before, the answers to these questions get blurry specially because platforms are constantly evolving, and software companies are adding more and more personalization-like capabilities to their A/B testing offerings.
At the same time, you might already have a successful Testing practice in place, and in fact are doing some level of personalization already, so why would you need anything else? There are 2 main aspects to consider, specially when you are planning on building a contextual marketing practice, or personalization at scale: associated issues, and the maturity of your marketing operations.
The Issues
Content Management
When you start personalizing at scale, you’ll soon realize that your A/B or Multivariate testing software is no place to store your content variations. It’s a good place to run small and time-constrained tests, but as soon as you want to operationalize it, and have a team of designers and copywriters creating different variations for different segments and audiences, you’ll find out it is no replacement for a CMS. They just lack the general capabilities of content management, authoring and governance that are needed.
Also, if you have a team of content authors working in a specific CMS platform, you want that team to continue working and building that practice there, and not split content management efforts in 2 separate platforms. You can work out a hybrid model, where you keep the content in the CMS and control the variation rules and delivery through testing, but you’ll have to be extremely disciplined and organized.
QA & Testing
When you have 2 or more separate platforms responsible for the final delivered experience, it’s hard to test it. If new content, variations and rules are being kept in a separate platform that is not the core Experience Platform, QA & Testing will have a hard time validating if something is working or not.
This issue gets multiplied when the owners of the A/B testing platform are not the same as the ones owning the underlying CMS, which is not that uncommon (e.g., CMS – content authors and devs, A/B testing – marketing sciences).
Be aware that high levels of collaboration will be required to make this approach work.
Codebase Synchronization
The majority of testing tools like Adobe Target, Optimizely and VWO are client-side tools, which means that they take the rendered page, and swap and modify the HTML/DOM of it to display what the user needs to see. So, on one hand you have a server-side platform (e.g., your CMS – WordPress, Drupal, Sitecore, etc.) that sends the HTML to the user, and the Testing software takes this HTML, manipulates it and shows it to the user.
What this causes is possible codebase synchronization issues. Testing software doesn’t own the underlying codebase, so the best they can do is to manipulate the HTML they see at that particular moment in time. As soon as there is a change in the codebase (i.e., in the CMS), the Testing platform’s version of the HTML will be different and it will break.
User Experience
Also related to testing tools being client-side technologies is the fact that it might impact the overall user experience. Client-side means that it happens on the user’s browsers, when the page has already been served and rendered by the browser.
Depending on how you are loading your scripts, or what exactly your test/personalization is, the user might perceive this change happening in real-time. Or, the user might see a white space before the final personalized content is finally rendered.
The impact of this type of experience is up to each project and client. Sometimes it is unacceptable, and sometimes it’s ok and not worth the trouble looking for alternatives or workarounds.
The Maturity & Vision
The other question to ask yourselves in terms of figuring out if your A/B testing tool can be used for long term personalization is about the marketing operations maturity you currently have, or what your vision is? This is the cliché of crawl > walk > run. But it’s important to be honest and analyze both the current maturity of your contextual marketing operations, as well as where you want to be. Not everyone needs to run. What if for your business and needs, walk is more than enough for the next few years? That happens, and that’s valid.
It also involves understanding the team, the roles, the process and possibly the organizational transformation necessary to achieve the level of contextual marketing you want to achieve. How you assess all of this? Sitecore provides a very nice checklist/guide you can use to start your own journey: https://www.sitecore.com/resources/sbos/site-readiness-checklists.
How to Use Testing & Personalization Together
- A/B testing for decision making, traffic splitting and segmentation, and testing simple personalization ideas.
- Personalization for content management, versioning, component/page library and context marketing at scale.
Testing & Personalization Process
The following diagram hopefully encapsulates how both Testing (A/B and Multivariate) and Personalization play a complementary role in your digital and contextual marketing journey. It also helps to visualize the process, the cadence of the work, the team and roles that are needed for each, etc. I use it all the time for my own projects when needed.
References
- The Complete Context Marketing Readiness Guide | Sitecore
- Plotting your Path to Personalization with Digital Experience Maturity Model | Sitecore
- Personalization at Scale | Adobe
- Website Personalization vs A/B Testing – A Marketer’s Perspective | Acquia
- A/B Testing | Optimizely
- Website Updates Need Rigorous Testing – Choose A/B Testing or Multivariate Testing
