marketing experiments project management

From Hypothesis to Results: Mastering the Art of Marketing Experiments

Max 16 min read

Click the button to start reading

Suppose you’re trying to convince your friend to watch your favorite movie. You could either tell them about the intriguing plot or show them the exciting trailer.

To find out which approach works best, you try both methods with different friends and see which one gets more people to watch the movie.

Marketing experiments work in much the same way, allowing businesses to test different marketing strategies, gather feedback from their target audience, and make data-driven decisions that lead to improved outcomes and growth.

By testing different approaches and measuring their outcomes, companies can identify what works best for their unique target audience and adapt their marketing strategies accordingly. This leads to more efficient use of marketing resources and results in higher conversion rates, increased customer satisfaction, and, ultimately, business growth.

Marketing experiments are the backbone of building an organization’s culture of learning and curiosity, encouraging employees to think outside the box and challenge the status quo.

In this article, we will delve into the fundamentals of marketing experiments, discussing their key elements and various types. By the end, you’ll be in a position to start running these tests and securing better marketing campaigns with explosive results.

Why Digital Marketing Experiments Matter

One of the most effective ways to drive growth and optimize marketing strategies is through digital marketing experiments. These experiments provide invaluable insights into customer preferences, behaviors, and the overall effectiveness of marketing efforts, making them an essential component of any digital marketing strategy.

Digital marketing experiments matter for several reasons:

• Customer-centric approach: By conducting experiments, businesses can gain a deeper understanding of their target audience’s preferences and behaviors. This enables them to tailor their marketing efforts to better align with customer needs, resulting in more effective and engaging campaigns.
• Data-driven decision-making: Marketing experiments provide quantitative data on the performance of different marketing strategies and tactics. This empowers businesses to make informed decisions based on actual results rather than relying on intuition or guesswork. Ultimately, this data-driven approach leads to more efficient allocation of resources and improved marketing outcomes.
• Agility and adaptability: Businesses must be agile and adaptable to keep up with emerging trends and technologies. Digital marketing experiments allow businesses to test new ideas, platforms, and strategies in a controlled environment, helping them stay ahead of the curve and quickly respond to changing market conditions.
• Continuous improvement: Digital marketing experiments facilitate an iterative process of testing, learning, and refining marketing strategies. This ongoing cycle of improvement enables businesses to optimize their marketing efforts, drive better results, and maintain a competitive edge in the digital marketplace.
• ROI and profitability: By identifying which marketing tactics are most effective, businesses can allocate their marketing budget more efficiently and maximize their return on investment. This increased profitability can be reinvested into the business, fueling further growth and success.

Developing a culture of experimentation allows businesses to continuously improve their marketing strategies, maximize their ROI, and avoid being left behind by the competition.

The Fundamentals of Digital Marketing Experiments

Marketing experiments are structured tests that compare different marketing strategies, tactics, or assets to determine which one performs better in achieving specific objectives.

These experiments use a scientific approach, which involves formulating hypotheses, controlling variables, gathering data, and analyzing the results to make informed decisions.

Marketing experiments provide valuable insights into customer preferences and behaviors, enabling businesses to optimize their marketing efforts and maximize returns on investment (ROI).

There are several types of marketing experiments that businesses can use, depending on their objectives and available resources.

The most common types include:

A/B testing

A/B testing, also known as split testing, is a simple yet powerful technique that compares two variations of a single variable to determine which one performs better.

In an A/B test, the target audience is randomly divided into two groups: one group is exposed to version A (the control). In contrast, the other group is exposed to version B (the treatment). The performance of both versions is then measured and compared to identify the one that yields better results.

A/B testing can be applied to various marketing elements, such as headlines, calls-to-action, email subject lines, landing page designs, and ad copy. The primary advantage of A/B testing is its simplicity, making it easy for businesses to implement and analyze.

Multivariate testing

Multivariate testing is a more advanced technique that allows businesses to test multiple variables simultaneously.

In a multivariate test, several elements of a marketing asset are modified and combined to create different versions. These versions are then shown to different segments of the target audience, and their performance is measured and compared to determine the most effective combination of variables.

Multivariate testing is beneficial when optimizing complex marketing assets, such as websites or email templates, with multiple elements that may interact with one another. However, this method requires a larger sample size and more advanced analytical tools compared to A/B testing.

Pre-post analysis

Pre-post analysis involves comparing the performance of a marketing strategy before and after implementing a change.

This type of experiment is often used when it is not feasible to conduct an A/B or multivariate test, such as when the change affects the entire customer base or when there are external factors that cannot be controlled.

While pre-post analysis can provide useful insights, it is less reliable than A/B or multivariate testing because it does not account for potential confounding factors. To obtain accurate results from a pre-post analysis, businesses must carefully control for external influences and ensure that the observed changes are indeed due to the implemented modifications.

How To Start Growth Marketing Experiments

To conduct effective marketing experiments, businesses must pay attention to the following key elements:

Clear objectives

Having clear objectives is crucial for a successful marketing experiment. Before starting an experiment, businesses must identify the specific goals they want to achieve, such as increasing conversions, boosting engagement, or improving click-through rates. Clear objectives help guide the experimental design and ensure the results are relevant and actionable.

Hypothesis-driven approach

A marketing experiment should be based on a well-formulated hypothesis that predicts the expected outcome. A reasonable hypothesis is specific, testable, and grounded in existing knowledge or data. It serves as the foundation for experimental design and helps businesses focus on the most relevant variables and outcomes.

Proper experimental design

A marketing experiment requires a well-designed test that controls for potential confounding factors and ensures the reliability and validity of the results. This includes the random assignment of participants, controlling for external influences, and selecting appropriate variables to test. Proper experimental design increases the likelihood that observed differences are due to the tested variables and not other factors.

Adequate sample size

A successful marketing experiment requires an adequate sample size to ensure the results are statistically significant and generalizable to the broader target audience. The required sample size depends on the type of experiment, the expected effect size, and the desired level of confidence. In general, larger sample sizes provide more reliable and accurate results but may also require more resources to conduct the experiment.

Data-driven analysis

Marketing experiments rely on a data-driven analysis of the results. This involves using statistical techniques to determine whether the observed differences between the tested variations are significant and meaningful. Data-driven analysis helps businesses make informed decisions based on empirical evidence rather than intuition or gut feelings.

By understanding the fundamentals of marketing experiments and following best practices, businesses can gain valuable insights into customer preferences and behaviors, ultimately leading to improved outcomes and growth.

Setting up Your First Marketing Experiment

Embarking on your first marketing experiment can be both exciting and challenging. Following a systematic approach, you can set yourself up for success and gain valuable insights to improve your marketing efforts.

Here’s a step-by-step guide to help you set up your first marketing experiment.

Identifying your marketing objectives

Before diving into your experiment, it’s essential to establish clear marketing objectives. These objectives will guide your entire experiment, from hypothesis formulation to data analysis.

Consider what you want to achieve with your marketing efforts, such as increasing website conversions, improving open email rates, or boosting social media engagement.

Make sure your objectives are specific, measurable, achievable, relevant, and time-bound (SMART) to ensure that they are actionable and provide meaningful insights.

Formulating a hypothesis

With your marketing objectives in mind, the next step is formulating a hypothesis for your experiment. A hypothesis is a testable prediction that outlines the expected outcome of your experiment. It should be based on existing knowledge, data, or observations and provide a clear direction for your experimental design.

For example, suppose your objective is to increase email open rates. In that case, your hypothesis might be, “Adding the recipient’s first name to the email subject line will increase the open rate by 10%.” This hypothesis is specific, testable, and clearly linked to your marketing objective.

Designing the experiment

Once you have a hypothesis in place, you can move on to designing your experiment. This involves several key decisions:

Choosing the right testing method:

Select the most appropriate testing method for your experiment based on your objectives, hypothesis, and available resources.

As discussed earlier, common testing methods include A/B, multivariate, and pre-post analyses. Choose the method that best aligns with your goals and allows you to effectively test your hypothesis.

Selecting the variables to test:

Identify the specific variables you will test in your experiment. These should be directly related to your hypothesis and marketing objectives. In the email open rate example, the variable to test would be the subject line, specifically the presence or absence of the recipient’s first name.

When selecting variables, consider their potential impact on your marketing objectives and prioritize those with the greatest potential for improvement. Also, ensure that the variables are easily measurable and can be manipulated in your experiment.

Identifying the target audience:

Determine the target audience for your experiment, considering factors such as demographics, interests, and behaviors. Your target audience should be representative of the larger population you aim to reach with your marketing efforts.

When segmenting your audience for the experiment, ensure that the groups are as similar as possible to minimize potential confounding factors.

In A/B or multivariate testing, this can be achieved through random assignment, which helps control for external influences and ensures a fair comparison between the tested variations.

Executing the experiment

With your experiment designed, it’s time to put it into action.

This involves several key considerations:

Timing and duration:

Choose the right timing and duration for your experiment based on factors such as the marketing channel, target audience, and the nature of the tested variables.

The duration of the experiment should be long enough to gather a sufficient amount of data for meaningful analysis but not so long that it negatively affects your marketing efforts or causes fatigue among your target audience.

In general, aim for a duration that allows you to reach a predetermined sample size or achieve statistical significance. This may vary depending on the specific experiment and the desired level of confidence.

Monitoring the experiment:

During the experiment, monitor its progress and performance regularly to ensure that everything is running smoothly and according to plan. This includes checking for technical issues, tracking key metrics, and watching for any unexpected patterns or trends.

If any issues arise during the experiment, address them promptly to prevent potential biases or inaccuracies in the results. Additionally, avoid making changes to the experimental design or variables during the experiment, as this can compromise the integrity of the results.

Analyzing the results

Once your experiment has concluded, it’s time to analyze the data and draw conclusions.

This involves two key aspects:

Statistical significance:

Statistical significance is a measure of the likelihood that the observed differences between the tested variations are due to the variables being tested rather than random chance. To determine statistical significance, you will need to perform a statistical test, such as a t-test or chi-squared test, depending on the nature of your data.

Generally, a result is considered statistically significant if the probability of the observed difference occurring by chance (the p-value) is less than a predetermined threshold, often set at 0.05 or 5%. This means there is a 95% confidence level that the observed difference is due to the tested variables and not random chance.

Practical significance:

While statistical significance is crucial, it’s also essential to consider the practical significance of your results. This refers to the real-world impact of the observed differences on your marketing objectives and business goals.

To assess practical significance, consider the effect size of the observed difference (e.g., the percentage increase in email open rates) and the potential return on investment (ROI) of implementing the winning variation. This will help you determine whether the experiment results are worth acting upon and inform your marketing decisions moving forward.

A systematic approach to designing growth marketing experiments helps you to design, execute, and analyze your experiment effectively, ultimately leading to better marketing outcomes and business growth.

Examples of Successful Marketing Experiments

In this section, we will explore three fictional case studies of successful marketing experiments that led to improved marketing outcomes. These examples will demonstrate the practical application of marketing experiments across different channels and provide valuable lessons that can be applied to your own marketing efforts.

Example 1: Redesigning a website for increased conversions

AcmeWidgets, an online store selling innovative widgets, noticed that its website conversion rate had plateaued.

They conducted a marketing experiment to test whether a redesigned landing page could improve conversions. They hypothesized that a more visually appealing and user-friendly design would increase conversion rates by 15%.

AcmeWidgets used A/B testing to compare their existing landing page (the control) with a new, redesigned version (the treatment). They randomly assigned website visitors to one of the two landing pages. They tracked conversions over a period of four weeks.

At the end of the experiment, AcmeWidgets found that the redesigned landing page had a conversion rate 18% higher than the control. The results were statistically significant, and the company decided to implement the new design across its entire website.

As a result, AcmeWidgets experienced a substantial increase in sales and revenue.

Example 2: Optimizing email marketing campaigns

EcoTravel, a sustainable travel agency, wanted to improve the open rates of their monthly newsletter. They hypothesized that adding a sense of urgency to the subject line would increase open rates by 10%.

To test this hypothesis, EcoTravel used A/B testing to compare two different subject lines for their newsletter:

• “Discover the world’s most beautiful eco-friendly destinations” (control)
• “Last chance to book: Explore the world’s most beautiful eco-friendly destinations” (treatment)

EcoTravel sent the newsletter to a random sample of their subscribers. Half received the control subject line, and the other half received the treatment. They then tracked the open rates for both groups over one week.

The results of the experiment showed that the treatment subject line, which included a sense of urgency, led to a 12% increase in open rates compared to the control.

Based on these findings, EcoTravel incorporated a sense of urgency in their future email subject lines to boost newsletter engagement.

Example 3: Improving social media ad performance

FitFuel, a meal delivery service for fitness enthusiasts, was looking to improve its Facebook ad campaign’s click-through rate (CTR). They hypothesized that using an image of a satisfied customer enjoying a FitFuel meal would increase CTR by 8% compared to their current ad featuring a meal image alone.

FitFuel conducted an A/B test on their Facebook ad campaign, comparing the performance of the control ad (meal image only) with the treatment ad (customer enjoying a meal). They targeted a similar audience with both ad variations and measured the CTR over two weeks. The experiment revealed that the treatment ad, featuring the customer enjoying a meal, led to a 10% increase in CTR compared to the control ad. FitFuel decided to update its

Facebook ad campaign with the new image, resulting in a more cost-effective campaign and higher return on investment.

Lessons learned from these examples

These fictional examples of successful marketing experiments highlight several key takeaways:

• Clearly defined objectives and hypotheses: In each example, the companies had specific marketing objectives and well-formulated hypotheses, which helped guide their experiments and ensure relevant and actionable results.
• Proper experimental design: Each company used the appropriate testing method for their experiment and carefully controlled variables, ensuring accurate and reliable results.
• Data-driven decision-making: The companies analyzed the data from their experiments to make informed decisions about implementing changes to their marketing strategies, ultimately leading to improved outcomes.
• Continuous improvement: These examples demonstrate that marketing experiments can improve marketing efforts continuously. By regularly conducting experiments and applying the lessons learned, businesses can optimize their marketing strategies and stay ahead of the competition.
• Relevance across channels: Marketing experiments can be applied across various marketing channels, such as website design, email campaigns, and social media advertising. Regardless of the channel, the principles of marketing experimentation remain the same, making them a valuable tool for marketers in diverse industries.

By learning from these fictional examples and applying the principles of marketing experimentation to your own marketing efforts, you can unlock valuable insights, optimize your marketing strategies, and achieve better results for your business.

Common Pitfalls of Marketing Experiments and How to Avoid Them

Conducting marketing experiments can be a powerful way to optimize your marketing strategies and drive better results.

However, it’s important to be aware of common pitfalls that can undermine the effectiveness of your experiments. In this section, we will discuss some of these pitfalls and provide tips on how to avoid them.

Insufficient sample size

An insufficient sample size can lead to unreliable results and limit the generalizability of your findings. When your sample size is too small, you run the risk of not detecting meaningful differences between the tested variations or incorrectly attributing the observed differences to random chance.

To avoid this pitfall, calculate the required sample size for your experiment based on factors such as the expected effect size, the desired level of confidence, and the type of statistical test you will use.

In general, larger sample sizes provide more reliable and accurate results but may require more resources to conduct the experiment. Consider adjusting your experimental design or testing methods to accommodate a larger sample size if necessary.

Lack of clear objectives

Your marketing experiment may not provide meaningful or actionable insights without clear objectives. Unclear objectives can lead to poorly designed experiments, irrelevant variables, or difficulty interpreting the results.

To prevent this issue, establish specific, measurable, achievable, relevant, and time-bound (SMART) objectives before starting your experiment. These objectives should guide your entire experiment, from hypothesis formulation to data analysis, and ensure that your findings are relevant and useful for your marketing efforts.

Confirmation bias

Confirmation bias occurs when you interpret the results of your experiment in a way that supports your pre-existing beliefs or expectations. This can lead to inaccurate conclusions and suboptimal marketing decisions.

To minimize confirmation bias, approach your experiments with an open mind and be willing to accept results that challenge your assumptions.

Additionally, involve multiple team members in the data analysis process to ensure diverse perspectives and reduce the risk of individual biases influencing the interpretation of the results.

Overlooking external factors

External factors, such as changes in market conditions, seasonal fluctuations, or competitor actions, can influence the results of your marketing experiment and potentially confound your findings. Ignoring these factors may lead to inaccurate conclusions about the effectiveness of your marketing strategies.

To account for external factors, carefully control for potential confounding variables during the experimental design process. This might involve using random assignment, testing during stable periods, or controlling for known external influences.

Consider running follow-up experiments or analyzing historical data to confirm your findings and rule out the impact of external factors.

Tips for avoiding these pitfalls

By being aware of these common pitfalls and following best practices, you can ensure the success of your marketing experiments and obtain valuable insights for your marketing efforts. Here are some tips to help you avoid these pitfalls:

• Plan your experiment carefully: Invest time in the planning stage to establish clear objectives, calculate an adequate sample size, and design a robust experiment that controls for potential confounding factors.
• Use a hypothesis-driven approach: Formulate a specific, testable hypothesis based on existing knowledge or data to guide your experiment and focus on the most relevant variables and outcomes.
• Monitor your experiment closely: Regularly check the progress of your experiment, address any issues that arise, and ensure that your experiment is running smoothly and according to plan.
• Analyze your data objectively: Use statistical techniques to determine the significance of your results and consider the practical implications of your findings before making marketing decisions.
• Learn from your experiments: Apply the lessons learned from your experiments to continuously improve your marketing strategies and stay ahead of the competition.

By avoiding these common pitfalls and following best practices, you can increase the effectiveness of your marketing experiments, gain valuable insights into customer preferences and behaviors, and ultimately drive better results for your business.

Building a Culture of Experimentation

To truly reap the benefits of marketing experiments, it’s essential to build a culture of experimentation within your organization. This means fostering an environment where curiosity, learning, data-driven decision-making, and collaboration are valued and encouraged.

Encouraging curiosity and learning within your organization

Cultivating curiosity and learning starts with leadership. Encourage your team to ask questions, explore new ideas, and embrace a growth mindset.

Promote ongoing learning by providing resources, such as training programs, workshops, or access to industry events, that help your team stay up-to-date with the latest marketing trends and techniques.

Create a safe environment where employees feel comfortable sharing their ideas and taking calculated risks. Emphasize the importance of learning from both successes and failures and treat every experiment as an opportunity to grow and improve.

Adopting a data-driven mindset

A data-driven mindset is crucial for successful marketing experimentation. Encourage your team to make decisions based on data rather than relying on intuition or guesswork. This means analyzing the results of your experiments objectively, using statistical techniques to determine the significance of your findings, and considering the practical implications of your results before making marketing decisions.

To foster a data-driven culture, invest in the necessary tools and technologies to collect, analyze, and visualize data effectively. Train your team on how to use these tools and interpret the data to make informed marketing decisions.

Regularly review your data-driven efforts and adjust your strategies as needed to continuously improve and optimize your marketing efforts.

Integrating experimentation into your marketing strategy

Establish a systematic approach to conducting marketing experiments to fully integrate experimentation into your marketing strategy. This might involve setting up a dedicated team or working group responsible for planning, executing, and analyzing experiments or incorporating experimentation as a standard part of your marketing processes.

Create a roadmap for your marketing experiments that outlines each project’s objectives, hypotheses, and experimental designs. Monitor the progress of your experiments and adjust your roadmap as needed based on the results and lessons learned.

Ensure that your marketing team has the necessary resources, such as time, budget, and tools, to conduct experiments effectively. Set clear expectations for the role of experimentation in your marketing efforts and emphasize its importance in driving better results and continuous improvement.

Collaborating across teams for a holistic approach

Marketing experiments often involve multiple teams within an organization, such as design, product, sales, and customer support. Encourage cross-functional collaboration to ensure a holistic approach to experimentation and leverage each team’s unique insights and expertise.

Establish clear communication channels and processes for sharing information and results from your experiments. This might involve regular meetings, shared documentation, or internal presentations to keep all stakeholders informed and engaged.

Collaboration also extends beyond your organization. Connect with other marketing professionals, industry experts, and thought leaders to learn from their experiences, share your own insights, and stay informed about the latest trends and best practices in marketing experimentation.

By building a culture of experimentation within your organization, you can unlock valuable insights, optimize your marketing strategies, and drive better results for your business.

Encourage curiosity and learning, adopt a data-driven mindset, integrate experimentation into your marketing strategy, and collaborate across teams to create a strong foundation for marketing success.

If you’re new to marketing experiments, don’t be intimidated—start small and gradually expand your efforts as your confidence grows. By embracing a curious and data-driven mindset, even small-scale experiments can lead to meaningful insights and improvements.

As you gain experience, you can tackle more complex experiments and further refine your marketing strategies.

Remember, continuous learning and improvement is the key to success in marketing experimentation. By regularly conducting experiments, analyzing the results, and applying the lessons learned, you can stay ahead of the competition and drive better results for your business.

So, take the plunge and start experimenting today—your marketing efforts will be all the better.

#ezw_tco-2 .ez-toc-title{ font-size: 120%; ; ; } #ezw_tco-2 .ez-toc-widget-container ul.ez-toc-list li.active{ background-color: #ededed; } Table of Contents

Manage your remote team with teamly. get your 100% free account today..

PC and Mac compatible

Teamly is everywhere you need it to be. Desktop download or web browser or IOS/Android app. Take your pick.

Get Teamly for FREE by clicking below.

No credit card required. completely free.

Teamly puts everything in one place, so you can start and finish projects quickly and efficiently.

Keep reading.

Building Belonging At Work – An Essential Part Of Your Workplace Culture

Building Belonging At Work – An Essential Part Of Your Workplace CultureOne of our many basic needs as humans is to feel connected to those around us, and our sense of belonging at work is no exception. We spend the majority of our day in the workplace, so when we lack a sense of belonging, …

Continue reading “Building Belonging At Work – An Essential Part Of Your Workplace Culture”

Max 9 min read

Act… Like a Leader

Act… Like a LeaderYou have probably heard the quote “leadership is an action, not a position”. This quote is widely used in leadership development programs and is credited to broadcasting executive Donald McGannon. It’s one thing to know and understand that leadership is an action, but it is another thing to put that action into …

Continue reading “Act… Like a Leader”

Max 6 min read

Book Summaries

Discovering the Power of “Buy Back Your Time” by Dan Martell – Chapter 12

Discovering the Power of “Buy Back Your Time” by Dan Martell – Chapter 12If you’re looking to boost your business productivity and improve team morale, chapter 12 is a must-read. It delves into the powerful impact of feedback and how fostering a feedback-rich culture can save your business. Let’s dive in! The Power of Feedback …

Continue reading “Discovering the Power of “Buy Back Your Time” by Dan Martell – Chapter 12”

Max 5 min read

Project Management Software Comparisons

Asana vs Wrike

Basecamp vs Slack

Smartsheet vs Airtable

Trello vs ClickUp

Monday.com vs Jira Work Management

Trello vs asana.

Get Teamly for FREE Enter your email and create your account today!

You must enter a valid email address

You must enter a valid email address!

Home Growth Marketing 21 Marketing Experiments & Test Ideas

21 Marketing Experiments & Test Ideas

Marketing works best when it’s led by evidence.

In other words, a series of marketing experiments that progressively build upon one another through iterative testing. 

This scientific approach to marketing works because it improves with every success or failure and plays into the strengths of marketing; An abundance of data, a clear goal, and a disproportionate amount of ideas to resources.  

One of the toughest parts of implementing a marketing experimentation program is getting the process right. Many teams flit from one shiny A/B test to another, with little accountability or long-term strategy. 

In this post, we’ll focus on two essential parts of marketing experimentation; Keeping a healthy backlog of ideas and experiments to test, as well as how to prioritise and track the outcomes once complete.

What are we looking at in this article?

The focus of this article is to suggest a range of marketing experiment and test ideas that will improve the performance of your website and campaigns. You should be able to take these ideas, apply them to your marketing strategies and work your way to better results.

More importantly, though, I hope these ideas provide enough examples for you to go away and find your own experiments, too.

So we’re going to start by explaining why marketing experiments are important and look at how you can run experiments successfully. Then, we’ll get into the experiment and test ideas that put these concepts into actionable examples that you can try for yourself.

As you’ll see from the experiment and test ideas later, some of these focus on very specific elements, such as landing page navigation, while others are broader concepts, like optimising pricing pages, that you’ll need to play around with more for yourself.

Regardless, each idea we cover in this article will look at specific concepts and insights that you can learn from and apply to your experiments.

Why are marketing experiments important?

Marketing experiments generally have one of two aims: to prove an existing theory or try something new entirely. In other words, they provide a means of knowing which marketing strategies work (or don’t) and a measurable approach to trying out new ideas – both of which are incredibly important.

In today’s data-driven age, marketers don’t have to put their faith in concepts and theories. We can prove the value of marketing campaigns, design choices, growth strategies and key business decisions.

As Google puts it, in a guide entitled Proving Marketing Impact (PDF) , “one of the most serious challenges in marketing is identifying the true impact of a given marketing spend change.”

“The ripple effect is that marketers must function as scientists, conducting experiments when it comes to allocating budgets—whether by adapting the media mix, trying out different forms of creative, or exploring new marketing channels altogether. By measuring the full value of digital media investments through controlled marketing experiments, we can prove what is effective and what is not.” Proving Marketing Impact: 3 Keys to Doing It Right , Think with Google

Without a refined experiment and testing system in place, you’re always shooting in the dark with your marketing actions. If you can’t prove the effectiveness of your campaigns, then you can’t know they’re profitable and business growth is going to suffer.

Essentially, you’re gambling with your marketing investment and the odds are stacked against you.

On the other hand, a data-driven model of testing and experimentation puts you in control of your marketing spend. It allows you to prove the ROI of marketing actions, optimise strategies to improve performance and make intelligent marketing decisions.

In practical terms, this allows you to:

• Prove the effectiveness of marketing campaigns
• Stop wasting money on ineffective strategies
• Test new design ideas
• Optimise campaigns and pages to maximise performance
• Try new marketing strategies
• Learn from previous experiments
• Make smarter business decisions
• Identify & stop expensive mistakes
• Innovate new ideas

A good example of this would be our journey with web form optimisation, which not only led us to 700+% higher conversion rates but also led us to innovate a multi-step form concept that nobody else was using at the time.

By innovating this idea (and proving that it worked), we reaped the benefits of the most effective web forms around while everyone else was stuck with lower conversion rates. And this is the thing about marketing innovation: by getting there first, you capitalise on the benefits while everyone else is playing catch-up.

Marketing experimentation is the only path toward true innovation and climbing your way to the top of your industry.

The secrets of a successful marketing experiment

In this article for HubSpot , Kayla Carmicheal runs through the five key steps you should follow for any successful marketing experiment:

• Make a hypothesis: Whether you’re proving an existing theory or going into an experiment hoping to prove that a new strategy is effective, the first step is always to define a specific and measurable hypothesis.
• Collect research: This should include audience research, reaffirming (possibly even modifying) your hypothesis and identifying challenges you’ll face throughout your experiment – e.g.: variables to consider, data attribution issues, etc.
• Define your measurement metrics: With your research complete, it’s time to confirm and define the measurement metrics that will prove your outcome and make sure you have the necessary tools to collect that data.
• Run your experiment: By now, you should have everything in place to run your experiment until it reaches statistical significance and your hypothesis is either proven right or wrong.
• Analyse the results: With your experiment complete, the final stage is to analyse the results and extract meaningful insights from your tests.

That’s a pretty good assessment of the process you should follow and you can find more details about each step in Kayla Carmicheal’s article – note: I’ve paraphrased each step above with my own explanation.

This process is applicable to every marketing experiment and test you could think of but it doesn’t guarantee success. To ensure your marketing experiments deliver valuable insights that result in better marketing and business decisions, there are a few key principles you need to master:

• Choosing the right experiments/tests: This starts with prioritising hypotheses and opportunities with the greatest potential benefit.
• Achieving reliable results: The key to this is mitigating variables that could skew results and running experiments long enough to achieve statistical significance.
• Learn from your results: Not only should you learn valuable lessons from individual experiments but also be building up a history of experience and insights from your ongoing, collective testing strategy.

As I explain in our Landing Page Optimisation: 101 Tips, Strategies & Examples article, the art of deciding what to test and experiment is understanding what truly impacts user behaviour.

“One of the worst A/B testing mistakes you can make is testing every minute detail (button colours, font styles, single words). Stick to testing meaningful variations like two different landing pages or one with multi-step forms and one without. These are factors that genuinely impact the consumer experience and whether people do business with you.”

Running marketing experiments require time and a certain amount of financial investment. So, as with all marketing strategies, you need to make sure that your tests generate a positive ROI .

Start by testing the strategies and elements on your page that should deliver the strongest positive outcome, use your research to reinforce your decisions and apply what you learn to future experiments.

If you’re short of ideas, don’t worry. We’ve got you covered through the remainder of this article.

Website test & experiment ideas

#1: optimise loading times.

It still amazes me that the average loading time of websites and landing pages remain as slow as they are in 2020. It’s not like the impact of page speed upon conversions and just about every user metric that matters is a secret – it’s been documented time and again.

Yet, according to research from Unbounce in 2019, only 3% of marketers listed improving page speed as one of their top priorities.

Unbelievably, page speed sat at the bottom of the priority list for marketers in the report while A/B testing or optimising pages came out on top. You have to wonder, what on Earth are marketers optimising if they’re not interested in making their pages faster.

Let’s go back to the key principle we talked about earlier: start with the factors that will have the greatest positive impact.

Page speed is clearly one of these so it earns the first place in our list. If you’re not sure how to approach page speed optimisation, you can find out how we improved our loading times by 70.39% with roughly 40 minutes’ worth of work .

Optimising page speed should be one of the most profitable experiments you run, too. There are plenty of free tools available for measuring loading times and pinpointing specific issues – such as GTmetrix and Google’s PageSpeed Insights tool.

Here’s a quick summary of the steps we took in the article linked above:

• Use a content delivery network (CDN)
• Use WP Engine for hosting WordPress sites
• Use a web caching plugin/service
• Add Expires headers
• Use a fast theme for WordPress or other CMS platforms
• Compress your images
• Clean up your database
• Compress your website with Gzip
• Fix your broken links
• Reduce the number of redirects
• Minify your CSS & JS files
• Replace PHP with static HTML where possible
• Link to stylesheets – don’t use @import
• Turn off pingbacks and trackbacks
• Enable Keep-Alive
• Specify image dimensions
• Specify a character set in HTTP headers
• Put CSS at the top and JS at the bottom
• Disable hotlinking of images
• Switch off all plugins you don’t use
• Minimise round trip times (RTTs)
• Use CSS Sprites

You can find more details about each of these steps in our page speed optimisation article . Collectively, these make a significant impact on loading times and, in turn, all of your marketing KPIs should benefit.

#2: Prioritise hero sections

The hero section is the main feature of every page (and email) that matters. This is the first view users see when they click through to your website, landing page or open up one of your emails and this is the space where your primary message makes its first impression.

In other words, this is one of the most important features of any page with a conversion goal.

Here at Venture Harbour, we’re constantly testing and experimenting with hero section designs to increase conversion rates, engagement and other crucial KPIs. If there’s one theme that’s been present throughout the years, it’s that simple design and clear messaging are key.

We’ve covered this before in our 3 Homepage Design Tweaks That Increased Our Conversion Rates By 500% article. Here’s a snippet that’s particularly relevant:

“As soon as we lifted the CTA from our hero section and replaced it with a more compelling value proposition, we saw email sign-ups from our homepage increase from 11% to 17% (a 55% increase). That’s pretty impressive when all we really did was remove something designed to convert and rethink our message.”

We’ve consistently found that hero sections without CTAs perform better than those with them – provided our message delivers enough of a value proposition to encourage users to scroll beyond the fold.

This may come as a surprise but multiple tests over the years have confirmed this for us, which is a perfect example of learning from previous tests and applying insights to future experiments.

#3: Remove navigation from landing pages

CTAs in hero sections aren’t the only common design convention that you can prove wrong through testing. In fact, you’ll find some of the most prevalent design patterns are surprisingly damaging for website performance once your testing process is up and running.

Another good example is the use of navigation on landing pages. If you stick to the Goody Two-Shoes guide to UX design, every page should have accessible navigation so that users can move to other parts of your website.

In the real world, though, we’ve got marketing objectives to think about and the last thing you want someone to do on your landing page is click through to another part of your site – away from your CTAs.

Over time, an increasing number of businesses have found that removing the header navigation from landing pages is the way to go.

There are genuine UX benefits for the end user, too. By removing the option of navigation, you increase the clarity of your landing page and improve user focus on your key message while reducing the risk of distractions and decision fatigue – all of which is good for your conversion goals and the short attention span of your typical visitor.

#4: Optimise your sales funnel

Optimising your sales funnel is one of the biggest tasks you can take on in marketing experimentation and conversion optimisation. There are no shortcuts to doing this (none that are worth taking, anyway) and this is something you’ll have to continue doing for the foreseeable future.

For an in-depth look at funnel optimisation, we’ve got two guides that you can read through and save for reference:

• Funnel Optimisation 101: 5 Steps to Fixing a Leaky Marketing Funnel
• Marketing Funnel Strategies: 5 Steps to Increasing Sales in 2020

Your sales funnel is designed to capture leads at every stage of the consumer journey and act as a framework for nurturing prospects along the buying process – from first-time visitor to paying customer.

The aim of optimising your sales funnel is to increase the percentage of leads that go on to make the initial purchase and the percentage of existing customers who go on to make further purchases.

One of your key objectives is to identify where leads are dropping out of your sales funnel and put strategies in place to keep these prospects moving along the buying process. This is where our guide to fixing a leaky marketing funnel will come in handy .

Content & messaging experiment ideas

#5: test different messages.

When it comes to convincing users to take profitable action, it’s your content and web copy that has the biggest influence. Ultimately, it’s your marketing messages that decide whether you convert users into leads or customers.

So the majority of your experiments and testing should be aimed at pinpointing the right message for campaigns, ads, pages and emails.

Focus on the key selling points and benefits of your offer, as well as the pain points your target audiences experience (and the solutions you provide). Be specific. Don’t try and squeeze every point into one message. Test different, highly-focused messages that hone in on specific selling points and run with the variation that performs most effectively.

If you can only deliver one message to every user, you need to find the message that’s most effective in the highest percentage of scenarios.

You can also test personalisation features to deliver different messages to users, based on their individual interests. For example, Unbounce offers a great feature called Dynamic Text Replacement , which matches the headline of your landing page to the search terms a user types into Google prior to clicking through.

This allows you to create and test multiple messages for different buyer personas and deliver them with accuracy. Instead of showing one message to all, you can pinpoint the unique pain points of different user types and deliver a more compelling offer.

#6: Experiment with cognitive biases

The secret of effective marketing messages is understanding the psychology of consumers. Even the biggest, baddest CEO is a human being flawed with cognitive biases that affect their decision-making capabilities.

The most successful marketing and advertising campaigns use these biases to create incentive, play with emotions and change the perception of potential buyers.

For an in-depth look at how you can use cognitive biases to create killer marketing messages, take a look at our 9 Cognitive Biases That Influence Buyer Decisions article where we explore the following:

• Confirmation bias : Where people seek, interpret and remember information in a way that confirms their existing ideas.
• Loss aversion : Which describes how we fear loss considerably more than we value gaining something of the same worth – e.g.: we’re more frustrated by losing £10 than we are happy to gain £10.
• Anchoring bias : Where people place more significance on the first piece of information they receive.
• The bandwagon effect : Which explains why people queue up for days to buy an iPhone, think they’re getting good deals on Black Friday and social media trends even exist.
• The mere exposure effect : The reason people are more likely to buy from brands they know well.
• The endowment effect : Explains why people place inflated value on items they own – or believe they own.
• Sunk cost bias : Where people continue to do something because they’ve already invested time, effort or money into it and fear this investment will be lost – even if they’re better off calling it quits now.
• The halo effect : How our first impressions influence the way we interpret further information.
• The serial position effect : The reason people interpret the first and last pieces of information on a list as the most important and remember them more clearly.

You can click on the bold link text of each cognitive bias listed above and it will take you to an article looking at how you can use each of them to create more effective marketing messages.

#7: Capture leads from content

Marketers and brands spend an immeasurable time creating content but how much of it actually generates leads? Think of all those blog posts just sitting there on your website and the time/money it took to produce them – are they paying their way?

If the answer is no, then you’ve got a content marketing ROI problem because everything you publish should contribute to your key marketing objectives: generating leads, sales and profit.

One of the most popular ways to generate leads from web content is to use exit-intent pop-ups and they’ve got their merits. In fact, we take a look at the pros and cons of using them in our Do Exit-Intent Popups Actually Increase Conversions? article.

You can also use live chat widgets and chatbots to prompt user action or request permission to send users notifications – two other strategies with their own benefits and drawbacks.

After testing out various solutions, our current approach is to add dynamic CTAs throughout our blog posts, promoting our most recent ventures.

The key to making this strategy work is to dynamically insert our CTAs in every blog post and continue to update our old content so that it continues to generate traffic, get CTAs in front of eyes and maintain a return on our content investment.

#8: Experiment with content types

As the internet and consumer habits evolve, different types of content assume new roles. Just about every content marketing trends article for the past five years has been talking about the rise of video content and we’ve seen this with the transformation of Instagram from an image-only network to one dominated by video clips.

In the past few years, podcasts have been one of the biggest growth opportunities for content marketers, journalists, celebrities and anyone with a voice.

Video game streaming is now a multibillion dollar industry where people pay to watch strangers play anything from the latest Fifa to retro console games.

Meanwhile, the infographics craze is over and some of the worst content trends like clickbait and listicles are dying a slow death.

Some of these trends are here to stay but it’s very difficult for new publishers to enter the podcast race or new gamers build a huge following on Twitch. You have to react quickly to content trends and build your audience before they’re soaked up by everyone else – just take a look at how hard it is for YouTubers starting out now.

Experiment with content types, find out what works with your target audiences and make your mark before opportunities are saturated by your rivals.

CRO test & experiment ideas

#9: test your ctas.

We’ve written plenty about testing and optimising calls to action (CTAs) on the Venture Harbour blog. Here’s a selection of some articles that will help you run the best tests and maximise results:

• 19 Compelling Call to Action Examples You Can Steal For Yourself
• Call to Action Best Practices: 15 Tips to Write Irresistible CTAs
• CTA Psychology: 11 Principles That Make Users Click

That should give you plenty of guidance on how to create effective CTAs, optimise them to improve results and an understanding of the psychology behind irresistible CTAs.

As mentioned earlier, the most important principle is to remember that the message in your CTA has the greatest influence upon users. If your CTA copy is compelling, you stand the best chance of convincing users to take action.

There are plenty of design concepts that are important, too, of course:

• Text size and weights
• Colour (again, mostly for contrast

Those are the fundamental design concepts you should focus on while refining the message in your CTA to make it as convincing as possible. Try not to get lost in testing button colours and font styles because they don’t have a great influence.

As always, it comes down to testing the details that have maximum impact.

In terms of design, it’s mainly about bold, full-screen CTAs with plenty of contrast that grabs attention and makes your message jump out of the page. Then, it’s all about the message itself and how successfully it compels users to take action.

One final thing to keep in mind is that the content before your CTAs is crucially important too (higher up the page and possibly on previous pages, too). This is the content that primes users for the message in your CTAs and warms them up for taking action – the CTA itself should give them the final push.

#10: Test multi-step forms

After almost a decade of marketing experiments and testing, one of our biggest breakthroughs to date has been the discovery of multi-step forms. You find out how we’ve managed to increase conversions by up to +743% over the course of several.

If we had simply followed the usual form design best practices, we never would have stumbled across the multi-step format. Instead, we would have simply tried to make our forms as short as possible.

However, we tested each and every best practice we knew (or thought we did) and consistently found that shorter forms were being outperformed by longer, more optimised alternatives – and it turned out we weren’t alone .

Generally speaking, longer forms convert more users, as long as you’re able to increase incentive sufficiently enough to justify the length. However, we were still finding conversion rates were lower than we wanted so we started pinpointing the real UX issues of completing forms:

• Typing fatigue
• Mobile optimisation
• Irrelevant fields
• Completion time

We perceive longer forms as being a UX issue because traditional forms are a pain to complete. It can take minutes for users to fill out a few fields successfully due to the amount of typing required and the workload only increases on mobile where selecting fields and typing is even more frustrating.

It’s only natural that, when users see a long form, they want to run for the hills.

So we decided to remove this perception of workload by only showing the first question in a multi-step format. Now, the psychological barrier preventing users from starting the form process disappears.

Next, we decided to solve the typing problem by switching from text inputs to selectable image buttons wherever possible. Not only does this remove typing fatigue and speed up completion times, it also allows us to guide users through predefined paths and segment leads as they complete our forms.

Taking this concept one step further, we began to use a rule-based technology called conditional logic that filters out form questions based on user inputs so they only see questions relevant to the information they’ve already provided.

For example, if they tell us that they’re a web design agency, they only see questions relevant to web design projects. This shortens the form experience, reduces fatigue and increases the relevance of every form experience.

You can find out more about the science behind our high-converting multi-step form here .

#11: Simplify onboarding

Web forms play a key role in the onboarding process of every customer and optimising to simplify the sign-up process is a high-impact experiment worth embarking on.

Expanding beyond this concept, you should consider running a series of experiments and tests looking to simplify the entire onboarding process with the aim of reducing the number of leads that slip away during these final hurdles.

You’ve done all the hard work and it sucks to lose them at this point.

Not long ago, we published an article looking at 10 CRO Case Studies You Can Replicate to Increase Conversions and one of the case studies listed provides a perfect example of simplifying the onboarding process.

By using conversion optimisation platform Hotjar , DashThis turned 50% more free trial users into paying customers by identifying onboarding issues and simplifying the process for users.

You can read their case study here .

After using Hotjar for 10 months, the company increased onboarding completion by 50%, increased customer satisfaction by 140% and created new features based on customer feedback to improve its product – not a bad set of results, by any means.

“After having analysed where users encountered roadblocks during the onboarding process, DashThis’ UX team discovered that users simply didn’t know where to click in order to add integrations. Buttons were not displayed in bold enough colours and on some smaller screen resolutions, they were completely hidden at the bottom of the page… To fix these problems, the team modified the layout of the list, added a search bar, and incorporated pop-ups, videos, and different types of content to guide the user step-by-step. The buttons were also modified to be bigger and bolder, and to accommodate those with a smaller screen resolution, buttons were added higher in the page.”

Once again, we see a business running a broad marketing experiment to identify and address specific issues at one of the most important stages of the consumer journey.

This is an experiment that promises high returns on your testing investment and makes a genuine impact upon the customer experience.

#12: Test on human beings

Sometimes, there’s no substitute for testing on real-life human beings but we’re not talking about clinical trials or anything scary like that – just some good, old-fashioned user testing.

The kind of user testing Evernote ran to increase user retention by 15% across all devices (PDF) .

Using a platform called UserTesting , Evernote connected with real-world users. While the company had run its own user testing programs in the past, it found them too time and cost-inefficient – a common problem for companies trying to run marketing experiments and CRO campaigns.

Thankfully, there are plenty of platforms like UserTesting available these days that make it easier for companies to test their websites and software products on real-world users.

“In addition to hearing the study participants as they narrate through their actions and decisions, Evernote product managers and designers are able to watch where the testers’ hands are physically tapping, swiping, and even resting. This was especially helpful on Android since multiple devices run on the platform. Because the device type had an impact on the experience, the team needed to be able to identify and fix ergonomics issues before new products were released to the public.”

By identifying and addressing these issues, Evernote increased user retention by 15%, which is the basis of the company’s entire business model of lead generation and upselling through prolonged engagement.

Lead generation experiment & test ideas

#13: test automated webinars.

Earlier, we talked about testing different content formats and this is something we take seriously at Venture Harbour. Over the years, we’ve heard time and again that B2B marketers prefer high-quality webinars over all other content formats and this is something our own testing supports.

The problem with webinar marketing is that it’s incredibly time consuming and it can be very expensive. Manually organising and recording events on a periodic basis requires a lot of resources, which doesn’t really fit with our company ethos of efficiency and automated strategies.

So we decided to experiment with methods of automating our own webinar strategy to tap into the effectiveness of this content format without excessive manual workload.

It was a bumpy process in the beginning, as we’d never automated a webinar strategy before and we didn’t have any real data to work with. However, we started running tests from day one and, within a year, we achieved an attendance rate of 75.62% – more than double the industry average of 46%.

The best part of this strategy is that it’s fully automated so, once you’ve got it up and running, it does all of the hard work for you.

#14: Find a way to offer free value

This strategy predates digital marketing and it’s so common that it borders on cliche but it remains one of the most effective tactics. The key is finding a way to offer perceived value that’s relatively low-cost to your brand but acts as a gateway to something of much higher value.

My favourite example of this is software companies creating free tools and using them to reel in potential customers. Ahrefs does this with its free Backlink Checker that allows anyone to type in a domain or URL and receive a backlink report.

The report in itself is genuinely useful for analysing a single domain or URL but you quickly realise it’s worth looking at the paid version of Ahrefs to get full access to its data and run multiple reports.

The free tool shows you what the platform is capable of and then leaves you craving the full features. Suddenly, that monthly price tag doesn’t seem so unreasonable and signing up for a free trial is only a few clicks away.

We see a similar approach from Crazy Egg , which offers a free heatmap to reel in the leads. Again, you can type in your website URL for a free report. However, the company takes a slightly more aggressive approach than Ahrefs and asks users to create a free account before receiving their report.

We’ve used the same strategy ourselves at Venture Harbour with free tools, such as Marketing Automation Insider , which marketers and business owners can use to find the best marketing software for their needs.

More recently, we’ve published a marketing ROI calculator that you can use and embed on your website.

#15: Test new ad networks

For years, the paid advertising landscape has been dominated by Google and Facebook but things are slowly changing. The rise of social advertising helped initiate change but growth has stalled in that area.

TikTok is writing a lot of headlines as the latest social trend and potential future advertising giant but we’ve seen flash-pan networks crash and burn countless times before.

We’re seeing more robust developments in the retail advertising space where Google is fighting it out with the likes of Amazon and eBay – two giant names without a long history in paid advertising.

Of course, Amazon and eBay are appealing to retailers and e-commerce entrepreneurs but the platforms are pioneering concepts for industries beyond the limits of online retail.

For example, B2B marketers should pay attention to the new advertising network available on eBay, which allows you to target eBay sellers directly – from major retail brands to solopreneurs selling on the platform.

Select your targeting options and you can reach eBay sellers to promote products and services that will help them grow their business – anything from website builders and accounting software to insurance and delivery services.

With competition increasing in the paid advertising space, marketers need to keep up with developments and be ready to test new ad networks as they emerge.

#16: Experiment with personalisation

Personalisation has been one of the biggest trends in marketing for half a decade or so but it doesn’t come easily. Delivering personalised experiences can be especially tricky in the age of GDPR and increased awareness about data privacy but plenty of brands are making it work.

If you’re struggling with data regulations, you should take a look at our 10 GDPR-Friendly Ways to Personalise Your Sales Funnel article for some ideas.

One of the brands we look at in that article is Stitch Fix, which has created an entirely personalised experience across the customer journey. The great thing about this approach is that the experience itself justifies the use of user data and customers understand that the more data they provide, the better their experience becomes.

Studies have shown that users are willing to exchange data for the right incentive (it comes back to perceived value). The challenge is designing GDPR-compliant consent in a way that doesn’t cripple your conversion rates.

Here’s an article that might help:

• 5 Ways to Get Form Consent Without Killing Conversions

Lead nurturing & customer retention experiment ideas

#17: optimise your pricing pages.

Going back to the principle of optimising elements that have the greatest impact, your pricing pages certainly fit into this category. This is the page that’s tasked with convincing users that your product is worth the asking price and alleviating any remaining purchase anxiety.

So, of course, making it clear which features users get in return for their money is crucial and it often helps to offer some kind of free plan or trial to get them on board. Money-back guarantees and other financial reassurances can also help them take the plunge, especially if you create a sense of them having nothing to lose.

On a more psychological level, one of the most common tricks you’ll see on pricing pages is the use of anchoring – one of the cognitive biases we discussed earlier. By placing the more expensive plans to the left or top of the screen, users instinctively use this as a gauge for the following prices, which all seem very reasonable in comparison.

Lead with the $0 free plan and everything that follows seems expensive.

On the Leadformly pricing page, we’re currently testing an interactive pricing page where users select how many leads they generate on a monthly basis to determine the price.

Our approach is to demonstrate how much potential customers are paying for each individual lead to demonstrate the fact that Leadformly more than pays for itself.

We also combine some classic reassurances and guarantees with a free trial to reduce purchase anxiety and create that sense of having nothing to lose, as mentioned above.

#18: Increase customisability / personalised experiences

We talked about personalisation in the previous section and one of the most effective strategies when it comes to customer retention is encouraging customisation.

This is particularly true for software products and account-based services where individualism is an asset. My favourite example of this is Spotify, which encourages you to create playlists of your favourite songs while its AI algorithms suggest new types of music you might enjoy, based on your listening habits.

By the time you’ve created a few playlists, you’re locked into the platform because you feel like you’re going to lose the time it took to create them (sunk cost bias) and those recommendations just keep on getting better.

The same principle can apply to any software product – all you need to do is build some kind of database that individual users don’t want to lose. It could be analytics reports, performance data, precious memories or anything of perceived value.

This can work for account-based retailers, too, whether you’re selling online or offline. Personalised offers, product recommendations, loyalty rewards and anything you can do to individualise the customer journey is an effective retention strategy.

#19: Test customer retention campaigns

Speaking of which, another high-impact priority for your marketing experiments is always going to be customer retention campaigns. The experiment is to find what works for your customers and keeps them coming back for more.

Email marketing is generally the easiest strategy to automate and this is the perfect platform for a low-input, high-output retention strategy.

#20: Speed up customer service

Customer service is another key component of retention. You have to keep customers happy if you want them to come back for more, even if the initial experience isn’t quite what they expected.

Studies show that customers are generally understanding reasonable issues if they’re dealt with in a timely manner.

Speed matters when it comes to dealing with complaints and this is one situation where automated emails can be frustrating. Chatbots and live chat widgets are perfect for providing an instant response to basic customer problems, buying you essential time to deal with tickets that need handling by human support members.

By automating instant responses, the frustration of one-way email exchanges disappears and you can use bots as an interface for prioritising cases that need the most attention.

#21: Automate re-engagement email campaigns

A common problem you’ll experience with customer retention is that a certain percentage simply stop engaging with your brand. Maybe they no longer use your software product or they’re not visiting your website as often as they used to, even if they haven’t cancelled their subscription or account.

Technically, you’ve still got this customer on board but they’re inactive and in danger of churning at some point.

This is where re-engagement campaigns are crucial as a strategy for reigniting the flame with your precious customers. You can automate these campaigns, too, so that they trigger after a certain period of inactivity, such as the Grammarly email above, which is sent to users who haven’t used its free app recently.

You don’t want to be too pushy with re-engagement campaigns but this is the magic of testing: you can establish the perfect time frames for sending out re-engagement emails to give your customers a gentle nudge without annoying them enough to unsubscribe from your email list.

Test & learn your way to success

The key theme throughout this article has been choosing experiments that are likely to yield high-impact results. If there’s one takeaway from everything we’ve looked at, make sure it’s this concept of high-impact testing.

Hopefully, the examples we’ve covered give you plenty of ideas and help you develop a sense of where the greatest opportunities are.

Just remember that marketing experimentation is an ongoing process and you should learn from the results, even if you don’t get the expected outcome. Invest some time in getting the best insights from your results and truly understanding what they mean so you can apply these findings to future marketing and business decisions.

Aaron Brooks is a copywriter & digital strategist specialising in helping agencies & software companies find their voice in a crowded space.

More from Aaron

You may also like.

Growth Marketing Software: 10 Essential Tools for Growth Marketers

Our team curated ten of our favourite tools for managing, planning and executing marketing experiments in 2022.

30+ Growth Marketing Strategy Examples

Growth marketing is the agile, evidence-based alternative to slow, opinion-based traditional marketing. But what does it look like in practice?

What Is Predictive Analytics & Can It Give You an Advantage?

Data empowers us with knowledge of what did or didn’t work in the past. In hindsight, we all have 20/20 vision. But what if you had 20/20 vision, not in…

TrueNorth Serene Automation Insider StackUp

Popular Guides

Email Marketing Software Marketing Automation Software ActiveCampaign Review Webinar Software

About Blog Careers Contact

Privacy Policy   |  Cookie Policy    | Editorial Policy  | Terms of Use

© 2024 Venture Harbour. Ltd is a company registered in England and Wales. Company No. 8291791. VAT No. 290356105. Registered office: Lytchett House, 13 Freeland Park, Wareham Road, Poole, Dorset, BH16 6FA

• Growth marketing articles
• Marketing ops articles
• Marketing strategy articles
• Marketing acquisition articles
• CRO articles
• Content marketing articles
• Affiliate marketing articles
• Lead nurturing articles
• A/B testing tools
• CRMs (with automation)
• Email marketing software
• Growth Marketing Software
• Landing page builders
• Lead generation tools
• Marketing calendar software
• Transactional email services
• Webinar software

Newly Launched - AI Presentation Maker

Researched by Consultants from Top-Tier Management Companies

AI PPT Maker

Powerpoint Templates

Icon Bundle

Kpi Dashboard

Professional

Business Plans

Swot Analysis

Gantt Chart

Business Proposal

Marketing Plan

Project Management

Business Case

Business Model

Cyber Security

Business PPT

Digital Marketing

Digital Transformation

Human Resources

Product Management

Artificial Intelligence

Company Profile

Acknowledgement PPT

PPT Presentation

Reports Brochures

One Page Pitch

Interview PPT

All Categories

Top 10 Marketing Experiment Templates with Samples and Examples

Yajur Sharma

A marketing experiment is the systematic testing of multiple marketing strategies, methods, or aspects to acquire insights about customer behavior, preferences, and the efficacy of approaches. It enables organizations to make better marketing decisions by offering proof of what works and what doesn't.

Picture this: You’re a fashion designer preparing for a new clothing line launch. You have two collection concepts: One influenced by nature, with earthy tones and organic materials, and another with solid and vibrant colors and edgy patterns. Unsure which way will appeal to your target market, you perform a marketing experiment. You create two teaser campaigns, each showcasing one of the concepts, and promote them on social media platforms. As interaction and reviews come in, it becomes clear that the nature-inspired collection creates greater interest and shares than the flashy, vibrant one. This insight leads you to focus on developing the nature-inspired collection for the launch, ensuring it aligns with your audience's preferences. This helps identify the most effective approach and guide your creative direction for maximum appeal and success.

Experiment and Excel

Marketing experiments help organizations optimize resources and maximize return on investment. Businesses can identify ways to reach and engage their target audience by testing multiple factors such as messaging, pictures, targeting, price, and channels. This optimization results in more efficient marketing efforts, improved conversion rates, and more revenue.

Businesses encouraging innovation and risk-taking can uncover new possibilities and outperform competitors in an ever-evolving economy.

Experimentation also enables businesses to adapt to shifting customer trends and tastes, allowing them to remain relevant and responsive to market dynamics.

While marketing experiments provide multiple benefits, they also present certain challenges businesses might confront. Analyzing data from marketing experiments can prove challenging, particularly when working with several factors or channels. 

You might struggle to evaluate reports effectively and generate useful insights from data, especially if you lack statistical analysis or data science competence.  

SlideTeam has put together these Top 10 Marketing Experiment Templates designed to address pain points related to marketing experiments. These templates are 100% editable and customizable. The content-ready nature of the slides provides you with much-needed head-start for your marketing experiments.  

Let’s explore the templates now! 

Template 1: Experimental Marketing Recap Proposal 

Designing a marketing strategy is an iterative process where they must add and subtract specific components or improve an existing one. One of the keys to success in the marketing recap is to compare marketing campaigns of previous years. Using this PowerPoint Template bundle in 34 slides ensures your clients have a long-term association with a promise to permute the marketing skills and services into sales conversions. Manage vendor marketing opportunities associated with the event and provide service with a better-customized marketing plan to surpass last year’s event results. This PPT Template helps your clients understand that you have established an experimental marketing recap of traffic results, progression status of attendees, pipeline measurement, leads by category, results, and flashbacks of past events. It illustrates an edge over the competitors by highlighting the company's background, vision, focus, mission, marketing team, and client testimonials.

CLICK HERE TO DOWNLOAD

Template 2: Marketing Experiment PowerPoint Slide Bundle

This PowerPoint Template on Marketing Experiment guides marketers through the process of conducting successful experiments to optimize their campaigns. Explore effective Marketing experiment strategies to test and refine your marketing initiatives. With this bundle, discover the step-by-step process to conduct a marketing experiment, from designing hypotheses to analyzing results. Dive into the specifics of running an email campaign experiment, precisely reaching your target audience and measuring its impact. Leverage insightful statistics and track your experiment's performance with a robust KPI Dashboard. This template gives valuable insights into your product performance and makes data-backed decisions to enhance your marketing efforts. 

Template 3: Purpose of Experimental Marketing Recap

Experimental Marketing Recap is an essential tool companies use to adapt and modify their marketing strategy in a changing market scenario. This PowerPoint Slide illustrates the effectiveness of experimental marketing initiatives. It measures the effectiveness, engagement, and customer reactions to new strategies, goods, or campaigns. Benefits include improving future marketing efforts by identifying effective methods, learning about consumer preferences, and receiving real-time feedback. It also helps to optimize budget allocation, improve brand impression, and promote innovation within the marketing plan. 

Template 4: Experimental Marketing Recap: Progression Status of Attendees

This PowerPoint Slide gives an overview of the progress of persons involved in experimental marketing activities. It monitors how participants interact with promotional events or campaigns, tracking their progress from first engagement to conversion or intended outcome. It evaluates participants' replies, behaviors, and conversions to assist marketing teams in analyzing the success of their plans. Analyzing progression status provides marketers with information regarding audience interests, preferences, and conversion routes, allowing them to adjust future ads for maximum effect and profitability. 

Template 5: Experimental Marketing Recap: Results and Flashback of Last Event

Experimental marketing involves creating unique interactions that engage consumers with a business. The summary of events emphasizes results and revisits essential moments, highlighting accomplishments and opportunities for development. Companies may improve their strategy by analyzing previous results and customer feedback, maximizing resources, and affecting efforts. It is also about learning from the past to change the future of customer involvement. This PowerPoint Slide includes an image with attractive icons to display information for maximum retention.

Template 6: Key Steps Involved in Effective Experiment Marketing

This PowerPoint Slide showcases a procedure for performing marketing experiments to measure successful strategies. Effective experimental marketing generates engaging experiences related to brands, which increase customer engagement and loyalty. The procedure includes brainstorming for experiments to be conducted, making hypotheses, determining measurable results, running experiments, and analyzing results. It also includes a visually appealing graph to display relevant information for maximum retention. Companies refine strategies by analyzing results, optimizing resources for effective future advertising, and eventually improving brand performance. 

Template 7: Marketing Experiment to Test Target Audience

Marketing experiments are the process of testing various techniques or methods to understand and target specific consumers properly. Companies may get helpful information about customer preferences, behaviors, and responses by conducting experiments. This PowerPoint Slide represents an experiment to determine the target audience for the product and its promotion. It includes various components such as objectives, demographics, conversion rate, and experiment results. This builds a stronger customer relationship, increases ROI, and leads to better-informed decision-making.

Template 8: Email Campaign Marketing Experiment to Increase Customer Conversion

Email campaign marketing is the process of sending targeted emails to a particular set of people sell products and services or engage consumers. It helps develop email marketing tactics to produce more personalized and interesting content, improving open rates, click-through rates, and conversion rates. Analyzing outcomes of these trials gives significant insights into consumer preferences and behaviors, allowing businesses to constantly optimize their email campaigns for improved performance, resulting in greater sales and revenue growth. This PowerPoint Slide displays an email campaign experiment to identify the most successful one. It includes comparing campaigns on parameters, such as open rate, click-through rate, click-to-open rate, unsubscribe rate, conversion rate, and spam rate.

Template 9: Strategies to Conduct Successful Marketing Experiment

Strategies for practical marketing experiments provide several advantages. They enable businesses to test concepts, improve strategies, and maximize resources. By collecting data on consumer behavior and intentions, businesses may make educated decisions, increase customer engagement, and ultimately improve the success of their marketing activities. This PowerPoint Slide showcases best practices for performing marketing experiments. It includes a graph explaining tactics, such as creating hypotheses, designing marketing experiments, gathering research, conducting A/B testing, and prioritizing suitable metrics.

Template 10: Steps to Perform Marketing Experiment for Website

Marketing experiments on websites are essential for improving user experience, content success, and conversion rates. This PowerPoint Slide illustrates the stages of performing landing page or website testing. It includes steps, such as designing hypotheses, determining targets, choosing an item to test, and performing A/A testing. It also includes A/B testing, determining the test’s sample size, deciding the experiment's duration, etc. The steps mentioned in the template are also showcased in the form of a graph for better comprehension and retention of information.

Data-Driven Decisions Start with Marketing Experiments

Marketing experiments are tools for businesses looking to improve marketing strategy, better understand customer behavior, and drive growth. These help firms innovate, remain competitive, and adapt to changing market dynamics by encouraging experimentation, continuous learning, and data-driven decision-making. It enables firms to seize new possibilities, increase consumer engagement, and achieve long-term success. Our PowerPoint Templates may have customizable elements that enable companies to personalize slides, while employing recognized best practices. 

Related posts:

• How to Design the Perfect Service Launch Presentation [Custom Launch Deck Included]
• Quarterly Business Review Presentation: All the Essential Slides You Need in Your Deck
• [Updated 2023] How to Design The Perfect Product Launch Presentation [Best Templates Included]
• 99% of the Pitches Fail! Find Out What Makes Any Startup a Success

Liked this blog? Please recommend us

Top 10 Logistics Business Plan Templates with Samples and Examples (Editable Word Doc, Excel, and PDF Included)

Top 10 Logistics PPT Templates with Samples and Examples

This form is protected by reCAPTCHA - the Google Privacy Policy and Terms of Service apply.

What Is Experiment Marketing? (With Tips and Examples)

Do you feel like your marketing efforts aren’t quite hitting the mark? There’s an approach that could open up a whole new world of growth for your business: marketing experimentation.

This isn’t your typical marketing spiel. It’s about trying new things, seeing what sticks, and learning as you go. Think of it as the marketing world’s lab, where creativity meets strategy in a quest to wow audiences and break the internet.

In this article, we’ll talk about what are marketing experiments, offer some killer tips to implement and analyze marketing experiments, and showcase examples that turned heads.

Ready to dive in?

Shortcuts ✂️

What is a marketing experiment, why should you run marketing experiments, how to design marketing experiments, how to implement marketing experimentation, how to analyze your experiment marketing campaign, 3 real-life examples of experiment marketing.

Marketing experimentation is like a scientific journey into how customers respond to your marketing campaigns.

Imagine you’ve got this wild idea for your PPC ads. Instead of just hoping it’ll work, you test it. That’s your experiment. You’re not just throwing stuff at the wall to see what sticks. You’re carefully choosing your shot, aiming, and then checking the impact.

Marketing experiments involve testing lots of things, like new products and how your marketing messages affect people’s actions on your website.

Running a marketing experiment before implementing new strategies is essential because it serves as a form of insurance for future marketing endeavors.

By conducting marketing experiments, you can assess potential risks and ensure that your efforts align with the desired outcomes you seek.

One of the main advantages of marketing experiments is that they provide insight into your target audience, helping you better understand your customers and optimize your marketing strategies for better results. 

By ensuring that your new marketing strategies are the most impactful, you’ll achieve better campaign performance and a better return on investment.

Now that we’ve unpacked what are marketing experiments, let’s dive deeper. To design a successful marketing experiment, follow the steps below.

1. Identify campaign objectives

Establishing clear campaign objectives is essential. What do you want to accomplish? What are your most important goals? 

To identify campaign objectives, you can:

• Review your organizational goals
• Brainstorm with your team
• Use the SMART framework (Specific, Measurable, Achievable, Relevant, Time-bound) to define your objectives

Setting specific objectives ensures that your marketing experiment is geared towards addressing critical business challenges and promoting growth. This focus will also help you:

• Select the most relevant marketing channels
• Define success metrics
• Create more successful campaigns
• Make better business decisions

2. Make a good hypothesis

Making a hypothesis before conducting marketing experiments is crucial because it provides a clear direction for the experiment and helps in setting specific goals to be achieved.

A hypothesis allows marketers to articulate their assumptions about the expected outcomes of various changes or strategies they plan to implement.

By formulating a hypothesis, marketers can create measurable and testable statements that guide the experiment and provide a basis for making informed decisions based on results.

It helps in understanding what impact certain changes may have on your customers or desired outcomes, thus enabling marketers to design effective experiments that yield valuable insights.

3. Select the right marketing channels

Choosing the right marketing channels is crucial for ensuring that your campaign reaches your customers effectively. 

To select the most appropriate channels, you should consider factors such as the demographics, interests, and behaviors of your customers, as well as the characteristics of your product or service.

Additionally, it’s essential to analyze your competitors and broader industry trends to understand which marketing channels are most effective in your niche. 

4. Define success metrics

Establishing success metrics is a crucial step in evaluating the effectiveness of your marketing experiments. 

Defining success metrics begins with identifying your experiment’s objectives and then choosing relevant metrics that can help you measure your success. You’ll also want to set targets for each metric.

Common success metrics include:

• conversion rate,
• cost per acquisition,
• and customer lifetime value.

When selecting appropriate metrics for measuring the success of your marketing experiments, you should consider the nature of the experiment itself – whether it involves email campaigns, landing pages, blogs, or other platforms.

For example, if the experiment involves testing email subject lines, tracking the open rate would be crucial to understanding how engaging the subject lines are for the audience.

When testing a landing page, metrics such as the submission rate during the testing period can reveal how effective the page is in converting visitors.

On the other hand, if the experiment focuses on blogs, metrics like average time on page can indicate the level of reader engagement.

Once you’ve finished designing your marketing experiments, it’s time to put them into action.

This involves setting up test groups, running tests, and then monitoring and adjusting the marketing campaigns as needed.

Let’s see the implementation process in more detail!

1. Setting up test groups

Establishing test groups is essential for accurately comparing different marketing strategies. To set up test groups, you need to define your target audience, split them into groups, create various versions of your content, and configure the test environment.

Setting up test groups ensures your marketing experiment takes place under controlled conditions, enabling you to compare results more accurately. 

This, in turn, will help you identify the most effective tactics for your audience.

2. Running multiple tests simultaneously

By conducting multiple tests at the same time, you’ll be able to:

• Collect more data and insights
• Foster informed decision-making
• Improve campaign performance

A/B testing tools that allow for simultaneous experiments can be a valuable asset for your marketing team. By leveraging these tools, you can streamline your experiment marketing process and ensure that you’re getting the best results from your efforts.

3. Monitoring and adjusting the campaign

Monitoring and adjusting your marketing experiment campaign is essential to ensure that the experiment stays on track and achieves its objectives. 

To do so, you should regularly:

• Review the data from your experiment to identify any issues.
• Make necessary adjustments to keep the experiment on track.
• Evaluate the results of those adjustments.

Proactive monitoring and adjustment of your campaign helps identify potential problems early, enabling you to make decisions based on data and optimize your experiments.

As discussed above, after implementing your marketing experiment you’ll want to analyze the results and learn from the insights gained.

Remember that the insights gained from your marketing experiments are not only valuable for the current campaign you’re running but also for informing your future marketing initiatives.

By continuously iterating and improving your marketing efforts based on what you learn from your experiments, you can unlock sustained growth and success for your business.

1. Evaluating the success of your campaign

Assessing the success of your marketing experiment is vital, and essentially it involves determining if the campaign met its objectives and whether the marketing strategies were effective. 

To evaluate the success of your marketing campaigns, you can:

• Compare website visits during the campaign period with traffic from a previous period
• Utilize control groups to measure the effect of the campaign
• Analyze data such as conversion rates and engagement levels

2. Identifying patterns and trends

Recognizing patterns and trends in the data from your marketing experiments can provide valuable insights that can be leveraged to optimize future marketing efforts. 

Patterns indicate that many different potential customers are experiencing the same reaction to your campaigns, for better or for worse. 

To identify these patterns and trends, you can:

• Visualize customer data
• Combine experiments and data sources
• Conduct market research
• Analyze marketing analytics

By identifying patterns and trends in your marketing experiment data, you can uncover insights that will help you refine your marketing strategies and make data-driven decisions for your future marketing endeavors.

3. Applying learnings to future campaigns

Leveraging the insights gained from your marketing experiment in future campaigns ensures that you can continuously improve and grow the effectiveness of your marketing efforts. 

Applying learnings from your marketing experiments, quite simply, involves:

• analyzing the data,
• identifying the successful strategies,
• documenting key learnings, and
• applying these insights to future campaigns

By consistently applying the learnings from your marketing experiments to your future  digital marketing efforts , you can ensure that your marketing strategies are data-driven, optimized for success, and always improving.

Now that we’ve talked about the advantages of experiment marketing and the steps involved, let’s dive into real-life cases that showcase the impact of this approach.

By exploring these experiment ideas, you’ll get a clear picture of how you can harness experiment marketing to get superior results.

You can take these insights and apply them to your own marketing experiments, boosting your campaign’s performance and your ROI.

Example 1: Homepage headline experiment

Bukvybag , a Swedish fashion brand selling premium bags, was on a mission to find the perfect homepage headline that would resonate with its website visitors.

They tested multiple headlines with OptiMonk’s Dynamic Content feature to discover which headline option would be most successful with their customers and boost conversion rates.

Take a look at the headlines they experimented with, which all focused on different value propositions.

Original: “ Versatile bags & accessories”

Variant A: “Stand out from the crowd with our fashion-forward and unique bags”

Variant B: “Discover the ultimate travel companion that combines style and functionality”

Variant C: “ Premium quality bags designed for exploration and adventure”

The results? Bukvybag’s conversions shot up by a whopping 45% as a result of this A/B testing!

Example 2: Product page experiment

Varnish & Vine , an ecommerce store selling premium plants, discovered that there was a lot they could do to optimize their product pages.

They turned to OptiMonk’s Smart Product Page Optimizer and used the AI-powered tool to achieve a stunning transformation.

First, the tool analyzed their current product pages. Then, it crafted captivating headlines, subheadlines, and lists of benefits for each product page automatically, which were tailored to their audience.

After the changes, the tool ran A/B tests automatically, so the team was able to compare their previous results with their AI-tailored product pages.

The outcome? A 12% boost in orders and a jaw-dropping 43% surge in revenue, all thanks to A/B testing the AI-optimized product pages.

Example 3: Email popup experiment

Crown & Paw , an ecommerce brand selling artistic pet portraits, had been using a simple Klaviyo popup that was underperforming, so they decided to kick it up a notch with a multi-step popup instead. 

On the first page, they offered an irresistible discount, and as a plus they promised personalized product recommendations.

In the second step, once visitors had demonstrated that they wanted to grab that 10% off, they asked simple questions to learn about their interests. Here are the questions they asked:

For the 95% who answered their questions, Crown & Paw revealed personalized product recommendations alongside the discount code in the final step.

The result? A 4.03% conversion rate, and a massive 2.5X increase from their previous email popup strategy. 

This is tangible proof that creatively engaging your audience can work wonders.

What is an example of a market experiment?

An example of a marketing experiment could involve an e-commerce company testing the impact of offering free shipping on orders over $50 for a month. If they find that the promotion significantly increases total sales revenue and average order value, they may decide to implement the free shipping offer as a permanent strategy.

What is experimental data in marketing?

Experimental data in marketing refers to information collected through tests or experiments designed to investigate specific hypotheses. This data is obtained by running experiments and measuring outcomes to draw conclusions about marketing strategies.

How do you run a marketing experiment?

To run a marketing experiment, start by defining your objective and hypothesis. Then, create control and experimental groups, collect relevant data, analyze the results, and make decisions based on the findings. This iterative process helps refine marketing strategies for better performance.

What are some real-life examples of experiment marketing?

Real-life examples of marketing experiments include A/B testing email subject lines to determine which leads to higher open rates, testing different ad creatives to measure click-through rates, and experimenting with pricing strategies to see how they affect sales and customer behavior. 

How to brainstorm and prioritize ideas for marketing experiments?

Start by considering your current objectives and priorities for the upcoming quarter or year. Reflect on your past marketing strategies to identify successful approaches and areas where performance was lacking. Analyze your historical data to gain insights into what has worked previously and what has not. This examination may reveal lingering uncertainties or gaps in your understanding of which strategies are most effective. Use this information to generate new ideas for future experiments aimed at improving performance. After generating a list of potential strategies, prioritize them based on factors such as relevance to your goals, timeliness of implementation, and expected return on investment.

Wrapping up

Experiment marketing is a powerful tool for businesses and marketers looking to optimize their marketing strategies and drive better results. 

By designing, implementing, analyzing, and learning from marketing experiments, you can ensure that your marketing efforts are data-driven, focused on the most impactful tactics, and continuously improving.

Want to level up your marketing strategy with a bit of experimenting? Then give OptiMonk a try today by signing up for a free account!

Nikolett Lorincz

You may also like.

How a Mystery Discount Skyrocketed Nexus Nutrition’s Email Subscribers

Holiday Marketing Campaigns & Ideas to Spark Your Festive Spirit in 2024

What We Learned From Analyzing 18,000 Ecommerce Stores: 5 CRO Tips

• Összes funkció
• Grow Your Email List
• Grow Your Messenger List
• Reduce Cart Abandonment
• Increase Avg. Cart Value
• Promote Special Offers
• Collect Customer Feedback
• Facilitate Social Sharing
• For Mid-Market/Enterprise users
• Partners program OLD
• Terms & Conditions
• Security & Privacy
• We’re Hiring! ????
• eCommerce Guides
• Case Studies
• All features
• Book a demo

Partner with us

• Partner program
• Become an affiliate
• Agency program
• Success Stories
• We're hiring
• Tactic Library
• Help center / Support
• Optimonk vs. Optinmonster
• OptiMonk vs. Klaviyo
• OptiMonk vs. Privy
• OptiMonk vs. Dynamic Yield
• OptiMonk vs. Justuno
• OptiMonk vs. Nosto
• OptiMonk vs. VWo
• © OptiMonk. All rights reserved!
• Terms of Use
• Privacy Policy
• Cookie Policy

Product updates: Introducing OptiMonk AI

Home / Experimentation / Marketing experimentation best practises [+ our best performing experiments]

Marketing experimentation best practises [+ our best performing experiments]

Article originally published in November 2022 by Stuart Brameld . Most recent update in April 2024.

Request a demo

Stuart is the Founder of Growth Method, Growth Advisor to B2B companies (currently Colt, Visio and MobiLoud) and Mentor at Growth Mentor.

Experiment results

Recent experiments results include competitor SEO, AI-driven content, exit-intent modals and AB testing homepage headlines.

"We are on-track to deliver a 43% increase in inbound leads this year. There is no doubt the adoption of Growth Method is the primary driver behind these results." 

How the Wright Brothers used agile & experimentation

In the late 1800s, governments around the world had invested over a billion dollars in aviation projects led by prominent scientific minds that had tried (and failed) to achieve powered man flight. Ultimately, they were beaten by two bicycle enthusiasts known as the Wright brothers.

But why did the Wright brothers succeed?

Instead of building an entire aircraft and trying to get it to fly, as many of their predecessors had done, the Wright brothers completed over 700 flights in gliders first. They started small, gathered data, and continuously iterated their way to success – building, measuring and learning along the way. The timeline below summarises the story of their success:

• 1899 – 1.5m wingspan kite
• 1900 – 5.3m wingspan tethered glider
• 1901 – 6.7m wingspan untethered glider
• 1902 – 9.8m untethered glider with rudder
• 1903 – 12.3m powered airplane, short straight-line flight
• 1904 – 12.3m powered airplane, short circular flight
• 1905 – 12.3m powered airplane, 30 minute flight duration

Experts believe it was this continuous learning and experimentation, what we would now call an agile approach, which led to their success.

The problem with campaigns & waterfall project delivery

Marketing projects typically follow the classic waterfall project delivery approach, and tend to look something like the following:

• Come up with a theme for your marketing project or campaign
• Seek approval from your manager and stakeholders
• Build out the plan based on the agreed requirements, with support from in-house teams, agencies, designers and developers
• Bring together all campaign assets and test the user journey

Unfortunately, with this big bang approach, you take one huge swing and if you miss – if your target audience is wrong, the messaging doesn’t resonate, people don’t engage or convert – it’s over.

If the initial plan was wrong (and let’s face it, unless you’re run exactly the same campaign before under very similar conditions, it’s likely more of a guess than a plan), you have just spent considerable time and money building a campaign that nobody pays attention to and that achieves nothing. Big bang = big risk.

The problem is that the waterfall project methodology was developed during the mass production era (car manufacturing, steel production, tobacco production etc) where problems were well-defined and the solutions clearly understood.

Using waterfall project management you can launch the “perfect” marketing campaign – one that is on time, on budget and beautifully executed – but that delivers absolutely nothing for the business. Eric Ries refers to this successful execution of a bad plan as “achieving failure”.

Ries also uses the term “success theatre” to describe charismatic individuals, often in larger organisations who are able to rally individuals and gather buy-in for projects that fail to deliver business value. Anyone can say “I have a vision for something big”, it’s far better to say “I have a vision for something big, and I’ve already run tests and proven that there is demand for it”.

“There is surely nothing quite so useless as doing with great efficiency what should not be done at all.” Peter Drucker

When our world is changing and evolving, customers are changing, customer expectations are changing, marketing channels are changing and your company strategy is changing, a 100 year old project management approach no longer works. Enter marketing experimentation.

The benefits marketing experimentation

What is needed is a new approach to marketing project management , one that is appropriate for marketers and marketing teams today that operate under conditions of uncertainty.

As a result of work done on agile methodology in IT and software development over the years we know a few things to be true about the big bang, waterfall approach to projects:

• Longer projects tend to get longer, they suffer from more scope creep and are more likely to get interrupted
• Cost and time increase with complexity
• There is often zero customer value delivered until right at the end

These problems are not unique to the IT world. The same applies when writing a book or essay, when writing code and in many other areas of life. Every creative human endeavour requires an enormous amount of trial-and-error.

Big bang equals big risk, and if we can decrease the time between pivots and changing the direction, we can increase the odds of project success. We need to shift from resource optimisation and instead focus on time-to-market optimisation. Marketing experimentation approach enables this shift from fixed time versus fixed scope.

The more seldom we release something into the world, the more expensive and risky each release it. Conversely, the more often we release things, the cheaper and safer those releases become.

This is where the marketing experimentation, and the minimum viable test comes in.

Marketing experimentation & the minimum viable test

“Humans will do marvellous things to avoid getting into the arena, where the possibility of failure is present. That’s why planning is so seductive. Planning can reduce uncertainty, and uncertainty is scary. But uncertainty can never be reduced to zero. So in most cases, it’s best to get momentum and solve the biggest problems as they come.”

Unless you’ve released the same thing, to the same audience, at the same time before, you’re not really planning, you’re guessing. And if you’re going to be wrong, you’re better off spending $100 and losing a few days work, than spending $100,000 and losing 3 months of work.

This is why modern marketers run marketing experiments. Marketing experimentation is all about making specific, concrete predictions ahead of time in order to increase learnings and reduce uncertainty over the long term.

The goal of marketing experimentation is to move away from a big bet culture towards a more agile scientific approach to marketing with the ultimate goal of reducing waste.

Anyone can put compounds in a beaker and heat it up, in the same way that any marketing team can produce a piece of content – neither are science. The science comes from having a hypothesis , a set of predictions on what is likely to happen. The goal is for marketing to be effective, not purely efficient . Any marketing team can produce things in an efficient way, but only some are effective.

Why use marketing experimentation?

There are many reason to use marketing experimentation, these include:

• Marketing experiments can help improve user experience, grow engagement and increase sign-ups from prospects and customers
• Marketing experiments help to isolate specific variables (such as conversion rate) to ensure the right decisions are being made
• Experiments allow us to quickly and easily prove that an opportunity is worthy of additional time and investment before it is too late.

You and your team will learn more and achieve more by implementing 3 ideas that have been properly considered and scoped, than by implementing 15 new ideas on a whim.

Being explicit with your experiment documentation also ensures that:

• You have clearly thought through your plan and how you’re doing to measure the results so that, once the experiment is complete, you can clearly prove or disprove the original hypothesis
• Team members are kept up to date with what is being done and continually learning from each others experiments, without the need for meetings
• New team members can go back and understand and/or repeat previous experiments (without having to talk to the person that ran it last time)
• Where changes are made, and experiments are unsuccessful, there is a clear log of configuration changes that should be “undone”

Lastly, building a culture of experimentation in larger organisations, especially where long established (often poor) practices have taken root, can be a challenging. Spreading awareness and evangelising your agile marketing programme by sharing results and data in a transparent manner is the best way to gather support from colleagues across the business.

Marketing experimentation examples

If you’ve used the Internet today, the chances are you’ve participated in a number of online experiments (officially know as randomised controlled trials) without knowing it. Here are a couple of famous marketing experiment examples from Google and Microsoft.

Google’s ’50 shades of Blue’ experiment

In the late 1990s, shortly after Google launched ad links in Gmail, they decided to test which shade of blue would result in the most clicks on the ad link.

Google ran a series of 1% AB test experiments where 1% of users were shown a particular shade of blue, then another 1% a different shade of blue, and so on. In total, over 40 different shades of blue were tested. The results were surprising and prove the potential value of data-driven decision making at scale.

“We saw which shades of blue people liked the most, demonstrated by how much they clicked on them. As a result we learned that a slightly purpler shade of blue was more conducive to clicking than a slightly greener shade of blue …. the implications of that for us, given the scale of our business, was that we made an extra $200m a year in ad revenue.” Dan Cobley, Managing Director, Google UK

Microsoft Bing search engine experiment

In 2012 an employee at Microsoft had an idea about changing the way search engine ad headlines were displayed. The suggestion was to move ad text to the title line to make it longer, as below.

The idea was delayed for months as it wasn’t seen as particularly valuable, until many months later an engineer felt it was simple enough to test and launched an AB test. The idea increased Bing’s revenue by 12% (over $120M at the time) without hurting user experience metrics, it was so impactful it triggered internal ad revenue monitoring alerts.

Marketing experimentation cycle time

“Success is a direct result of the number of experiments you perform.” Anthony Moore

The best marketing teams are built on a system of compounding loops. The more experiments you run, the more you learn (about your users, your product and your marketing channels), and the more you are able to apply those learnings over time to increase your ratio of successful experiments and grow.

Rapid iterations will beat the competition and built team morale and so the higher the velocity of testing, the faster your team will learn how to accelerate growth. Relatively few tests are likely to produce dramatic gains hence finding wins, both big and small, is a numbers game.

“You could beat any grandmaster at chess if you could move twice every time he moved once.” James Currier

Your teams goal should always be to maximise learning and as a result most agile marketing teams run experiments in 4-week or 6-week cycles.

Marketing experiment owners

Every experiment or project is assigned an owner. The owner is completely in charge of the experiment and gets to choose how the experiment is executed. They are in charge of getting it done, from initial documentation, to the process, to the delegation, to the goals, to the deadlines and logging the results.

This ownership allows everyone to work in a way that they feel comfortable but within defined boundaries. It gives the whole team trust, creative freedom and the chance to prove themselves and share new ways of doing things with the rest of the team.

Marketing experiments & the minimum viable test

The minimum viable test is a core principle and lean startup and design thinking that can equally be applied to marketing opportunities.

Our goal in discovery is to validate our ideas the fastest, cheapest way possible. Discovery is about the need for speed. This lets us try out many ideas, and for the promising ideas, try out multiple approaches. There are many different types of ideas, many different types of products, and a variety of different risks that we need to address (value risk, usability risk, feasibility risk, and business risk). So, we have a wide range of techniques, each suitable to different situations. Marty Cagan, Inspired

Experimentation doesn’t have to be a big project that requires developers, designers, expensive software, or complicated data analysis. It just needs to be a way for you and your team to test out the hypotheses and learn from the data. Do the smallest, minimum amount of work to get the insight you’re looking for. Wes Kao calls this the minimum effective dose .

Remember that when introducing a new idea, the goal isn’t perfection. The goal is initial feedback and learning.

Test one thing at a time – like the title or imagery – so that you’re able to learn from each experiment which variable is causing a difference in results.

The more variables you have in one experiment, the less meaningful your results will be. If you test one landing page design against a completely different landing page (without any experiments in between), your results won’t tell you much about what actually worked in that experiment.

Also remember that with modern tools you don’t necessarily have to roll out a change that affects every website visitor or app user.

There are no magical ideas or silver bullets in marketing and so you must learn to avoid becoming overly invested in the idea outcome.

The age of the waterfall project is over. Digital disruptors such as AirBnb, Uber, Netflix, Amazon and others stay ahead of their competitors in tiny sprints.

Your goal should be to find the least resource intensive way to test the hypothesis such that it still delivers a meaningful experiment outcome (i.e. that still proves or disproves the hypothesis). Your aim is to test your riskiest assumptions at the same time as collecting feedback to guide a future iteration of the idea. This is known as the Minimum Viable Test.

The bigger the test, the more resources required, the more likely your work will affect other teams, and the higher the stakes. Fighting feature creep and scope creep and maintaining a good testing cadence is key to the growth marketing process.

Experiments do not, and should not have to be perfect before they see the light of day. The trap of perfection is that whilst you’re working on perfection, your competitor is chatting up your customer.

Aim to minimise these potential downsides by keeping your test as small as possible. If the results are good, then you double down.

What does good marketing experimentation look like?

1. test methodology.

One of the most important aspects of experimentation is the research and planning phase. This planning forces us to think through the “why” of an experiment, including what we expect to happen and determining whether it’s worth doing in the first place.

Following on from the growth hypothesis , the experiment research and methodology may include:

2. Success Metric

Record your primary success metric (goal) and any success criteria at the same time as refining your hypothesis. You should understand how you are going to measure success before you get started in order to avoid making the data fit your own preconceived ideas or hopes for the outcome.

It is likely you may need to measure more than one metric, include metrics downstream from the experiment that may be impacted. However, keep in mind that more data equals more time and more risks in muddying the focus and inundating individuals with too much information.

Additionally, however rational and objective you think you are, we all want to win and to bring in results. Whether intentional or not, it is easy to create a successful experiment by cherry-picking the data once it’s complete.

Here’s a real-world example from Buffer a/b testing the text in a tweet to one of their articles. Can you tell which the winner was?

Without knowing the metric they were looking to influence, it’s impossible to know which was the winner. If clicks were the goal, tweet 1 was the winner, if retweets were the goal, tweet 2 was the winner.

You should consider:

• What data can be collected to prove or disprove the hypothesis? What is the clear measure of success? Collect this data and no more.
• Is the above data currently being recorded? e.g. required Google Analytics events
• What metric(s) are you trying to improve?
• What determines success versus failure?

3. Baseline & Predicted Impact

Before starting your test you should create a baseline value – a reference point of the current state – so that you can measure progress or change. If baseline data is unavailable use benchmark data or a reference point, such as average industry conversion stats.

“Experiments need a baseline, so you can measure results, otherwise you’re just spinning your wheels” Tim Wu, Director of Growth, Framed.io

Your baseline may be based on:

• Quantitative Primary Data – previous experiments, surrounding data, funnel data
• Qualitative Primary Data – surveys, support emails, user testing recordings
• Secondary Data – blogs, competitor observation, case studies, things you have read or heard from others

4. Experiment notifications

Once your experiments is ready a notification should be sent to your team that it is being launched so that everyone is aware of live changes and there are no surprises. Another notification should be sent with results once the experiment is complete.

5. Constant experimentation

Your experiments shouldn’t be a one-off piece of work, it is incredibly unlikely a marketing project cannot be improved. Aim to continually revise and improve your work as good growth is built on a culture of constant experimentation and compounding results.

How to implement marketing experimentation today

Looking for growth marketing experiment templates for Pipefy, Trello, Airtable, Excel, see our article on growth experiment templates here .

Looking to get started with a growth marketing project management tool? We’ll love to show you Growth Method .

Other articles you might like

Here are some related articles and further reading you may find helpful.

• Evidence-guided v opinion-based growth
• Growth Marketing Templates for Pipefy, Trello, Airtable, Notion and more
• In-depth Planview AgilePlace review 2023
• Growth marketing & the theory of constraints
• UpGrow growth experiment management | An independent review

We don't have a newsletter

but we do share our in-house growth experiments.

Recent experiments include competitor SEO, AI-driven content, exit-intent modals and AB testing homepage headlines.

How to Plan & Test Marketing Experiments

Learn how to run marketing experiments the right way in 2022. We show you how every marketing strategy can be a learning strategy.

Today’s marketing leaders are using marketing tests and experiments to increase insight and action. And organisations committed to marketing experiments are transforming their businesses with increased sales. 

How? Because experiments are the best way to create new knowledge systematically and to learn more about your customers and your audience.

They force you to question your own ideas, beliefs, and the best practices you’ve read so much about. 

Continually testing and learning is crucial to making future marketing decisions that are based on proven results rather than opinions. 

After all, at the pace customer behaviours are evolving, it’s becoming even more challenging for marketers to stay on top of how brands and customers can connect. 

Why Run Marketing Experiments?

In the short and long run, marketing experiments are pivotal for success. On a deeper level, testing what works and analysing your data makes it easier for you to determine your ROI and make data-driven decisions. 

Here are 3 of the top reasons why you should prioritise marketing experiments:

1) Improved customer experience: Marketing experiments help you understand customer needs, and how you can deliver more personalised customer experiences. 

2) Better decision making: Marketing experiments help you make better-informed, data-driven decisions. By conducting marketing experiments, you’ll be able to get a snapshot of what’s working and what’s not working.

3) Understand customers more deeply: Marketing experiments allow you to easily learn and predict customer behaviour based on data and make course corrections along the way. 

Get Inspired : 21 Marketing Experiments & Test Ideas

The Planning and Testing Process

Before setting up a marketing experiment, make sure you plan for maximum effectiveness. We’ve pulled together our top tips for planning and testing your marketing experiments.

Start with a goal and hypothesis : Good marketing experiments start with a clear goal to validate. A hypothesis forces you to think about the experiment. Ask yourself questions about how and why you think outcomes will occur. 

Analyse where you are now : Once you understand what needle you’re trying to move, look at existing data in your marketing management tool to see where you are, so you can know when you reach your goal. 

Brainstorm ideas : Equipped with insight on your starting point, you can meet with your team to come up with the highest-impact ideas to get where you want to be with marketing and sales. 

Prioritise your ideas : Use the ICE (Impact, Confidence, Ease) method to create an abundance of ideas, before you need them, so you can respond rapidly to performance data. After this step, you should have a clear idea of which experiments to go after. 

Run the experiments : Start small and prove the value of experimentation with small wins. Allow campaigns of at least 3 weeks to gather info about what works best to optimise your results. 

Measure campaign success : And last (but definitely not least) in your marketing experiment process, measurement solutions are crucial when it comes to understanding impact, and making any needed real-time adjustments. 

Final Marketing Experiment Tips 

Most marketing experiments will have one of two objectives: optimising performance or unlocking growth.

To optimise performance, you can start by testing out different tactics, tools, and strategies. Try new content types or ad formats.

In terms of looking for growth opportunities, try something completely different. Throw your existing landing page out the window, and try something new. If the fresh version works better, use that as your benchmark going forward. 

Ideally, you’d want to test one variable at a time, but you also have to realise the impact of a small change will be small.

The final precondition for a good marketing experiment is that it’s run as a split test. This way, you can really compare the effectiveness of different ideas.

Bonus point: Build a culture of data-led experimentation. Encourage testing through ideation sessions with your team. Prioritise discussing ideas and reviewing results in order to develop learning for further growth.

Whether the results are good or not so good, every marketing experiment is an opportunity to learn.

Are you ready to start your first marketing experiment?

In an environment that’s always evolving, constant learning becomes crucial to ensure we stay ahead of the curve. 

Experimentation makes us think about what works, why it works, and what might be done differently. This is part of what makes it an important tool for marketers to keep a finger on the pulse of the evolving nature of marketing.

TrueNorth brings all of your data together to run marketing experiments at scale. It helps by continually surfacing insights to help you course-correct as you go. 

With marketing experiments software, there’s no more need to log into multiple tools to track important marketing metrics.

You can even automate ideation alerts and goal projections, and respond rapidly to drops in performance to stay on track.

Get started with your first marketing experiments for free with TrueNorth.

Marcus Taylor

Marcus is the CEO of TrueNorth, a growth marketing platform that helps marketing teams focus, align and track marketing in one place.

More from Marcus

• Follow Marcus

You may also like

Redefining growth marketing with truenorth.

Growth marketing requires a scientific and collaborative approach. Here’s how TrueNorth gets your whole team working (and growing) together.

The Growth Marketing Handbook (2022 Edition)

Everything you need to know about growth marketing – from what it is and how it differs to traditional marketing, to the frameworks, tools and tactic

25+ Books, Courses & Tools to Learn Growth Marketing

We’ve rounded up the top books, podcasts, courses, communities, guides and more to help you learn growth marketing in your preferred medium.

Leave a Reply Cancel Reply

Save my name, email, and website in this browser for the next time I comment.

Try TrueNorth for free

Free for 14 days, then $99/mo for all features & your whole team.

Please provide an email address

Something went wrong, please try here

• Getting Started
• Affiliate Program
• System status
• Growth Projection
• Growth Tracking
• Experiments
• Integrations
• What’s New?
• Create a marketing plan
• Plan your marketing calendar
• Growth marketing
• Marketing management
• Marketing planning
• Marketing forecasting

© 2024 Venture Harbour Ltd.

Privacy Terms Cookies

Guide your team in the right direction

Focus on the most impactful ideas

Plan, test & measure to move fast

Capture insights & results

Track growth & campaign performance automatically

Manage your clients’ marketing

Generate your best ideas to accelerate
your growth

Watch a 2-minute demo explaining how TrueNorth helps marketing teams hit their goals.

Learn how TrueNorth can streamline agency operations & wow your clients.

Get your team up and running in days with this handy guide to getting started.

Learn growth marketing – from the frameworks to the tools and tactics.

• Try for free

TABLE OF CONTENTS

How to Test and Measure Marketing Experiments with Trello

Home Blog Digital Marketing How to Test and Measure Marketing Experiments with Trello

Updated on January 22nd 2024

Sam Warren | 8 min read

Designing marketing experiments , monitoring their progress, and measuring results are all essential components to the success of any marketing campaign. Far too often however, the methodology behind this process is either overlooked, disorganized or otherwise poorly managed. To be fair, it can be  just a little bit complicated. This isn’t helped by the fact that most marketing departments will be running multiple experiments simultaneously  and across many different channels. That means there won’t usually be one simple dashboard to view results, nor one tool to create and run the experiments themselves. Have you or your marketing team struggled with this very problem? If so, take a deep breath and relax. There’s a simple solution, and it doesn’t cost a fortune either. In fact, follow my methods and you can track all of your marketing experiments from inception to completion without spending a dime. All you need is Trello .

Trello? What’s Trello?

Only the best project management tool available. Okay, I’m opinionated. So sue me. I’ve only become such a fanboy because Trello has changed my life both personally and professionally. Sure there are other powerful tools out there. I’m not going to knock any of them or discourage you from trying them. In fact, depending on your specific circumstances there may or may not be a better solution than Trello. After all, this is hardly a one-size-fits-all situation. Nevertheless, the focus of this guide is how to use Trello to manage experiments specifically. At its most basic level, Trello is a highly flexible tool that you can use to organize any manner of projects. But the simple and intuitive collaboration is what really makes it special and effective for us. It’s particularly well-suited to planning, executing, and learning from marketing experiments. By using custom labels, a solid high-level structure, and “power ups,” you’ll be able to effectively manage complicated experiments from beginning to end. Let’s take a look at what I mean by “high-level structure.”

Top-level organization

How you organize your board will depend on your team’s structure, your responsibility set, and what you want to accomplish. But I’ll share what works for my team. I organize my team’s boards in “sprints,” unofficial Agile style . This helps keep us focused on top priorities week over week, and further allows me to give my bosses a quick and accessible view of what we’ve completed, what we’re working on, and where we’re headed. You can hit the ground running by using Trello’s Sprint Template . Or you can use my own version  of this approach. Here’s a quick glance at how it’s structured. Moving cards from one week to the next is a simple drag and drop affair, and assigning labels and team members is a two-click process. The takeaway is that Trello’s interface makes it really easy to stay nimble  and keep the top level strategy in mind while you’re “in the weeds.”

“One card – one experiment”

If you checked out my sprint template above, you probably noticed that I like to keep a separate list for live experiments. I do this for a few reasons. Primarily, it keeps them separate from tasks that are more accurately described as “check the box” activities. Secondly, experiments need to be reevaluated later on. If you run into snags or start seeing lacking results, it’s important that you’re quickly able to access historical information about them. That’s why I recommend what I call the “one card, one experiment” methodology. Each experiment gets its own card. This way it’s never confusing or difficult to find any given experiment. I’d also recommend using simple and concise naming conventions for the cards individually. Trust me, it’s worth the extra few keystrokes. By keeping experiments contained on their own cards, your interface will remain accessible, simple and team-friendly.

How does an experiment begin?

The inception of an experiment card should follow a set protocol that you and your team agree upon ahead of time. When team members start just throwing ideas out onto cards, your board can quickly become a jumbled mess. Instead, I have team members “pitch” experiments that they think we should set up. If one of our marketing team members wants to run an experiment (that isn’t just a website tweak or something simple), I have them use the following template that you’re more than welcome to copy:

• What do I want to do?
• What will the impact be if this works?
• How confident am I that this will work?
• How much time/money/effort is required?
• How will I measure results?

On an actual card, it should look something like this: We find this method to be effective for a couple of pretty important reasons.

• It keeps us focused on objectives aimed at improving KPIs (key performance indicators).
• It encourages every team member to take responsibility.
• It demands that every project be properly researched and “defensible.”
• It helps us allocate budgets for every sprint, every month, etc.

Alternatively, you could use a completely different strategy for the organization of your experiments and still follow my methodology for tracking and measuring results. To explore that possibility, take a look at Trello’s own “ Growth Template .” You could use it on an entirely separate board to vet, discuss, and make determinations about what experiments to move forward with and when.

Tracking the experiments

So you’ve chosen the experiments you want to run and gone ahead and started some. Awesome! Now you need to be able to monitor them in real-time to get an idea of how they’re performing. This is where that “Live Experiments” list comes in handy. You can use it and its experiment cards to keep up to date and collaborate with teammates. Maybe you want to make tweaks to an ad campaign. Maybe you want to kill an experiment. Maybe you just want to give your colleague a high-five for coming up with such a great idea. Either way, you can do all of this on the cards themselves. I’d also recommend checking out a few of the “Power-Ups” as they can really make a difference.

Card aging power-up

I really love this power-up. It’s a very simple addition, but I find it extremely useful. It graphically “ages” cards when they haven’t received any updates in a while. If you’re running dozens of experiments, it can often be hard to remember if you’ve checked in on all of them recently. Card aging makes this a breeze. Pro tip:  Use the “pirate” mode. Because it’s just better. Obviously.

Calendar power-up

Speaking of losing track of things… Got 15 experiments running with various due dates? Having some trouble organizing which ones need your attention ASAP? Using the calendar power-up you’ll be able to get a simple and clear view of all of your cards with due dates, laid out on a calendar. Image Credit: Trello Simple and just plain awesome.

Harvest power-up

If you have a distributed team, or just want to keep tabs on how long certain projects take, this is the powerup for you. I like using this one for any “wildcard” projects I roll the dice on. Sometimes we’ll have an experiment that we want to give a go, but there’s concern that it may take more setup time than expected. Using Harvest, you’ll be able to quickly assess if the project is going to work out. to see if this becomes a problem. If one of your team members logs 8 hours on a card but you wanted the project to take 5 hours tops, it might be time to revisit the scope and expected ROI of the experiment.

Measuring the results

The grand finale! The last leg of the journey! Last call! You’ve made it. Let’s imagine one of your experiments is ready to be measured and analyzed. Now all that’s left is to dive into the data and start analyzing. If you’re lucky, you’ll see some growth trending and can implement those changes for either permanent inclusion or further testing. The job of the card is far from over however. For starters, I’d highly recommend posting your final thoughts and any conclusions you were able to draw from analyzing the data. Side note: You can bold words in Trello by encapsulating them in **double asterisks**. You can also divide the main descriptive text using a line break followed by three dashes (shown below). Take a few minutes to summarize your analysis. Attach screenshots. Use the GDrive power-up  to share spreadsheets with teammates. You get the idea. Last but not least, I like having an “Archived Test Results” list on my Trello board, where I keep completed experiment cards. You may decide you want to revisit an idea that failed before, or you may just want to be reminded of why  you made a particular decision later on down the road. For example: Two weeks after wrapping up an experiment, I might have a senior moment and find myself asking “why did I go with this particular landing page design ?” Well, if I was rigorous with my documentation process I’ll be able to easily retrace my steps and see actual data on the experiment card. I simply can’t tell you how many times this has saved me a lot of trouble. That’s all there is to it folks. A simple and effective way to track and measure marketing experiments using Trello. Do you have another method of tracking experiments? Do you have other Trello tips that help with projects like these? Looking forward to learning from your experiences in the comments below.

Guest Blogger @Mention

Get the latest and greatest digital marketing + social media tips every week!

How to Use Marketing Experimentation to Boost Your Marketing ROI

Product marketing is a nuanced, fast-paced, and multi-faceted endeavor. In a competitive environment with no guarantees, marketing experimentation serves as a surefire way to maximize reach, optimize conversions, and improve the user experience.

In this guide, we’ll walk you through what marketing experiments are, why you should run them, how to conduct experiments successfully, and seven experiments to try for yourself!

• Marketing experimentation helps you come up with ideas, test strategies, identify mistakes, optimize campaigns, uncover opportunities, and make data-driven decisions.
• You should run marketing experiments to gain unique insights into customer behavior that focus groups simply can’t offer. It’ll also drive sustained growth at lower costs by measuring the impact that different marketing efforts have on product growth .
• Conducting marketing experiments comes down to choosing your goal, hypothesis, audience, and metrics. After that, it’s simply a matter of actually running the experiment so you can analyze its results.
• A few common experimentation targets include marketing campaigns, subject lines, onboarding flows, landing pages, and in-app messages . You can also analyze more technical aspects, such as conversion paths or software automations.
• A/B testing helps you compare different versions of content/copy to see which ones perform best across your target market/user base. This will help you optimize messaging for both new prospects and existing customers.

What is marketing experimentation?

Marketing experimentation is an approach to generating fresh ideas, testing strategies, identifying your mistakes, optimizing campaigns, and making data-informed decisions moving forward.

It can also uncover hidden opportunities for organizations in new markets, segments, or use cases.

Why you should run marketing experiments?

There are many benefits to running a marketing experiment but a few notable ones are:

• Behavioral insights. Marketing experimentation can offer key insights into customer behavior so you have a better understanding of your users’ preferences.
• Sustained growth. Marketing experiments help you achieve sustained growth by testing new ideas and doubling down on the channels that offer the best marketing ROI.
• Data-driven optimization. A marketing experiment can guide future business decisions by highlighting the marketing strategies that produce the best results at a relatively low cost.

Clearly, marketing experiments are crucial to every stage of the growth journey, from gathering insights to driving growth and optimizing results.

How to conduct a successful marketing experiment?

A lot goes into setting up, running, and then analyzing a marketing experiment.

Running marketing experiments generally comes down to these six steps:

• Deciding the goal
• Making a hypothesis
• Choosing the audience
• Selecting your metrics
• Running the experiment
• Analyzing the results

Let’s take a closer look at each step.

Decide on the goal of your marketing experimentation

First and foremost, you’ll need proper goal-setting to ensure clarity on what these experiment ideas are supposed to achieve.

Brainstorm to find the most impactful goals, consider why that goal is valuable, and make sure your goals are SMART (Specific, Measurable, Achievable, Relevant, and Time-Bound).

Make a hypothesis

A hypothesis is just an explanation made using limited data that can serve as a starting point for your experiment.

For instance, you could hypothesize that adding live chat embeds to your website landing pages would increase the flow of potential customers within your sales pipeline.

Choose the audience

Not all experiments are suitable for every user. As such, utilizing user segmentation to run experiments on the most relevant customers will offer the best insights and increase the odds of achieving the desired results.

Decide on the key metrics you will use to track marketing success

A/B testing metrics will only be actionable if you use the right key performance indicators (KPIs) to measure results. These metrics give your experiments a measurable hypothesis to objectively investigate rather than a vague theory to aimlessly pursue.

The metrics you choose will depend on what you’re trying to prove. Metrics could be website traffic, click-through rates (CTR), conversion rates, marketing reach, customer engagement , retention rates , and more.

All that matters is that you select relevant metrics that will be able to prove or disprove your hypothesis.

Run your marketing experimentation

The actual testing process will vary depending on the goal, hypothesis, audience, and metrics.

Not every experiment will be carried out by marketing teams alone. Certain cross-functional projects will also require collaboration from customer success, support, sales, and development teams.

When the marketing team collaborates with other departments, it should ensure that everyone involved is briefed on:

• What the hypothesis is
• Which metrics will be used to track it
• When the experiment will begin/end

Analyze the results

Finally, it’s time to collect data and analyze the experiment’s results. Make sure you have enough data to accurately determine statistical significance. Small sample sizes or shallow data could lead to an independent variable skewing results one way or another.

Using the metrics you’ve chosen and the data you’ve collected, you should be able to conclude whether or not your hypothesis was correct. Proving a hypothesis takes longer when making minor changes like testing button colors versus experiments with a significant impact, like an overhauled checkout flow.

In either case, successful results will guide you on which future experiments to try or future campaigns to deploy. If the experiment had a genuine impact on user acquisition and revenue growth, then you’ll also have more budget to work with on subsequent experiments.

Note: In results, the independent variable is the cause, while the dependent variable is the effect.

7 common marketing experiments to try in SaaS

Now that you know what marketing experiments are, why you should run them, and how to conduct them, it’s time to go through a few popular options.

These marketing experiments are conducted fairly commonly within the SaaS industry and will serve as a good starting point for your experimentation:

Run experiments with different digital marketing campaigns

Experimenting with different ad copy, marketing strategies, and other platforms will help you bring in more customers (or prospects) in a cost-effective manner.

When conducting this type of experiment, it’s best to fail fast (and cheaply) so you can test the next campaign as soon as possible.

For instance, you might test different content formats to see if email marketing has a bigger impact on business growth compared to content marketing campaigns, for instance. The ideal marketing campaign will depend on your company, target audiences, and consumer behavior within your industry.

Experiment with email subject lines to see which one improves the open rate for onboarding emails

It’s no secret that customers don’t always read onboarding emails . As such, you should experiment with your onboarding email subject lines to see which variants yield the highest open rates. Fortunately, software products make A/B testing different subject lines super easy.

Tools like ActiveCampaign make it possible to show one subject line to half of your customers and another version to the other half — so you can quickly experiment to find the messaging that resonates the most with your user base.

A/B test different onboarding flows to improve customer experience

A/B testing lets you experiment with different onboarding flows to identify winners that increase trial-to-paid conversion rates . Be sure to collect research on which flows were completed before a user upgraded from their free account to a paid subscription.

Run multivariate testing to optimize website landing page conversion rates

In addition to A/B testing, a company could also run multivariate testing to optimize conversion rates.

For example, running a multivariate test on your website’s landing page will help you determine which version (or specific changes) leads to the highest percentage of website visitors completing a signup form or compels users to schedule a demo.

When conducting a multivariate test, the variables you should include are:

• Two headline versions
• Three body text versions
• Four call-to-action (CTA) versions

The messaging and buttons used for CTAs are the easiest to test yet provide the largest changes so testing four or more variants is well worth it. In the example above, you would be testing 24 different combinations (2 headlines x 3 content x 4 CTAs) to see which permutation performs best.

By nature of this testing process, the winner of each multivariate test will always be the one with the best combination of variables. This is important because some variables (like a CTA button) may be effective in isolation but lackluster when combined with different headlines or body text.

Bear in mind that multivariate tests require more data than A/B tests to produce statistically significant insights. This is because there are more variables to account for and multiple combinations to compare (versus A/B tests that only compare two versions).

Compare marketing messages in-app to see which resonates with your customer base

While it’s rare for a company to change its brand voice or messaging, experimenting with a different marketing message can offer valuable insights into what resonates with your customers. This could be applied to in-app messaging or communications sent via other channels like email or social media posts.

For instance, you could use two different modals to invite your users to a webinar and then see which modal resulted in the higher number of signups. This will clue you in on which colors, designs, and language your users respond to.

The same is true for other forms of in-app messaging , such as tooltips, banners, and slideouts.

Compare different paths to see which one leads to higher conversion rates

See how users progress through multiple conversion paths and compare the similarities/differences between each path. What touchpoints are prospective customers interacting with in one path versus another, and which factors seem to be the biggest determiner of conversions?

To experiment with conversion paths, simply select a starting point within your product (such as the home dashboard, analytics page, or a specific feature) and see how your users proceed from there. This type of analysis will help you identify problem areas within your paths and patch any funnel leaks.

Userpilot will actually be launching a path-tracking feature soon to help users compare conversion paths faster and extract more detailed insights!

Experiment with automation to see its result on customer experience

Experiment with different automation types to see how they impact the customer’s experience. For instance, you could onboard some users with a chatbot and the rest (your control group) without a chatbot and then see how this affects their engagement, activation, and adoption rates down the line.

You can also use predictive analytics to gauge how your users may respond to a particular automation (but your mileage may vary depending on the amount of data you have to work with). Incorporating new tools and widgets will have outsized — good or bad — effects on the product experience (PX) .

The goal of automation experimentation is to find ways to reduce manual effort while maintaining (or improving) the customer’s experience. This type of market research will help you craft a tech stack that’s not only suited to your business needs but that of your customers as well.

As you can see, marketing experiments can help you get your software product in front of more eyes at less cost and with better lead conversion. There are countless variables, from your users’ cognitive biases to the product’s price and even what season it is.

This means that frequent and strategic testing will help you cut through the noise to extract actionable insights that will actually benefit your business. Whether the goal is acquisition, retention, or expansion, a little experimentation can go a long way.

If you’re ready to start running in-app marketing experiments, split-testing messaging and creating effective onboarding flows without writing a single line of code, then it’s time to get your free Userpilot demo today!

Leave a comment Cancel reply

Save my name, email, and website in this browser for the next time I comment.

Get The Insights!

The fastest way to learn about Product Growth,Management & Trends.

The coolest way to learn about Product Growth, Management & Trends. Delivered fresh to your inbox, weekly.

The fastest way to learn about Product Growth, Management & Trends.

You might also be interested in ...

What are vanity metrics in product marketing definition & examples.

Aazar Ali Shad

What is Customer-led Marketing & Growth? + Best Practices

9 marketing research methods to refine your marketing strategy.

Saffa Faisal

• SUGGESTED TOPICS
• The Magazine
• Newsletters
• Managing Yourself
• Managing Teams
• Work-life Balance
• The Big Idea
• Data & Visuals
• Reading Lists
• Case Selections
• HBR Learning
• Topic Feeds
• Account Settings
• Email Preferences

A Step-by-Step Guide to Smart Business Experiments

• Eric T. Anderson and Duncan Simester

Every company can profit from testing customers’ reactions to changes. Here’s how to get started.

Reprint: R1103H

The power of analytics in decision making is well understood, but few companies have what it takes to successfully implement a complex analytics program. Most firms will get greater value from learning to do something simpler: basic business experiments.

Managers need to become adept at routinely using techniques employed by scientists and medical researchers. Specifically, they need to embrace the “test and learn” approach: Take one action with one group of customers, a different action (or no action at all) with a control group of customers, and then compare the results. The feedback from even a handful of experiments can yield immediate and dramatic improvements.

In this article, the authors provide a step-by-step guide to conducting business experiments. They look at organizational obstacles to success and outline seven rules to follow.

The Idea in Brief

Companies today understand the power of analytics, but dissecting past data is a complicated task that few firms have the technical skills to master. Most companies will get more value from simple business experiments.

To grow profits, managers need to become adept at techniques used by lab scientists and medical researchers: They should establish control and treatment groups to test the effects of changes in price, promotion, or product variation. They should also grasp the opportunities provided by general changes in the business—like store openings—that constitute natural experiments in consumer behavior.

Creating a culture of experimentation requires companies to overcome internal political and organizational obstacles. And not every experiment will succeed. But over time, companies that embrace a test-and-learn approach are more apt to find the golden tickets that will drive growth.

Over the past decade, managers have awakened to the power of analytics. Sophisticated computers and software have given companies access to immense troves of data: According to one estimate, businesses collected more customer information in 2010 than in all prior years combined. This avalanche of data presents companies with big opportunities to increase profits—if they can find a way to use it effectively.

• EA Eric T. Anderson is the Hartmarx Professor of Marketing at Northwestern’s Kellogg School of Management. Duncan Simester is the NTU Professor of Management Science at MIT’s Sloan School of Management.

Partner Center

How to Do A/B Testing: 15 Steps for the Perfect Split Test

Planning to run an A/B test? Bookmark this checklist for what to do before, during, and after to get the best results.

THE COMPLETE A/B TESTING KIT

Inside: Intro to A/B Testing, Significance Calculator, and Tracking Template

Published: 05/23/24

So, you want to discover what truly works for your audience, and you’ve heard about this mythical form of marketing testing. But you have questions like: “What is A/B testing in marketing, anyway?” and “Why does it matter?”

Don’t worry! You’ll get all the answers to your burning questions. I’ll even tell you the second answer straight away…

When marketers like us create landing pages , write email copy, or design call-to-action buttons, it can be tempting to use our intuition to predict what will make people click and connect.

But as anyone who’s been in marketing for a minute will tell you, always expect the unexpected. So, instead of basing marketing decisions on a “feeling,” you’re much better off running an A/B test to see what the data says.

Keep reading to learn how to conduct the entire A/B testing process before, during, and after data collection so you can make the best decisions based on your results.

The Complete A/B Testing Kit for Marketers

Start improving your website performance with these free templates.

• Guidelines for effective A/B testing
• Running split tests for email, landing pages, and CTAs
• Free simple significance calculator
• Free A/B test tracking template.

Download Free

All fields are required.

You're all set!

Click this link to access this resource at any time.

Table of Contents

What is A/B testing?

History of a/b testing.

• Why is A/B Testing important?

How does A/B testing work?

A/b testing in marketing, what does a/b testing involve, a/b testing goals, how to design an a/b test, how to conduct a/b testing, how to read a/b testing results, a/b testing examples.

• 10 A/B Testing Tips from Marketing Examples

What Is A/B Testing?

A/B Testing In Marketing

A/B testing, also known as split testing, is a marketing experiment wherein you split your audience to test variations on a campaign and determine which performs better. In other words, you can show version A of a piece of marketing content to one half of your audience and version B to another.

A/B testing is helpful for comparing two versions of a webpage, email newsletter, subject lines, designs, apps, and more or to see which is more successful.

Split testing takes the guesswork out of discerning how your digital marketing materials should look, operate, and be distributed. I'll walk you through everything you need to know about split testing, but I've got you covered if you're a visual learner. 

The video below walks you through everything you need to know. 

It’s hard to track down the “true” origins of A/B testing. However, in terms of marketing, A/B testing — albeit in its initial and imperfect form — arguably started with American advertiser and author Claude Hopkins.

Hopkins tested his ad campaigns using promotional coupons.

Still, Hopkins’ “Scientific Advertising” process didn’t include the key principles we use in A/B testing today. We have 20th-century biologist Ronald Fisher to thank for those.

Fisher, who defined statistical significance and developed the null hypothesis, helped to make A/B testing more reliable.

That said, the marketing A/B testing we know and love today started in the 1960s and ‘70s. It was also used to test direct response campaign methods. Another key marketing moment came to us in 2000.

At this time, Google engineers ran their first A/B test. (They wanted to know the best number of results to display on the search engine results page.)

Why is A/B testing important?

A/B testing has many benefits to a marketing team, depending on what you decide to test. For example, there is a limitless list of items you can test to determine the overall impact on your bottom line.

But you shouldn’t sleep on using A/B testing to find out exactly what your audience responds best to either. Let’s learn more.

You Can Find Ways To Improve Your Bottom Line

Let’s say you employ a content creator with a $50,000/year salary. This content creator publishes five articles weekly for the company blog, totaling 260 articles per year.

If the average post on the company’s blog generates 10 leads, you could say it costs just over $192 to generate 10 leads for the business ($50,000 salary ÷ 260 articles = $192 per article). That’s a solid chunk of change.

Now, if you ask this content creator to spend two days developing an A/B test on one article, instead of writing two posts in that time, you might burn $192, as you’re publishing fewer articles.

But, if that A/B test finds you can increase conversion rates from 10 to 20 leads, you just spent $192 to potentially double the number of customers your business gets from your blog.

… in a Low Cost, High Reward Way

If the test fails, of course, you lost $192 — but now you can make your next A/B test even more educated. If that second test succeeds, you ultimately spent $384 to double your company’s revenue.

No matter how many times your A/B test fails, its eventual success will almost always outweigh the cost of conducting it.

You can run many types of split tests to make the experiment worth it in the end. Above all, these tests are valuable to a business because they’re low in cost but high in reward.

You Can Find Out What Works for Your Audience

A/B testing can be valuable because different audiences behave, well, differently. Something that works for one company may not necessarily work for another.

Let’s take an unlikely B2B marketing tactic as an example. I was looking through HubSpot’s 2024 Industry Trends Report data for an article last week.

I noticed that 10% of B2B marketers planned to decrease their investment in NFTs as part of their strategy in 2024.

My first thought was, “ Huh, NFTs in B2B? ”

Then it hit me. To have that decrease, B2B marketers must’ve been using NFTs in the first place. Even more surprising than this revelation was that 34% of marketers plan to increase investment in NFTs as part of their B2B strategy.

That’s just one example of why conversion rate optimization (CRO) experts hate the term “best practices.” Because that “best practice”? Well, it may not actually be the best practice for you.

But, this kind of testing can be complex if you’re not careful. So, let’s review how A/B testing works to ensure you don’t make incorrect assumptions about what your audience likes.

To run an A/B test, you need to create two different versions of one piece of content, with changes to a single variable .

Then, you’ll show these two versions to two similarly-sized audiences and analyze which one performed better over a specific period. But remember, the testing period should be long enough to make accurate conclusions about your results.

Image Source

A/B testing helps marketers observe how one version of a piece of marketing content performs alongside another. Here are two types of A/B tests you might conduct to increase your website’s conversion rate.

Example 1: User Experience Test

Perhaps you want to see if moving a certain call-to-action (CTA) button to the top of your homepage instead of keeping it in the sidebar will improve its click-through rate.

To A/B test this theory, you’d create another, alternative web page that uses the new CTA placement.

The existing design with the sidebar CTA — or the “ control ” — is version A. Version B with the CTA at the top is the “ challenger .” Then, you’d test these two versions by showing each to a predetermined percentage of site visitors.

Ideally, the percentage of visitors seeing either version is the same.

If you want more information on how to easily perform A/B testing on your website, check out HubSpot’s Marketing Hub or our introductory guide.

Example 2: Design Test

Perhaps you want to find out if changing the color of your CTA button can increase its click-through rate.

To A/B test this theory, you’d design an alternative CTA button with a different button color that leads to the same landing page as the control.

If you usually use a red CTA button in your marketing content, and the green variation receives more clicks after your A/B test, this could merit changing the default color of your CTA buttons to green from now on.

Here are some elements you might decide to test in your marketing campaigns:

• Subject lines.
• Fonts and colors.
• Product images.
• Blog graphics.
• Navigation.
• Opt-in forms.

Of course, this list is not exhaustive. Your options are countless and differ depending on the type of marketing campaign you’re A/B testing. (Blog graphics, for example, typically won’t apply to email campaigns.)

But product images can apply to both email and blog testing.)

But let’s say you wanted to test how different subject lines impacted an email marketing campaign’s conversion rates. What would you need to get started?

Here’s what you’ll need to run a successful A/B test.

• A campaign: You’ll need to pick a marketing campaign (i.e., a newsletter, landing page, or email) that’s already live. We’re going with email.
• What you want to test: You’ll need to pick the element(s) you wish to A/B test. In this case, that would be the subject line used in an email marketing campaign. But you can test all manner of things, even down to font size and CTA button color. Remember, though, if you want accurate measurements, only test one element at a time.
• Your goals: Are you testing for the sake of it? Or do you have well-defined goals? Ideally, your A/B testing should link to your revenue goals. (So, discovering which campaign has a better impact on revenue success.) To track success, you’ll need to select the right metrics. For revenue, you’d track metrics like sales, sign-ups, and clicks.

A/B testing can tell you a lot about how your intended audience behaves and interacts with your marketing campaign.

Not only does A/B testing help determine your audience’s behavior, but the results of the tests can help determine your next marketing goals.

Here are some common goals marketers have for their business when A/B testing.

Increased Website Traffic

You’ll want to use A/B testing to help you find the right wording for your website titles so you can catch your audience’s attention.

Testing different blog or web page titles can change the number of people who click on that hyperlinked title to get to your website. This can increase website traffic.

Providing it’s relevant, an increase in web traffic is a good thing! More traffic usually means more sales.

Higher Conversion Rate

Not only does A/B testing help drive traffic to your website, but it can also help boost conversion rates.

Testing different locations, colors, or even anchor text on your CTAs can change the number of people who click these CTAs to get to a landing page.

This can increase the number of people who fill out forms on your website, submit their contact info to you, and “convert” into a lead.

Lower Bounce Rate

A/B testing can help determine what’s driving traffic away from your website. Maybe the feel of your website doesn’t vibe with your audience. Or perhaps the colors clash, leaving a bad taste in your target audience’s mouth.

If your website visitors leave (or “bounce”) quickly after visiting your website, testing different blog post introductions, fonts, or featured images can retain visitors.

Perfect Product Images

You know you have the perfect product or service to offer your audience. But, how do you know you’ve picked the right product image to convey what you have to offer?

Use A/B testing to determine which product image best catches the attention of your intended audience. Compare the images against each other and pick the one with the highest sales rate.

Lower Cart Abandonment

E-commerce businesses see an average of 70% of customers leave their website with items in their shopping cart. This is known as “shopping cart abandonment” and is, of course, detrimental to any online store.

Testing different product photos, check-out page designs, and even where shipping costs are displayed can lower this abandonment rate.

Now, let’s examine a checklist for setting up, running, and measuring an A/B test.

Designing an A/B test can seem like a complicated task at first. But, trust us — it’s simple.

The key to designing a successful A/B test is to determine which elements of your blog, website, or ad campaign can be compared and contrasted against a new or different version.

Before you jump into testing all the elements of your marketing campaign, check out these A/B testing best practices.

Test appropriate items.

List elements that could influence how your target audience interacts with your ads or website. Specifically, consider which elements of your website or ad campaign influence a sale or conversion.

Be sure the elements you choose are appropriate and can be modified for testing purposes.

For example, you might test which fonts or images best grab your audience’s attention in a Facebook ad campaign. Or, you might pilot two pages to determine which keeps visitors on your website longer.

Pro tip: Choose appropriate test items by listing elements that affect your overall sales or lead conversion, and then prioritize them.

Determine the correct sample size.

The sample size of your A/B test can have a large impact on the results — and sometimes, that is not a good thing. A sample size that is too small will skew the results.

Make sure your sample size is large enough to yield accurate results. Use tools like a sample size calculator to help you figure out the correct number of interactions or visitors to your website or participants in your campaign you need to obtain the best result.

Check your data.

A sound split test will yield statistically significant and reliable results. In other words, your A/B test results are not influenced by randomness or chance. But how can you be sure your results are statistically significant and reliable?

Just like determining sample size, tools are available to help verify your data.

Tools, such as Convertize’s AB Test Significance Calculator , allow users to plug in traffic data and conversion rates of variables and select the desired level of confidence.

The higher the statistical significance achieved, the less you can expect the data to occur by chance.

Pro tip: Ensure your data is statistically significant and reliable by using tools like A/B test significance calculators.

Schedule your tests.

When comparing variables, keeping the rest of your controls the same is important — including when you schedule to run your tests.

If you’re in the ecommerce space, you’ll need to take holiday sales into consideration.

For example, if you run an A/B test on the control during a peak sales time, the traffic to your website and your sales may be higher than the variable you tested in an “off week.”

To ensure the accuracy of your split tests, pick a comparable timeframe for both tested elements. Run your campaigns for the same length of time to get the best, most accurate results.

Pro tip: Choose a timeframe when you can expect similar traffic to both portions of your split test.

Test only one element.

Each variable of your website or ad campaign can significantly impact your intended audience’s behavior. That’s why looking at just one element at a time is important when conducting A/B tests.

Attempting to test multiple elements in the same A/B test will yield unreliable results. With unreliable results, you won’t know which element had the biggest impact on consumer behavior.

Be sure to design your split test for just one element of your ad campaign or website.

Pro tip: Don’t try to test multiple elements at once. A good A/B test will be designed to test only one element at a time.

Analyze the data.

As a marketer, you might have an idea of how your target audience behaves with your campaign and web pages. A/B testing can give you a better indication of how consumers really interact with your sites.

After testing is complete, take some time to thoroughly analyze the data. You might be surprised to find that what you thought was working for your campaigns was less effective than you initially thought.

Pro tip: Accurate and reliable data may tell a different story than you first imagined. Use the data to help plan or change your campaigns.

To get a comprehensive view of your marketing performance, use our robust analytics tool, HubSpot's Marketing Analytics software .

Follow along with our free A/B testing kit , which includes everything you need to run A/B testing, including a test tracking template, a how-to guide for instruction and inspiration, and a statistical significance calculator to determine whether your tests were wins, losses, or inconclusive.

Before the A/B Test

Let’s cover the steps to take before you start your A/B test.

1. Pick one variable to test.

As you optimize your web pages and emails, you’ll find there are many variables you want to test. But to evaluate effectiveness, you’ll want to isolate one independent variable and measure its performance.

Otherwise, you can’t be sure which variable was responsible for changes in performance.

You can test more than one variable for a single web page or email — just be sure you’re testing them one at a time.

To determine your variable, look at the elements in your marketing resources and their possible alternatives for design, wording, and layout. You may also test email subject lines, sender names, and different ways to personalize your emails.

Pro tip: You can use HubSpot’s AI Email Writer to write email copy for different audiences. The software is built into HubSpot’s marketing and sales tools.

Keep in mind that even simple changes, like changing the image in your email or the words on your CTA button , can drive big improvements. In fact, these sorts of changes are usually easier to measure than the bigger ones.

Note: Sometimes, testing multiple variables rather than a single variable makes more sense. This is called multivariate testing.

If you’re wondering whether you should run an A/B test versus a multivariate test, here’s a helpful article from Optimizely that compares the processes.

2. Identify your goal.

Although you’ll measure several metrics during any one test, choose a primary metric to focus on before you run the test. In fact, do it before you even set up the second variation.

This is your dependent variable , which changes based on how you manipulate the independent variable.

Think about where you want this dependent variable to be at the end of the split test. You might even state an official hypothesis and examine your results based on this prediction.

If you wait until afterward to think about which metrics are important to you, what your goals are, and how the changes you’re proposing might affect user behavior, then you may not set up the test in the most effective way.

3. Create a 'control' and a 'challenger.'

You now have your independent variable, your dependent variable, and your desired outcome. Use this information to set up the unaltered version of whatever you’re testing as your control scenario.

If you’re testing a web page, this is the unaltered page as it exists already. If you’re testing a landing page, this would be the landing page design and copy you would normally use.

From there, build a challenger — the altered website, landing page, or email that you’ll test against your control.

For example, if you’re wondering whether adding a testimonial to a landing page would make a difference in conversions, set up your control page with no testimonials. Then, create your challenger with a testimonial.

4. Split your sample groups equally and randomly.

For tests where you have more control over the audience — like with emails — you need to test with two or more equal audiences to have conclusive results.

How you do this will vary depending on the A/B testing tool you use. Suppose you’re a HubSpot Enterprise customer conducting an A/B test on an email , for example.

HubSpot will automatically split traffic to your variations so that each variation gets a random sampling of visitors.

5. Determine your sample size (if applicable).

How you determine your sample size will also vary depending on your A/B testing tool, as well as the type of A/B test you’re running.

If you’re A/B testing an email, you’ll probably want to send an A/B test to a subset of your list large enough to achieve statistically significant results.

Eventually, you’ll pick a winner to send to the rest of the list. (See “The Science of Split Testing” ebook at the end of this article for more.)

If you’re a HubSpot Enterprise customer, you’ll have some help determining the size of your sample group using a slider.

It’ll let you do a 50/50 A/B test of any sample size — although all other sample splits require a list of at least 1,000 recipients.

If you’re testing something that doesn’t have a finite audience, like a web page, then how long you keep your test running will directly affect your sample size.

You’ll need to let your test run long enough to obtain a substantial number of views. Otherwise, it will be hard to tell whether there was a statistically significant difference between variations.

6. Decide how significant your results need to be.

Once you’ve picked your goal metric, think about how significant your results need to be to justify choosing one variation over another.

Statistical significance is a super important part of the A/B testing process that’s often misunderstood. If you need a refresher, I recommend reading this blog post on statistical significance from a marketing standpoint.

The higher the percentage of your confidence level, the more sure you can be about your results. In most cases, you’ll want a confidence level of 95% minimum, especially if the experiment was time-intensive.

However, sometimes, it makes sense to use a lower confidence rate if the test doesn’t need to be as stringent.

Matt Rheault , a senior software engineer at HubSpot, thinks of statistical significance like placing a bet.

What odds are you comfortable placing a bet on? Saying, “I’m 80% sure this is the right design, and I’m willing to bet everything on it,” is similar to running an A/B test to 80% significance and then declaring a winner.

Rheault also says you’ll likely want a higher confidence threshold when testing for something that only slightly improves the conversion rate. Why? Because random variance is more likely to play a bigger role.

“An example where we could feel safer lowering our confidence threshold is an experiment that will likely improve conversion rate by 10% or more, such as a redesigned hero section,” he explained.

“The takeaway here is that the more radical the change, the less scientific we need to be process-wise. The more specific the change (button color, microcopy, etc.), the more scientific we should be because the change is less likely to have a large and noticeable impact on conversion rate,” Rheault says.

7. Make sure you're only running one test at a time on any campaign.

Testing more than one thing for a single campaign can complicate results.

For example, if you A/B test an email campaign that directs to a landing page while you’re A/B testing that landing page, how can you know which change increased leads?

During the A/B Test

Let's cover the steps to take during your A/B test.

8. Use an A/B testing tool.

To do an A/B test on your website or in an email, you’ll need to use an A/B testing tool.

If you’re a HubSpot Enterprise customer, the HubSpot software has features that let you A/B test emails ( learn how here ), CTAs ( learn how here ), and landing pages ( learn how here ).

For non-HubSpot Enterprise customers, other options include Google Analytics , which lets you A/B test up to 10 full versions of a single web page and compare their performance using a random sample of users.

9. Test both variations simultaneously.

Timing plays a significant role in your marketing campaign’s results, whether it’s the time of day, day of the week, or month of the year.

If you were to run version A for one month and version B a month later, how would you know whether the performance change was caused by the different design or the different month?

When running A/B tests, you must run the two variations simultaneously. Otherwise, you may be left second-guessing your results.

The only exception is if you’re testing timing, like finding the optimal times for sending emails.

Depending on what your business offers and who your subscribers are, the optimal time for subscriber engagement can vary significantly by industry and target market.

10. Give the A/B test enough time to produce useful data.

Again, you’ll want to make sure that you let your test run long enough to obtain a substantial sample size. Otherwise, it’ll be hard to tell whether the two variations had a statistically significant difference.

How long is long enough? Depending on your company and how you execute the A/B test, getting statistically significant results could happen in hours... or days... or weeks.

A big part of how long it takes to get statistically significant results is how much traffic you get — so if your business doesn’t get a lot of traffic to your website, it’ll take much longer to run an A/B test.

Read this blog post to learn more about sample size and timing .

11. Ask for feedback from real users.

A/B testing has a lot to do with quantitative data... but that won’t necessarily help you understand why people take certain actions over others. While you’re running your A/B test, why not collect qualitative feedback from real users?

A survey or poll is one of the best ways to ask people for their opinions.

You might add an exit survey on your site that asks visitors why they didn’t click on a certain CTA or one on your thank-you pages that asks visitors why they clicked a button or filled out a form.

For example, you might find that many people clicked on a CTA leading them to an ebook, but once they saw the price, they didn’t convert.

That kind of information will give you a lot of insight into why your users behave in certain ways.

After the A/B Test

Finally, let's cover the steps to take after your A/B test.

12. Focus on your goal metric.

Again, although you’ll be measuring multiple metrics, focus on that primary goal metric when you do your analysis.

For example, if you tested two variations of an email and chose leads as your primary metric, don’t get caught up on click-through rates.

You might see a high click-through rate and poor conversions, in which case you might choose the variation that had a lower click-through rate in the end.

13. Measure the significance of your results using our A/B testing calculator.

Now that you’ve determined which variation performs the best, it’s time to determine whether your results are statistically significant. In other words, are they enough to justify a change?

To find out, you’ll need to conduct a test of statistical significance. You could do that manually, or you could just plug in the results from your experiment to our free A/B testing calculator . (The calculator comes as part of our free A/B testing kit.)

You’ll be prompted to input your result into the red cells for each variation you tested. The template results are for either “Visitors” or “Conversions.” However, you can customize these headings for other types of results.

You’ll then see a series of automated calculations based on your inputs. From there, the calculator will determine statistical significance.

14. Take action based on your results.

If one variation is statistically better than the other, you have a winner. Complete your test by disabling the losing variation in your A/B testing tool.

If neither variation is significant, the variable you tested didn’t impact results, and you’ll have to mark the test as inconclusive. In this case, stick with the original variation or run another test.

You can use failed data to help you figure out a new iteration on your new test.

While A/B tests help you impact results on a case-by-case basis, you can also apply the lessons you learn from each test to future efforts.

For example, suppose you’ve conducted A/B tests in your email marketing and have repeatedly found that using numbers in email subject lines generates better clickthrough rates. In that case, consider using that tactic in more of your emails.

15. Plan your next A/B test.

The A/B test you just finished may have helped you discover a new way to make your marketing content more effective — but don’t stop there. There’s always room for more optimization.

You can even try conducting an A/B test on another feature of the same web page or email you just did a test on.

For example, if you just tested a headline on a landing page, why not do a new test on the body copy? Or a color scheme? Or images? Always keep an eye out for opportunities to increase conversion rates and leads.

You can use HubSpot’s A/B Test Tracking Kit to plan and organize your experiments.

Download This Template Now

As a marketer, you know the value of automation. Given this, you likely use software that handles the A/B test calculations for you — a huge help. But, after the calculations are done, you need to know how to read your results. Let’s go over how.

1. Check your goal metric.

The first step in reading your A/B test results is looking at your goal metric, which is usually conversion rate.

After you’ve plugged your results into your A/B testing calculator, you’ll get two results for each version you’re testing. You’ll also get a significant result for each of your variations.

2. Compare your conversion rates.

By looking at your results, you’ll likely be able to tell if one of your variations performed better than the other. However, the true test of success is whether your results are statistically significant.

For example, variation A had a 16.04% conversion rate. Variation B had a 16.02% conversion rate, and your confidence interval of statistical significance is 95%.

Variation A has a higher conversion rate, but the results are not statistically significant, meaning that variation A won’t significantly improve your overall conversion rate.

3. Segment your audiences for further insights.

Regardless of significance, it’s valuable to break down your results by audience segment to understand how each key area responded to your variations. Common variables for segmenting audiences are:

• Visitor type, or which version performed best for new visitors versus repeat visitors.
• Device type, or which version performed best on mobile versus desktop.
• Traffic source, or which version performed best based on where traffic to your two variations originated.

Let’s go over some examples of A/B experiments you could run for your business.

We’ve discussed how A/B tests are used in marketing and how to conduct one — but how do they actually look in practice?

As you might guess, we run many A/B tests to increase engagement and drive conversions across our platform. Here are five examples of A/B tests to inspire your own experiments.

1. Site Search

Site search bars help users quickly find what they’re after on a particular website. HubSpot found from previous analysis that visitors who interacted with its site search bar were more likely to convert on a blog post.

So, we ran an A/B test to increase engagement with the search bar.

In this test, search bar functionality was the independent variable, and views on the content offer thank you page was the dependent variable. We used one control condition and three challenger conditions in the experiment.

The search bar remained unchanged in the control condition (variant A).

In variant B, the search bar was larger and more visually prominent, and the placeholder text was set to “search by topic.”

Variant C appeared identical to variant B but only searched the HubSpot Blog rather than the entire website.

In variant D, the search bar was larger, but the placeholder text was set to “search the blog.” This variant also searched only the HubSpot Blog.

We found variant D to be the most effective: It increased conversions by 3.4% over the control and increased the percentage of users who used the search bar by 6.5%.

2. Mobile CTAs

HubSpot uses several CTAs for content offers in our blog posts, including ones in the body of the post as well as at the bottom of the page. We test these CTAs extensively to optimize their performance.

We ran an A/B test for our mobile users to see which type of bottom-of-page CTA converted best.

We altered the design of the CTA bar for our independent variable. Specifically, we used one control and three challengers in our test. For our dependent variables, we used pageviews on the CTA thank you page and CTA clicks.

The control condition included our normal placement of CTAs at the bottom of posts. In variant B, the CTA had no close or minimize option.

In variant C, mobile readers could close the CTA by tapping an X icon. Once it was closed out, it wouldn’t reappear.

In variant D, we included an option to minimize the CTA with an up/down caret.

Our tests found all variants to be successful. Variant D was the most successful, with a 14.6% increase in conversions over the control. This was followed by variant C with an 11.4% increase and variant B with a 7.9% increase.

3. Author CTAs

In another CTA experiment, HubSpot tested whether adding the word “free” and other descriptive language to author CTAs at the top of blog posts would increase content leads.

Past research suggested that using “free” in CTA text would drive more conversions and that text specifying the type of content offered would help SEO .

In the test, the independent variable was CTA text, and the main dependent variable was conversion rate on content offer forms.

In the control condition, the author CTA text was unchanged (see the orange button in the image below).

In variant B, the word “free” was added to the CTA text.

In variant C, descriptive wording was added to the CTA text in addition to “free.”

Interestingly, variant B saw a loss in form submissions, down by 14% compared to the control. This was unexpected, as including “free” in content offer text is widely considered a best practice.

Meanwhile, form submissions in variant C outperformed the control by 4%. It was concluded that adding descriptive text to the author CTA helped users understand the offer and thus made them more likely to download.

4. Blog Table of Contents

To help users better navigate the blog, HubSpot tested a new Table of Contents (TOC) module. The goal was to improve user experience by presenting readers with their desired content more quickly.

We also tested whether adding a CTA to this TOC module would increase conversions.

The independent variable of this A/B test was the inclusion and type of TOC module in blog posts. The dependent variables were conversion rate on content offer form submissions and clicks on the CTA inside the TOC module.

The control condition did not include the new TOC module — control posts either had no table of contents or a simple bulleted list of anchor links within the body of the post near the top of the article (pictured below).

In variant B, the new TOC module was added to blog posts. This module was sticky, meaning it remained onscreen as users scrolled down the page. Variant B also included a content offer CTA at the bottom of the module.

Variant C included an identical module to variant B but with the CTA removed.

Both variants B and C did not increase the conversion rate on blog posts. The control condition outperformed variant B by 7% and performed equally with variant C.

Also, few users interacted with the new TOC module or the CTA inside the module.

5. Review Notifications

To determine the best way of gathering customer reviews, we ran a split test of email notifications versus in-app notifications.

Here, the independent variable was the type of notification, and the dependent variable was the percentage of those who left a review out of all those who opened the notification.

In the control, HubSpot sent a plain text email notification asking users to leave a review. In variant B, HubSpot sent an email with a certificate image including the user’s name.

For variant C, HubSpot sent users an in-app notification.

Ultimately, both emails performed similarly and outperformed the in-app notifications. About 25% of users who opened an email left a review versus the 10.3% who opened in-app notifications.

Users also opened emails more often.

10 A/B Testing Tips From Marketing Experts

I spoke to nine marketing experts from across disciplines to get their tips on A/B testing.

1. Clearly define your goals and metrics first.

“In my experience, the number one tip for A/B testing in marketing is to clearly define your goals and metrics before conducting any tests,” says Noel Griffith, CMO at SupplyGem .

Griffith explains that this means having a solid understanding of what you want to achieve with your test and how you will measure its success. This matters because, without clear goals, it’s easy to get lost in the data and draw incorrect conclusions.

For example, Griffith says, if you’re testing two different email subject lines, your goal could be to increase open rates.

“By clearly defining this goal and setting a specific metric to measure success (e.g., a 10% increase in open rates), you can effectively evaluate the performance of each variant and make data-driven decisions,” says Griffith.

Aside from helping you focus your testing efforts, Noel explains that having clear goals also means you can accurately interpret the results and apply them to improve your marketing strategies.

2. Test only ONE thing during each A/B test.

“This is the most important tip for A/B marketing from my perspective... Always decide on one thing to test for each individual A/B test,” says Hanna Feltges , growth marketing manager at Niceboard .

For example, when A/B testing button placement in emails, Feltges makes sure the only difference between these two emails is the button placement.

No difference should be in the subject line, copy, or images, as this could skew the results and make the test invalid.

Feltges applies the same principle to metrics by choosing one metric to evaluate test results

“For emails, I will select a winner based on a predefined metric, such as CTR, open rate, reply rate, etc. In my example of the button placement, I would select CTR as my deciding metric and evaluate the results based on this metric,” Feltges says.

3. Start with a hypothesis to prove or disprove.

Another similarly important tip for A/B testing is to start with a hypothesis. The goal of each A/B test is then to prove the hypothesis right or wrong, Feltges notes.

For example, Feltges poses testing two different subject lines for a cold outreach email. Her hypothesis here is:

“Having a subject line with the prospect’s first name will lead to higher open rates than a subject line without the prospect’s first name,” she says.

Now, she can run multiple tests with the same hypothesis and can then evaluate if the statement is true or not.

Feltges explains that the idea here is that marketers often draw quick conclusions from A/B tests, such as “Having the first name in the subject line performs better.” But that is not 100% true.

A/B tests are all about being precise and specific in the results.

4. Track key test details for accurate planning and analysis.

“I keep a running log of how long my A/B tests for SEO took, and I make sure to track critical metrics like the statistical significance rate that was reached,” says NamePepper Founder Dave VerMeer.

VerMeer explains that the log is organized in a spreadsheet that includes other columns for things like:

• The type of test.
• Details about what was tested.

“If I notice any factors that could have influenced the test, I note those as well,” he adds. Other factors could be a competitor having a special event or something that happened in the news and caused a traffic spike.

“I check the log whenever I’m planning a series of A/B tests. For example, it lets me see trends and forecast how the seasonality may affect the test period lengths. Then I adjust the test schedule accordingly,” VerMeer says.

According to VerMeer, this form of tracking is also helpful for setting realistic expectations and providing clues as to why a test result did or didn’t match up with past performance.

5. Test often…

When I spoke to Gabriel Gan, head of editorial for In Real Life Malaysia , for my guide on running an email marketing audit , he set out two main rules for A/B testing.

For the A/B testing email, Gan recommends setting email A as the incumbent and email B as the contender.

Like Hanna, Gabriel emphasizes changing only one variable at a time. “For example, in email B, when testing open rates, only tweak the subject line and not the preview,” says Gan.

That’s because if you have more than one variable changed from the old email, “it’s almost impossible to determine which new addition you made has contributed to the improvement in OPR/CTR.”

Aside from only changing one variable at a time, Gan recommends testing often until you find out what works and what doesn’t.

“There’s a perception that once you set up your email list and create a template for your emails, you can ‘set it and forget it.’” Gan says. “But now, with the power of A/B testing, with just a few rounds of testing your headlines, visuals, copy, offer, call-to-action, etc., you can find out what your audience loves, do more of it, and improve your conversion rates twofold or threefold.”

6. …But don’t feel like you need to test everything.

“My top tip for A/B testing is only to use it strategically,” says Joe Kevens , director of demand generation at PartnerStack and the founder of B2B SaaS Reviews .

Kevens explains that “strategically” means that only some things warrant an A/B test due to the time and resources it consumes.

“I’ve learned from experience that testing minor elements like CTA button colors can be a waste of time and effort (unless you work at Amazon or some mega-corporation that gets a gazillion page visits, and a minor change can make a meaningful impact),” Kevens says.

Kevens recommends that instead, it’s more beneficial to concentrate on high-impact areas such as homepage layouts, demo or trial pages, and high-profile marketing messages.

That’s because these elements have a better shot to impact conversion rates and overall user experience.

Kevens reminds us that “A/B testing can be powerful, but its effectiveness comes from focusing on changes that can significantly impact your business outcomes.”

7. Use segmentation to micro-identify winning elements.

“When using A/B testing in marketing, don’t limit your target audience to just one set of parameters,” says Brian David Crane , founder and CMO of Spread Great Ideas .

Crane recommends using criteria like demographics, user behavior, past interactions, and buying history to experiment with A/B testing of these different segments. You can then filter the winning strategy for each segment.

“We use core metrics like click-through rates, bounce rates, and customer lifetime value to identify the combination that converts the most,” explains Crane.

8. Leverage micro-conversions for granular insights.

“I know that it’s common to focus on macro-conversions, such as sales or sign-ups, in A/B testing. However, my top tip is to also pay attention to micro-conversions,” says Laia Quintana , head of marketing and sales at TeamUp .

Quintana explains that micro-conversions are smaller actions users take before completing a macro-conversion.

They could be actions like clicking on a product image, spending a certain amount of time on a page, or watching a promotional video.

But why are these micro-conversions important? Quintana states, “They provide granular insights into user behavior and can help identify potential roadblocks in the conversion path.”

For example, if users spend a lot of time on a product page but do not add items to their cart, there might be an issue with the page layout or information clarity.

By A/B testing different elements on the page, you can identify and rectify these issues to improve the overall conversion rate.

“Moreover, tracking micro-conversions allows you to segment your audience more effectively. You can identify which actions are most indicative of a user eventually making a purchase and then tailor your marketing efforts to encourage those actions. This level of detail in your A/B testing can significantly enhance the effectiveness of your marketing strategy,” says Quintana.

9. Running LinkedIn Ads? Start with five different versions and A/B test them.

“A best practice when running LinkedIn Ads is to start a campaign with five different versions of your ad,” says Hristina Stefanova , head of marketing operations at Goose’n’Moose .

Stefanova reminds us that it’s important to tweak just one variable at a time across each version.

For a recent campaign, Stefanova started with five ad variations — four using different hero images and three having the CTA tweaked.

“I let the campaign run with all five variations for a week. At that point, there were two clearly great performing ads, so I paused the other three and continued running the campaign with the two best-performing ones,” says Stefanova.

According to Stefanova, the two ads performed best and had the lowest CPC. The A/B testing exercise helped not only the specific campaign but also helped her to better understand what attracts their target audience.

So what’s next? “Images with people in them are better received, so for upcoming campaigns, I am focusing right away on producing the right imagery. All backed up by real performance data thanks to A/B testing,” Stefanova says.

10. Running SEO A/B tests? Do this with your test and control group URLs.

“Given that the SEO space is constantly evolving, it’s getting increasingly difficult to run any sort of experiments and get reliable and statistically significant results. This is especially true when running SEO A/B tests,” says Ryan Jones , marketing manager at SEOTesting .

Luckily, Jones explains that you can do things to mitigate this and make sure that any SEO A/B tests you run now — and in the future — are reliable. You can then use the tests as a “North Star” when making larger-scale changes to your site.

“My number one tip would be to ensure that your control group and test group of URLs contain as identical URLs as you can make them. For example, if you’re running an A/B test on your PLP pages as an ecommerce site, choose PLPs from the same product type and with the same traffic levels. This way, you can ensure that your test data will be reliable,” says Jones.

Why does this matter? “Perhaps the number one thing that ‘messes’ with A/B test data is control and variant groups that are too dissimilar.

But by ensuring you are testing against statistically similar URLs, you can mitigate this better than anything else,” Jones says.

Start A/B Testing Today

A/B testing allows you to get to the truth of what content and marketing your audience wants to see. With HubSpot’s Campaign Assistant , you’ll be able to generate copy for landing pages, emails, or ads that can be used for A/B testing.

Learn how to best carry out some of the steps above using the free ebook below.

Editor's note: This post was originally published in May 2016 and has been updated for comprehensiveness.

Don't forget to share this post!

Related articles.

How to Determine Your A/B Testing Sample Size & Time Frame

How The Hustle Got 43,876 More Clicks

What Most Brands Miss With User Testing (That Costs Them Conversions)

Multivariate Testing: How It Differs From A/B Testing

How to A/B Test Your Pricing (And Why It Might Be a Bad Idea)

11 A/B Testing Examples From Real Businesses

15 of the Best A/B Testing Tools for 2024

These 20 A/B Testing Variables Measure Successful Marketing Campaigns

How to Understand & Calculate Statistical Significance [Example]

What is an A/A Test & Do You Really Need to Use It?

Learn more about A/B and how to run better tests.

Marketing software that helps you drive revenue, save time and resources, and measure and optimize your investments — all on one easy-to-use platform

• Project management

5 strategies to improve your marketing team’s project management

October 09, 2023

Effective project management is a cornerstone of success for marketing teams. It’s about more than managing tasks. It’s a compound approach that includes collaboration, motivation, productivity, priorities, experiments, learning, and more.

If done correctly, marketing team project management drives performance and marketing strategy execution. But how can we achieve it? In this article, we’ll explore the importance of project management in marketing and strategies for achieving marketing team success. Let’s dive in!

What is marketing team project management?

Marketing team project management involves planning, organizing, and controlling resources to achieve specific marketing goals within a set time frame. It also manages risks and issues that may arise during the project lifecycle, correcting the strategy if needed. 

It is one of the most effective structured approaches for successful project completion . In a nutshell, it’s about guiding a team to reach a particular marketing goal, like launching a new product, executing a content strategy, or hosting an event.

A marketing project manager, for example, a team lead, CMO, or head of marketing, is usually responsible for initiating and leading marketing project management. This ensures marketing projects meet deadlines, budgets, and goals. 

According to a study , organizations that don’t value project management as a strategic competency for driving change experience a 67% higher project failure rate. It also found that projects are two and a half times more successful and 13 times less likely to waste money when implementing project management practices.

These numbers underscore the significance of efficient marketing team project management. But let’s review its value from a broader perspective.

Importance of marketing team project management

Marketing project management serves as a proactive shield against cascading effects within team projects: identifying issues, managing resources, mitigating risks , and establishing precise task dependencies to prevent setbacks. 

At the same time, there are four key impacts of marketing team project management.

Marketing planning

Detailed planning sets the foundation for a successful project. Confidence in the project’s success is directly proportional to the level of detail and thought in the planning stage. With proper project management, marketing planning gets a backlog of activities and tasks, explicit descriptions, deadlines, priorities, and implementation steps.

Alignment with business strategy

Alignment with business goals is a critical aspect of project management. It ensures that all marketing activities are strategically designed to support and achieve the business’s objectives. It’s about creating a cohesive, focused, and practical approach to marketing. In practice, the lack of clearly defined objectives is one of the most prevalent reasons for project failures.

Marketing team efficiency

In marketing, teamwork is crucial, much like the harmony in a symphony orchestra. Effective marketing team project management ensures that each team member has a scope of projects.

With each project, team members improve their skills and understanding through cross-team communication , knowledge sharing, and experiments. It leads to fruitful teamwork for the whole group and its leader. 

Marketing consistency

Marketing consistency means that all marketing initiatives align with the overall strategy, ensuring the brand message remains consistent across all channels. 

The project management for marketing methodology keeps marketing campaigns on track and stakeholders informed throughout the project lifecycle. 

This consistency strengthens the brand’s identity and brand awareness . It also ensures that all team members are on the same page, reducing confusion.

Key challenges of marketing project management

To effectively use marketing team project management, you must anticipate potential issues. Among various challenges, there are four main.

Scope creep

Scope creep is when the scope of a project increases over time, often without the necessary resources or budget. It can lead to delays, missed deadlines, and poor-quality work. Scope creep can occur when the project scale is poorly defined, documented, or controlled. To fix scope creep , the project manager identifies its root cause, communicates with stakeholders about potential risks, establishes a change process, and monitors the project scope for corrective action. 

Team collaboration

Marketing teams often consist of people with different skills and backgrounds. This can make it challenging to ensure all team members work towards the same goals. Following this, it’s crucial to establish clear roles and responsibilities and hold regular team meetings to keep everyone on track.

Marketing attribution

Marketing attribution assigns credit to different marketing channels in driving conversions or sales. However, it can be challenging to track the results of marketing campaigns and determine which channels drive the most results. 

Therefore, marketing project managers must understand the marketing attribution models to allocate their budget and resources to the most effective channels and tactics.

Concerning this matter, enabling comprehensive marketing analytics is a must. Marketing managers must gather and monitor marketing data. Using analytics tools and marketing dashboard examples , you can solve marketing attribution challenges and, as a result, optimize project management for marketing.

Marketing projects can be expensive, so it is essential to demonstrate their return on investment. This can be tough, especially for big or complicated projects, because it’s hard to trace how each marketing action directly affects revenue. 

To overcome this challenge, marketing analytics again comes in handy. It helps you to track your progress and make adjustments on the go. Before starting a project, it is crucial to establish clear metrics and KPIs as well. The final point here is a careful budget and resource allocation for marketing campaigns and experiments.

How to improve your marketing project management

Reapproaching your marketing team’s project management strategies can be a game-changer, improving productivity, efficiency, and results. 

It can be demanding but worth it in the long run. Following the recommendations below, you can develop a system to help your team and improve the overall PM processes.

Lean marketing

Based on lean manufacturing principles, this philosophy emphasizes continuous improvement, efficiency, agility, and quick action.

Lean marketing is an Agile approach to marketing activities that perfectly fits the current dynamic business environment. It lets you focus on the most significant activities and eliminate/postpone less important ones. 

In parallel, the approach encourages marketing project managers to emphasize customer-centric projects. The approach involves running quick experiments with campaigns and testing hypotheses. The goal is to learn from these experiments and make necessary adjustments.

To effectively implement lean marketing in your department, you should consider:

• Agile techniques. Lean marketing teams can adapt quickly to market changes and customer needs. You can achieve this by using short feedback loops and iterative processes.
• Quick actions. With project prioritization embraced by lean marketing principles, you can shape your scope and take further steps structurally. Also, you should consider training marketing specialists to be decisive and to be able to work independently.
• Customer-centric schemes. You can conduct regular customer development research, conjoint analysis , revise user personas, and analyze competitors. It gives you an understanding of your customers and potential leads; based on this knowledge, you can polish your marketing projects.
• Experiments and hypotheses testing. Lean marketing gives you space for experiments and testing, though you need to do them quickly. It’s possible through epic decomposition and the use of short, iterative test cycles.

As a result, lean marketing principles can impact project accomplishment and boost team productivity and adaptability. Particularly, it is beneficial when shaping a marketing strategy and roadmap.

Flat team structure

A flat team structure has few or no hierarchy levels at all. This type of structure is well-aligned with lean marketing principles. 

A marketing team consists of different roles, but they all work towards the same strategy. They work independently, but everyone can collaborate freely with each other, including the head of marketing, who also does hands-on tasks. 

Each team member, from marketing interns to senior marketers, have key responsibilities and projects. They each make intervening practical decisions, while strategic decisions are discussed within the team. 

The benefits of a flat team structure include:

• Improved communication and collaboration. There are fewer communication barriers, and team members are more likely to feel empowered to share their ideas.
• Increased flexibility and adaptability. Team members can make decisions and take action without going through multiple levels of approval. 
• Improved employee motivation. With a flat team approach, team members feel more valued and respected, and their expertise is utilized. It may impact proactivity and interest in new tasks, projects, and experiments.

A flat team structure can be highly beneficial in project management for marketing. As a result, you’ll have a team of T-shaped specialists with a wide range of skills and deep expertise in one area.

Data-driven decisions

In marketing, you should ground even seemingly minor decisions in data. A data-driven approach offers greater accuracy and effectiveness than decisions based on intuition.

To adopt a data-driven strategy, you should delve into comprehensive studies of users, markets, competitors, previous marketing campaigns, and experiments’ outcomes. By harnessing these records, you can shape hypotheses and adjust both your strategy and ongoing projects.

Moreover, each marketing channel demands its dedicated analysis. So, each marketer should evaluate and share the outcomes of their own projects. It’ll help to prioritize tasks, improve processes, or even postpone or decline some projects due to inefficiency. 

Also, a well-structured analytics and reporting approach will enable you to build a data-driven culture within a team and navigate the complex marketing landscape. 

Marketing goals setting

Your marketing goals, setting, and planning should include several key elements.

Firstly, marketing goal setting should involve forming both realistic and optimistic goals. You can use SMART goals . It’ll provide clarity in goal setting and should motivate the team.

Secondly, you should align marketing goals with overall business goals. It’s critical for supporting your company’s growth. Interim marketing goals have to follow the marketing strategy and roadmap.

Thirdly, when planning marketing activities, adopting a customer-centric approach is paramount. Prioritizing your customers’ needs and preferences lets you focus on what truly matters to them. This perspective fosters stronger customer relationships and brand loyalty.

Lastly, being consistent in setting and pursuing marketing goals is vital to long-term marketing success. With lean marketing, you can adjust objectives promptly, but the core canvas should remain consistent. It will help team members to focus better and achieve better results in the long run.

With goal setting, there’s planning and monitoring. Goals are typically set monthly, with bi-weekly review sessions. It is a way for team members to get results, re-organize activity, and discuss further steps. 

When marketing planning is done in a team meeting, all team members understand their colleagues’ work scope, and there’s a space for knowledge sharing and brainstorming.

Collaboration with a product team

Collaboration with the product team is about working closely for successful product launches and campaigns. It means talking often, knowing who does what, and sharing the same product vision.

By working together, the two teams can ensure that the products and services meet the target market’s needs. 

By collaborating with the marketing team, product teams can gain valuable insights into the needs and wants of the target market. This information can be a source to develop product features and services. 

Meanwhile, marketing teams can better understand the products and services they promote when they connect with product teams. Besides, notes for upcoming product updates shared by a product lead or owner can help marketers better plan their future campaigns.

You can use project management software to establish proper collaboration between teams.  One such tool worth considering is Backlog by Nulab. Backlog offers Kanban boards , task tracking, Gantt charts, and other features to streamline project management effectively.

Key highlights

In conclusion, revamping your marketing team’s project management can significantly improve your team’s performance and results. Here are the key takeaways:

• Lean marketing. Stay agile and customer-centric, run quick experiments, and test hypotheses.
• Flat team structure. Foster a collaborative environment where everyone contributes, shares ideas, and is self-managed.
• Data-driven decisions . Make marketing decisions based on data rather than intuition.
• Marketing goals setting. Set realistic and optimistic, monitor and adjust them quickly.
• Collaboration with a product team. Work closely with the product team for successful product launches and campaigns.

Remember, adaptability and constant improvement are key to successful marketing team project management. Don’t be afraid to try new strategies and adjust your approach based on the results. Start implementing these tactics and witness your team thrive!

Dmytro Zaichenko is a Marketing Specialist at Coupler.io , an all-in-one data analytics and automation platform. He has over three years of experience in digital marketing, particularly in SaaS. Apart from experimenting with marketing tactics, he’s a huge NBA fan.

The project manager’s guide to activity sequencing 

Best project management software for small teams

Subscribe to our newsletter.

Learn with Nulab to bring your best ideas to life

Oct 8, 2020

How to run marketing experiments: practical lessons from four marketing leaders

14-MINUTE READ | By Pinja Virtanen

[ Updated Oct 8, 2020 ]

“Marketing is one big experiment. Some experiments just have a longer shelf life than others.”

That’s the first thing Andy Culligan, CMO of Leadfeeder, said to me when I asked him about experimentation in marketing.

And to help you understand how to run those experiments, I interviewed Andy and three other seasoned marketing leaders. 

After reading this article, you’ll know exactly:

• Why you absolutely should run experiments in marketing
• What’s stopping marketers from experimenting — and how to overcome those challenges
• What all good marketing experiments have in common
• Which 7 steps you should include in your experimentation process
• Examples of 4 successful experiments

Ready? Let’s go!

Should more marketers run experiments? And why?

It’ll probably come as no surprise to you that when asked whether more marketers should run experiments, all four of our experts came back with some variation of “hell yes”.

Mari Luukkainen, Head of Growth at Icebreaker.vc , explains that since she has a background in affiliate marketing, a data-driven iteration with the goal of business growth is simply the only type of marketing that makes sense to her.

She says, “To figure out what works and what doesn’t to grow your business, you need experimentation. There’s no point for a marketing team or even a business function that isn’t running experiments to find a better, faster, or a more optimized way to grow their area of the business.”

Both Michael Hanson, Founder & Sales Consultant at Growth Genie , and Andy from Leadfeeder bank on experimentation because the market is a moving target.

Michael explains, “If you don’t run tests in marketing, you’re always going to fail. What worked last year, won’t necessarily work so well this year. Take organic Facebook for example. 10 years ago, you could get good reach just by posting from your company account. But it doesn’t work like that anymore. And so if you don’t constantly measure performance and try to improve, you’re definitely going to fail.”

Andy adds, “Everything you do in marketing is a test anyway. Some tests work longer than others but the point is, you need to help your marketing evolve.”

And finally Mikko Piippo, Founder & Digital Analytics & Optimization Consultant at Hopkins , argues that experiments are great for reducing bias.

Mikko says, “Everyone has an opinion, and sometimes expert opinion is not worth much more than tossing a coin. Experiments are the best way to systematically create new knowledge, to learn more about your audience and your customers. They force you to question your own ideas, beliefs, and the best practices you’ve read so much about. This can be somewhat uncomfortable if the data doesn’t support your own ideas.”

And now that the jury has reached a unanimous verdict, let’s move on to the next big question, i.e. what’s stopping marketers from experimenting.

So why isn’t everyone and their grandma already running experiments?

According to Mari, the biggest problem isn’t that marketers don’t want to experiment, it’s that they’re working towards the wrong goals, lacking the routine, and/or afraid of failing.

She explains, “Far too many marketing teams are still struggling to set goals that directly correlate with business performance. But as soon as you set goals that make sense for the business, you’ll start systematically working towards them. And that’s when you’ll need to start experimenting.”

Working with larger corporations, Mari has also noticed that sometimes the company culture works against a fundamental part of experimentation: failure.

Mari says, “The other big issue can be that marketers are so afraid of failing that they won’t feel comfortable trying anything new. In some corporations, failing means that you’ll get fired. That’s when there’s no incentive for marketers to run experiments.”

If you recognize any of these problems in your organization, Mari offers three alternatives to overcoming the issues:

• Get buy-in for experimentation from as high up the organizational ladder as possible (investors, the board of directors, or the management team)
• Start small and prove the value of experimentation with small wins (the problem with this approach is that it can be painfully slow)
• Replace your team with people who know exactly how to run experiments (this is quick but can be a painful process)

And now, with any possible obstacles out of the way, let’s look at what a good experiment looks like.

The 3 things all good marketing experiments have in common

According to our four experts, all marketing experiments should have these three things in common.

1. They’re systematic and measured with data

The first rule of experimentation is that you have to stick to a process and make sure to use data to determine how successful the experiment was.

Mari says, “All good experiments are systematic and measured with data.”

Mikko follows with, “Good experiments follow a plan or a process. In a bad experiment, for example, a marketer would set a goal only after seeing the metrics.”

To summarize, a systematic process and a healthy relationship with data are what ultimately make or break an experiment.

Like Michael says, “If you’re going to test something, you need to measure its success. Otherwise it’s not really an experiment, is it?”

2. They’re big enough (but not too big)

The other, perhaps a more controversial, requirement for a good experiment is that it’s big enough. In other words, yes, you should absolutely forget about the button color A/B tests of yesteryear.

Mikko says, “Be bold. Test complete landing page redesigns instead of button colors, experiment with product pricing instead of call-to-action microcopy, experiment with different automated advertising strategies instead of tweaking single ads, experiment with budget allocation over different advertising platforms instead of micromanaging individual platforms.”

Because the problem with small tests is that even when they’re successful, they’ll yield small results.

Andy explains, “If you only test one small thing at a time, you’re never going to get big enough of an uplift. So if you do run tests, you need to try something completely different. Throw the existing landing page out the window and try something new instead. If the new version works better, then use that as your benchmark going forward.”

Michael says, “Obviously you don’t want to change the company name or logo every 5 minutes. But beyond that, you have to be flexible with the scope of the experiment.” 

He continues, “I had an ex-colleague who had all these wacky ideas and when we tried them, they always came through. The point is, even though ideally you’d want to test one variable at a time, you also have to realize that the impact of a small change will be small. And if you want quick results, you have to think bigger. So I’m all for experimenting with wacky ideas.”

And with that, the verdict is in: think big when you’re experimenting.

3. They’re run as split tests

The final precondition for a good experiment is that it’s run as a split test. This means that you’re testing one variable against a different one.

Mikko explains, “Good marketing tests are usually split tests. You split the audience (website visitors, advertising audience) into two or more groups. Then you offer different treatments to different groups — and you keep some percentage of the audience separately as a control group. This way, you can really compare the effectiveness of different treatments.”

Mikko also emphasizes that even if you don’t have a ton of media budget or website traffic, you can still run experiments. “With low website traffic, the methods just aren’t as scientific as with high traffic.” The point is: don’t let external variables like low traffic stop you from experimenting.

Bonus: They may or may not have a hypothesis — depending on who you ask

As the bonus criteria for running good experiments, let’s look at the one word that always comes up when we’re talking about experiments: hypothesis.

So, do you need one?

Mikko and his team at Hopkins, for one, are strong believers in setting a hypothesis before running an experiment. Mikko says, “Good marketing experiments start from a hypothesis you try to validate or refute. Actually, without a hypothesis I wouldn’t even call something an experiment. For example, it’s easy to add a couple of more ad versions to an ad set or ad group. Most marketers don’t follow any logic here, they just add some random ad versions. Doing this might improve the results, but they wouldn’t know why.”

He continues, “A hypothesis forces you to think about the experiment: Why do I expect something to change for the better if I change something else? Why would people behave differently?”

Andy, on the other hand, would go easy on the hypothesis setting. He explains, “In my opinion, really analytical marketers like to make experimentation into rocket science and it doesn’t have to be that. I’m data-driven but purely from a revenue perspective. I don’t tend to get too deep into the grass, the weeds, and the bushes. You’re only going to end up in a rabbit hole. If it’s working, it’s working — and that’s all I care about.”

And that’s why, rather than spending a lot of time forming hypotheses, Andy likes to tie the Leadfeeder marketing team’s experiments into their quarterly OKRs. 

For example, if a key result is to increase Leadfeeder’s tracker install rate by 10%, the team will simply come up with a number of changes to get there.

To conclude, whether or not you should set a hypothesis for your experiment depends on this question: will you benefit from knowing the contribution of each individual change?

If the answer is no, you’re in team Andy.

And if the answer is yes, well… Welcome to team Mikko.

And now that we got that out of our systems, let’s look at the steps you need to take to actually run an experiment.

How to run a marketing experiment: step-by-step instructions

Even though there are clearly some things our expert panelists disagree about, the actual experimentation process all four of them follow is pretty uniform.

Step 1: Start by setting (or checking) your goal

The very first step in the experimentation process comes down to understanding what KPI you’re trying to influence. 

For example, if like Andy’s team at Leadfeeder you’re using OKRs, you can use your key results as the goals for your experiments.

So for him, a goal would look something like “increase our tracker installation rate by 10% within the next 3 months.”

Like Andy, you’ll want to make your goal unambiguous and give it a clear timeframe.

Step 2: Analyze historical data

Once you understand what needle you’re trying to move, it’s time to analyze your existing data. Mari suggests that at this point, you “analyze where you are and how you got there”. 

Similarly, Mikko says that for his team, this step involves, “looking at existing data from our ad platforms and web analytics tools.”

Step 3: Come up with ideas

Equipped with your analysis of historical performance, you can probably list a dozen (or more) things that may or may not influence the metric you’re trying to influence.

At this point, your only job is to list those ideas down.

Step 4: Prioritize your ideas

Mari suggests that you prioritize the ideas you came up with based on “resources efficiency, success probability, and scalability.”

Alternatively, you can use a scoring system like ICE , which stands for impact, confidence, and ease.

After this step, you should have a clear idea of which experiments to go after.

Step 5: Run the experiment(s)

Now this one’s a bit of a no-brainer. Now that you know the expected impact of your experiments, it’s time to run the one(s) you think will have the biggest impact.

But can you run multiple experiments at once? Yes and no.

I’ll refer you back to the discussion we had earlier about hypotheses: if you absolutely need to know which tactic was the most successful, you should isolate your experiments and run one at a time.

If, on the other hand, you’re just trying to move the needle as quickly as possible and don’t really care about figuring out correlation and causation, feel free to run multiple experiments at the same time.

Step 6: Measure success

Whether it’s while your experiment is still running or after it has ended, it’s time to look at data. Did the experiment drive the expected results?

Is there anything you can do to optimize it (if it’s still running)?

Psst! This is where Supermetrics for Google Sheets comes in handy: you can automate data refreshes and email alerts to cut down the time you would otherwise have to spend on manually collecting data from multiple platforms.

Step 7: Rinse and repeat

Depending on the scope and results of your experiment, you might want to start from the very beginning, or simply go back to Step 4 and choose new experiments to run off the back of your results.

And finally, if you need any inspiration for your upcoming experiments, keep reading. Because in the next section Mari, Michael, and Andy will spill awesome examples of successful experiments they’ve run.

4 examples of successful marketing experiments

To get you in the mood for planning your own experiments, here are quick examples from our experts.

Freska: experimenting both offline and online

When I asked Mari about the most memorable experiments she ran at Freska, a modern home cleaning service, she came back with two examples.

Mari starts from an offline experiment, “At Freska our hypothesis was that people who have expensive hobbies that consume lots of time would buy home cleaning services. We tested this by going to a boat expo instead of the usual “baby expos” cleaning companies go to. And we ended up getting surprisingly good results.”

The second experiment that has stayed with her was of the online variety. 

Mari says, “At Freska, our original hypothesis was that people are afraid of seeing the price of home cleaning services in ads because it feels kind of expensive. But when we conducted a deeper analysis with offline surveys, people actually thought that home cleaning services are way more expensive (over 1000 €/month) and our price is actually a pleasant surprise (150 €/month for a biweekly 2-3-room apartment). So we tested showing the price in ads and it actually increased conversions.”

Growth Genie: building the perfect outbound cadence one iteration at a time

Michael’s favorite experiment to date is of a more persistent kind. Over time, his team has perfected the art and science of cold outreach , one iteration at a time.

Michael says, “It’s all about cadences here at Growth Genie; how many calls, how many emails, and how many LinkedIn messages does it take — and in which order — to book a call with a prospect who’s never heard of you before.”

He continues, “We’ve learned what works simply by experimenting and iterating. For example, instead of asking for a meeting in the first few touchpoints, we quickly noticed that we can get much better results by giving away these valuable content snippets and only then asking for a meeting.”

Psst! If you’re interested in absorbing the TL;DR version of Michael’s learnings, check out his recent LinkedIn post below.

And if you want to swipe Growth Genie’s “ultimate outbound sales cadence”, you can access it here .

Leadfeeder: pivoting lead generation for the “new normal”

When COVID-19 hit Europe and the US in March 2020, the Leadfeeder team needed to quickly cut their marketing budget by a third to increase runway.

For Andy, that meant figuring out a channel that would quickly generate pipeline without taking a ton of time or budget upfront.

Andy says, “We started pushing up webinars and those exploded. We quickly got 11,000 leads by only spending something like $1,000. But as everyone started doing webinars, the numbers began to drop. And that’s why we decided to start recycling the webinars into short 10-15 minute videos, rebranded as “The B2B Rebellion” series on YouTube .” 

He continues, “The webinars were a test and because they were working, we doubled down, and eventually moved into the video concept. The video concept has been working nicely, and now we’re experimenting with new speakers and distribution channels.”

Overall, this constantly evolving experiment has allowed the Leadfeeder team to maintain their pre-pandemic lead volume at a third of the cost.

Andy says, “Without constant experimentation, you’re not going to win and your marketing will go stale.”

Over to you! ?

What are some of the most successful (or surprising) marketing experiments you’ve run? 

Let me know on Twitter or LinkedIn !

Stay in the loop with our newsletter

Be the first to hear about product updates and marketing data tips

• Product overview
• All features
• Latest feature release
• App integrations

CAPABILITIES

• project icon Project management
• Project views
• Custom fields
• Status updates
• goal icon Goals and reporting
• Reporting dashboards
• asana-intelligence icon Asana AI
• workflow icon Workflows and automation
• portfolio icon Resource management
• Capacity planning
• Time tracking
• my-task icon Admin and security
• Admin console
• Permissions
• list icon Personal
• premium icon Starter
• briefcase icon Advanced
• Goal management
• Organizational planning
• Project intake
• Resource planning
• Product launches
• View all uses arrow-right icon

Featured Reads

• Work management resources Discover best practices, watch webinars, get insights
• Customer stories See how the world's best organizations drive work innovation with Asana
• Help Center Get lots of tips, tricks, and advice to get the most from Asana
• Asana Academy Sign up for interactive courses and webinars to learn Asana
• Developers Learn more about building apps on the Asana platform
• Community programs Connect with and learn from Asana customers around the world
• Events Find out about upcoming events near you
• Partners Learn more about our partner programs
• Asana for nonprofits Get more information on our nonprofit discount program, and apply.
• Project plans
• Team goals & objectives
• Team continuity
• Meeting agenda
• View all templates arrow-right icon
• Marketing |
• Marketing project management: How to st ...

Marketing project management: How to structure your strategy

Marketing project management is a methodology used to keep marketing campaigns on track and stakeholders informed throughout the project lifecycle. It provides clarity among teams, keeps your projects within scope, and helps team members meet customer needs. In this piece, we’ll discuss the challenges of marketing campaigns and explain how marketing project management can help you succeed.

Marketing initiatives can be crucial to your business plan because they give you the chance to tell your brand story and send leads down the sales funnel. Without effective marketing, you may struggle to bring in revenue and secure loyal customers. A focused marketing plan ensures that your message resonates with your audience so you can walk away from every campaign feeling proud of the work your team put in.

What is marketing project management?

Marketing project management is a methodology used to keep marketing campaigns on track and stakeholders informed throughout the project lifecycle. It provides clarity among teams, keeps your projects within scope, and helps you meet customer needs.

To manage marketing projects, you’ll start with the same project management principles that other teams and industries use. But marketing project management differs from the traditional project management approach in a few ways, which we’ll cover below. 

The five project management phases are:

Performance

In marketing project management, you’ll add a marketing strategy phase where you’ll gather market research and data and use your findings to set your project plan in motion.

Why is marketing project management important?

Marketing project management is important because how you manage a project impacts everyone involved with the marketing campaign. When you use the right methodology, others will follow your lead and reap the benefits of your strong leadership. 

Picture yourself at the center of the project. As the circle expands, more people get involved in the project. Once you realize you’re only the first person in the project life cycle, it’s easier to see why project management is so important.

It takes a village to manage a marketing project. The three most important stakeholder groups are:

The marketing project manager:  As the leader and facilitator of marketing campaigns, you’re at the center of everything that happens during a marketing project. This includes things like project timeline delays, email marketing troubleshooting, and KPI monitoring. 

Internal stakeholders: Internal stakeholders are team members within your organization who have a stake in your project. These people may include executives, sales representatives, creatives, or technicians. How you manage your marketing campaign affects internal stakeholders. They’re often either involved with the marketing campaign, impacted by the campaign’s deliverables, or informed about your overall goals.

External stakeholders: External stakeholders are people outside of your organization who have a stake in your project. These people may include vendors, end users, clients, or investors. You’ll need project management skills to keep external stakeholders informed and satisfied with your project deliverables.

10 steps of the marketing project management process

The marketing project management methodology has 10 key steps. While your marketing agency may tackle complex projects in niche areas like SEO or social media, you can use these steps as a general framework for most marketing campaigns.

You can divide the 10 steps below into five project phases. These five phases resemble the traditional project management phases, but they also include additional marketing strategies to ensure you’re setting yourself—and your marketing project—up for success. 

Objectives and analysis

The goal of the objectives and analysis phase of marketing project management is to focus on planning your marketing campaign. This involves defining the project’s end goals and outlining success metrics.

Define end goals: Make your end goals clear at the beginning of every project you work on. That way, team members know what to strive for during project execution and stakeholders know what to expect. 

Identify success metrics : It’s critical to identify KPIs at the beginning of your campaign so you can use these metrics to monitor your progress throughout the project lifecycle. 

Marketing strategy

Use your project objectives from phase one to drive your marketing strategy. During this phase, you’ll also use market research and data to find the most effective way to achieve your strategic goals .

Pinpoint your audience: Identifying your target audience is the first step to achieving a high ROI. Your target audience is the group of people who are most likely to resonate with your brand. If you can reach this audience, you increase your chance of selling your product or service. 

Set message and CTAs: Determine the message you want to send to your target audience. Your message should include strategic calls to action for your product or service. 

Project scheduling

Your marketing campaign may require creative assets and a detailed plan of how and where to distribute these assets. During the project scheduling phase, establish a team to assist you with asset creation. 

Clarify scope: Clarify your project scope so everyone knows the limitations of your project timeline, resources, and budget. It's also important to ensure stakeholders are aware of the project scope to limit change requests.

Delegate tasks : Delegating work is crucial if you hope to stay organized and avoid duplicate work. Create a project timeline and assign tasks to team members. Use a Gantt chart or other task management tool so team members can visualize project milestones and dependencies between tasks. 

Campaign launch

After you’ve scheduled your campaign, the action begins. This is the phase when your team develops your creative assets and sends them out to the masses. This part of marketing project management is exciting because you get to see your strategy in motion. 

Create project deliverables: Produce deliverables that will outshine your competitors’ and wow your audience. Employ a team of writers and graphic designers that can deliver your message using strong copy and impressive visuals. 

Distribute across marketing channels: Determine which marketing channels will help you reach your target audience and when they’re on them. Place your deliverables across these channels so you get as many eyes on them as possible.

Monitor and review

Use the success metrics you set during the project planning phase to monitor your project progress. Once you’ve tracked your progress, you can also  use your performance results to learn lessons for future projects . 

Monitor results: Use project management software to monitor your KPIs in real time. Once you’ve launched your marketing campaign, you can assess how well your campaign performed and what adjustments you should make to your future marketing strategy.

Set future standards: Use any lessons you learn from monitoring your campaign to set standards for future projects. For example, if your campaign performed poorly with a specific age bracket, set audience limitations on this group for future campaigns.  

Common challenges in marketing campaigns

Many marketing teams face challenges when implementing their marketing campaigns. Luckily, the most common challenges are preventable or easily mitigated with marketing project management. 

Use the solutions to the challenges below as part of your marketing project management workflow.

1. Project risks

Marketing campaigns experience risk in many areas, and it’s difficult to predict what these risks will be or when they’ll occur. But if you’re not prepared to mitigate a project risk once it takes hold, the problem can affect project quality. Some common areas of project risks include:

Technical risk: Technical risk can particularly affect email or digital marketing campaigns . Security incidents, cyberattacks, password theft, or service outages could delay a marketing campaign or derail it completely. 

Market risk: These are risks that affect the entire market. These may include risk of recession, margin risk, interest rate risk, and currency risk. While these risks are uncontrollable, your team can prepare for them so you can react quickly if they do happen.

Organizational risk: Organizational risk occurs from issues with internal operations. Events that fall under this category include reputational damage, communications failure, lawsuits, and supply chain disruptions. 

Solution: Use project risk management to prevent and mitigate risk in your marketing campaigns. During the planning phase, set up a risk analysis to assess which project risks are most likely to occur, as well as which risks are of highest priority. Then, use insights to shape your campaign and prepare for potential mishaps. 

2. Scope creep

Scope creep occurs when your marketing campaign expands beyond the initial expectations you set. Marketing campaigns often suffer from scope creep because teams don’t establish clear requirements during project planning. If you don’t communicate your limitations to stakeholders, they may request changes that your project team has trouble keeping up with. 

Solution: Define project objectives during the initial stages of your marketing campaign and share these objectives with your stakeholders. Maintain clear lines of communication so your stakeholders understand your project requirements, including the limits of your project timeline and budget . If necessary, you can also establish a change control process to regulate change requests.  

3. Poor communication with stakeholders

Poor communication with stakeholders is a challenge many marketing teams face . You can see above that this challenge has consequences, with scope creep being just one of those consequences. Other consequences of communication issues include:

Unclear project expectations 

Inconsistencies in goals and results

Reduced team morale

Insufficient project funding

Duplicate work

Solution: Use project management software to establish a strong line of communication with stakeholders. Share real-time updates with everyone involved in your marketing campaign, and encourage stakeholders to provide feedback along the way. Set project milestones as checkpoints for collective evaluation of the campaign.

4. No single source of truth

Marketing teams that rely on face-to-face, email, phone, or video chat to communicate with stakeholders will experience challenges when managing their marketing campaigns. You shouldn’t retire these traditional forms of communication, but they don’t offer essentials like:

Document sharing

Real-time status updates

Software integrations

Task management

Central source of truth

Your marketing strategy should be transparent to all stakeholders. Transparency strengthens team communication and improves project quality.

Solution: Use project management software as your single source of truth. There are many types of project management with varying levels of functionality. Some tools compile your project information, while others compile information from outside sources. Use a tool like Asana to customize project views and keep everyone—from team members to stakeholders—on the same page.

Use project management software to structure your marketing strategy

Marketing project management can eliminate some of the common challenges faced by marketing departments. When you use a structured management methodology, you’ll improve communication flow and streamline your work process. Use project management software to promote collaboration among stakeholders and to establish a single source of truth. 

Related resources

How Asana streamlines strategic planning with work management

Write better AI prompts: A 4-sentence framework

What is content marketing? A complete guide

Smooth product launches are simpler than you think

26 Marketing Experiments That Brands Can Use To Unlock New Insights

• Apr 4, 2019

Marketing experiments differentiate innovative brands from the bland ones.

What is a Marketing Experiment?

A marketing experiment is a strategic approach used by businesses to test different marketing tactics, strategies, and activities in order to discover the most effective ways to reach their target audience, engage potential customers, and improve overall marketing performance. This process involves setting up controlled tests where various elements of marketing campaigns are modified – such as messaging, digital advertising platforms, content formats, or audience segments – to evaluate their impact on defined metrics like engagement rates, conversion rates, or return on investment (ROI).

By systematically analyzing the outcomes of these experiments, marketers can gather valuable insights that inform data-driven decisions, leading to optimized marketing strategies that drive better results. Essentially, marketing experimentation embodies the principle of learning through trial and error and is fundamental for brands aiming to stay ahead in the rapidly evolving digital landscape.

Experimentation is what leads to breakthroughs—allowing your company to stand out while your competitors blend in.

Yet marketers often make the mistake of assuming that experimenting is for things like email subject lines, button colors and display copy. And sure, those experiments are all fine and good, but if that’s all you can envision, you’re limiting yourself.

In this article, I’ve got a list of impactful marketing experiments you may have never even considered. Use these to ramp up your own experimentation and improve your campaigns.

But before you do, you may want to check out this video:

Title Formats For Your Content

The titles of your landing pages, blog posts, resources, and other assets play a massive role in search engine optimization, from the position of your page in the SERP to the likelihood that searchers will click.

After Googling the phrase “marketing advice,” here are the titles you may see:

If you already have a page ranking for this keyword and want to move up the ladder, experiment with your title. Try adding a date. Try adding the phrase “UPDATED”. Try adding “NEW”. Try using phrases that make the content seem more recent.

You may find that one of these variations increases your click-through rate and propels your page higher up the search results—maybe even to the #1 spot.

Emojis In Your Email Subject Lines

Rocket ships. Money bags. Smiley faces.

I’ve seen them all in my inbox.

As you evaluate your email marketing efforts, consider emojis. Yes, emojis.

Marketers debate whether brands should use emojis, but I’m a firm supporter—why not go for it if emojis increase your open rate or response rate?

Even if you’re skeptical, at least experiment with emojis and see how your audience responds. Experian examined email subject lines with emoji and found that including an emoji in the subject line led to a 56% higher open rate compared to text-based subject lines.

Here’s a list of emojis and the top emojis by subject lines:

Related reading: Should B2B Brands Use Emojis In Their Content Marketing?

People In Your Live Chat

Studies have shown that people respond differently to women and men in live chats. Women are often times met with crude comments and more trolling in comparison to men. When Julia Enthoven the Co-founder of Kapwing experimented with different photos on their chat bot, the results were unfortunate. She tried 4 different images and tracked the trolling.

Here’s how it turned out:

Their informal study showed bias against women when it came to their name and photo. This is a sad reality…but it’s still a reality, and we have to recognize it as one of the many biases that influence our customers’ actions as horrible as it may be. In fact, Julia wrapped up the blog post with this exact takeaway:

“ If you’re a female with a chat box, you might want a pseudonym : Hiding behind Kapwing’s logo and company name on our messenger makes customer support easier and less discouraging. With the “Team Kapwing” name, people tend to assume that the person responding to them is a dude.”

It’s a shame. I 100% agree…

But Sarah Betts of Olark found a very similar situation when she ran an experiment using a male name (Samuel) for online support rather than her own. She found that her advice was taken more seriously by customers as Samuel, and her suggestions for fixing complicated technical issues were more likely to be accepted.

That said, when the data was complete and she ran the numbers it turned out that Sarah was rated higher by visitors. On the flipside, Sarah frequently had to escalate cases to engineers while Samuel had almost no escalated chat cases. This is why it’s not a bad idea to experiment with your own live chat window. Can you test a different name? Can you use photos of people with different genders? Can you analyze the data surrounding your customer support team based on gender? Can you try giving everyone the logo design as their profile picture?  All of these different things are worth testing!

You might end up disappointed with society.

But you might also unlock a powerful insight that improves your live chat experience.

Onboarding Emails

A quality on-boarding experience can be the difference between a customer who sticks around for years and a customer who churns in a month.

But the only way to really understand how your messaging affects new customers is through experimentation.

As new users sign up for your product, send half of them your existing onboarding email and the other half a completely different email. This means that you will have to craft two types of onboarding emails, which you can paraphrase online if you need inspiration. Run the test until you have enough data to determine which onboarding email worked best. If you noticed that one segment is using more key features or sticking around longer after a free trial – use this info to make adjustments.

Even if the difference is negligible, you will have learned that … So create another email and keep trying new onboarding content to see if things improve.

If you determine that your first email was and is the best welcome email you can send, congratulations—you’re a legend.

Sequences For Different Types Of Users

Not every prospect is the same. Not every customer is the same.

So why do we think it’s a good idea to interact the exact same way with every individual who signs up for our products or services?

Rather than trying to fit every customer into the same marketing sequence, develop an array of emails that target different customers (based on data) and create design experiences (landing pages or remarketing ads) that differ depending on their needs, preferred features, use cases, etc.

Experiment With New Content & Visuals

How many blog posts do you have on your site? Do you have some pages or articles that haven’t been doing so hot lately?

Give them a refresh.

Update these assets with new graphics, links, videos (try using text to video AI , as an example), and stats that make the content feel fresh and relevant.

You may even want to hire a designer—the sites below are great for finding designers (or templates, if a designer is not in the budget) that can give your tired assets a facelift:

…into something more engaging and on-brand:

All of this will make your content more appealing to the reader. The more appealing the content, the more time the reader will spend on your site.

The same rejuvenation efforts can be applied to the written content on your site. If it’s been a few years since you published that in-depth guide on your blog, rewriting the post for today’s audience could help the page rank higher and deliver more value to readers. You can even use a sentence rewriter tool to freshen up your content and make it more engaging.

Some simple things to update:

•       References to old content
•       Links going out to 404 pages
•       References to old dates
•       Outdated stats
•       Ineffective writing (e.g., bad ledes)

All of these things can have a negative impact on your content. But what if you have a lot of content on your site? Which pages should you update first? Look for…

•       Content that has tons of links but isn’t generating much traffic
•       Content that used to generate traffic but is quickly losing its SERP position
•       Content that has high engagement but has never ranked for keywords

Revisit your assets to look for these red flags. You can make changes in-house or hire a content writer to give it a tuneup and apply SEO best practices . Related reading: Safe Or Risky SEO: How Dangerous Is It REALLY To Change Your Article Dates?

Tip: A close companion to content is any tool that helps more people see it. For instance, if you run a small business, experiment with a local business schema to help push your business higher in search results when locals are looking.

Experiment With Paid Media Opportunities

Sponsored podcasts. eSports sponsorships. Online communities. Sponsored posts. Twitter media. Influencer marketing .

All of these opportunities over index for human usage & content consumption yet under index for paid media spend. This is where you might find a real opportunity that’s likely being overlooked by your competition.

Take podcasts, for example—over the last decade, the growth of podcasts in the US has been massive:

This growth is an opportunity for brands to experiment with a medium that has lots of engagement but is still relatively cheap because the market overlooks its potential.

It’s the same for influencer marketing, which is so widely viewed as a B2C opportunity that B2B brands are blind to the potential for their audience.

Here’s some insight on how B2B brands can leverage influencer marketing:

Funnels For Paid Media

Paid media efforts can burn through your budget very quickly and, if you’re not careful, result in very few leads and very little ROI.

Obviously that’s not good for business. ( Maybe not good for anybody )

One of the best ways to counter this common failure is to create experiential funnels that change for every ad set you create. Rather than launching one marketing funnel for a series of keywords, create funnels based on specific problems and let the users go through each until you start getting data for key metrics like time to close, conversion rate, LTV, CAC, etc. Send the right people to the right content.

Experiment with Local Search Optimization

In addition to paid media, local optimization also presents a fair opportunity for you to experiment and build your brand presence in the local market.

Since paid media can also burn through your marketing budget rapidly, local optimization helps get free engagement for your brand and increases revenue over time. It’s also another way to counter increased ad spending.

Experimenting with local search optimization includes creating business profiles on Google, Yelp, and other listing pages. Moreover, you can experiment with creating local content, including location-specific keywords, and building more authority through reviews.

Keeping local search ranking factors in mind at the time of optimization will help boost your brand’s presence and awareness in the local market and contribute to your marketing too For example, if you’re a self storage SEO agency or a local dentist, individuals seeking storage options or dental care in those areas will discover your website more easily if it’s optimized for local SEO.

Visuals On Your Homepage

In a recent study, we found that 56% of SaaS companies use photos of real people on their homepage and 70% of SaaS companies use custom illustrations . If you’re a SaaS company (heck, even if you’re not a SaaS company) experiment with these two options and see what your customers are more drawn to when it comes to visuals.

Value Propositions

If you’re early in the product development stage, you might have yet to hit on the most effective value proposition. At this stage, your SaaS product management  team is still in the process of getting feedback from beta users and working to tailor the product development roadmap accordingly.

Rather than playing a guessing game with your value proposition and crossing your fingers that it aligns with your customers’ needs and wants, experiment with the message and use user feedback to help define your message.

With Google Optimize, Unbounce or the landing page optimizer/creator of your choice, test a few different value propositions with site visitors and see how signups are influenced.

For example, HubSpot could experiment with two different value propositions on their homepage. One might say “There’s a better way to grow” kinda like this…

And another might say “Generate more leads and close more deals” Comparing the two would reveal what resonates with their customers and how HubSpot should tell their story moving forward.

While you’re at it, you could even test to see whether including keywords in this value proposition improves your search ranking position.

Salutations In Chat

Hey! Hi! Hello! What’s Up? Yo! Sup? Howdy! Bonjour!

You get the idea.

And sure, they all mean roughly the same thing. But these salutations may land differently depending on the industry. If you’re using a service like Drift or Intercom to acquire potential customers, you want to reduce friction as much as possible, so test out a range of chat greetings to find out which ones are most effective.

Sales Calls

One of my favorite technologies on the market today is the AI-driven phone service—you know, services like Chorus, Dialpad, ExecVision and Gong that analyze your voice calls using artificial intelligence and machine learning . Marketing and sales teams can use tools like these to track what their salespeople are saying to prospects and craft conversations that are more likely to convert.

I have a friend who manages a SaaS sales team. He listened to his team’s calls and found that when salespeople shared the story behind the company’s founding, prospects were 33% more likely to close. That’s huge! So he encouraged his team to inject the founding story into every pitch they could, and as a result the entire team saw a sustainable spike in their close rates.

Long-Form vs. Short-Form

There’s so much debate in the industry about whether you should use long-form or short-term content that I’ve decided the only answer is…

Experiment.

You knew I was going to say that, didn’t you?

The best way to truly understand whether your customers, prospects and leads would prefer a short content over a long content is to test ’em both.

One of the best blog posts I’ve read on the topic of long form vs. short form content is this piece from Joanna Wiebe about landing page length . She outlines how important it is for brands to think about the awareness levels of their audience at the time they’re reading their copy… And how their level of awareness can influence how much you need to explain.

Here’s a graphic she created to showcase the thinking behind it:

Brilliant right?

High awareness = Say less…

Low awareness = Say more…

No matter the content whether it’s a landing page, onboarding email, drip campaign, or cold email outreach experiment with message length across all aspects of your email marketing efforts until you have the data you need.

Pricing Page Layout & Design

One of the pages that matters most yet often gets very little love is your pricing page. This page is a key point in the buying cycle—the point where your customer decides whether your product or service is worth it.

Your pricing page is a gold mine for experimentation:

•       Should you highlight your most popular plan?
•       In what order should you list your product features?
•       Testimonials or no testimonials?
•       Should the default be annual or monthly pricing?
•       For enterprise solutions, is it better to include a price or just a “Contact Us” button?
•       Does the entire layout need a redesign?
•       Should you run remarketing ads on the page?
•      Do you need an  exit intent popup ?

All of these little experiments can add up to massive increases in revenue for your business. But don’t look at this list and assume you have to do ALL the things.

Identify which of these experiments make the most sense for your pricing page and your goals, like achieving a higher LTV or a lower CAC.

Social Media Display Copy

Every modern-day marketer has experimented with email subject lines . You can do something similar on social media by sharing two posts with the same link but different copy to drive folks to the content.

Let’s say you want to promote your blog post about wearable alerts and their safety aspects. Your first post could answer an FAQ while your second post could repurpose a customer review as a graphic, with both linking to your blog.

Whatever you originally titled your blog post doesn’t have to be what goes in your tweet. Try a new variation of that title, reshare it with your followers and see the reaction.

The same thing works on Facebook, Instagram and even Reddit .

Messaging Through Paid Media

Not 100% sure how best to broadcast your value proposition?

You’re not alone.

Tons of startups struggle with communicating clearly what they do and how they do it. Rather than leaving this to chance and a dream – invest in a paid media campaign on a channel where your audience spends time and showcase your message. Communicate a few variations of your value proposition on both the ad and the landing page. Does one value proposition resonate more? Is one value proposition generating more clicks and sign ups? If so, it’s likely that you can use this insight to guide your approach to communicating what it is you do.

New Channels For Distribution

A lot of brands stick to one marketing and distribution channel until the well runs dry. I get it—you need to fish where the fish are. But there are many other ponds and lakes out there you may be overlooking.

We often make the mistake of assuming that our audience is singular in their channel usage. We think the people who use Instagram only use Instagram, and the people who use LinkedIn only use LinkedIn. In reality, these people also frequent Quora, Medium, Reddit, Yelp, G2 Crowd, Capterra, Facebook, Twitter, YouTube, Vimeo…

So how do you experiment on these channels?

Start with a hypothesis. For example: If we create long form content on Medium and distribute it to the ABC publication we will be able to drive their audience to our site through a lead magnet.

Not sure where to start? Try using assets or topics that have worked well elsewhere. For example, if a blog post worked really well on your own site, it’ll likely do well on Medium.

The next step: Test it.

Create the content, publish it and gauge whether the effort was a success based on the response you get.

Blog Post Titles

When it comes to gaining initial traction for your blog posts , the title is often the most important element. Some people use their titles purely as clickbait, but the best marketers recognize that the title is where you should communicate the value of the blog post.

But figuring out how to communicate that can be tough. So… test it!

Experiment with multiple titles. One way to do this is by sharing the post on social media. Let’s say you’ve published this blog post:

Rather than assuming that this title is the one , share the article on Twitter with different headlines:

• Ending the Debate: What’s Better—B2B or B2C eCommerce?
• What’s the Difference Between B2B and B2C eCommerce in 2020? 
• B2B vs B2C: Let’s End the Debate Once and for All

See which title generates the most likes, retweets and clicks, and run with that title going forward.

Related reading: A Scientific Guide to Writing Great Headlines on Twitter, Facebook, and Your Blog

On-Site Personalization

Your jaw hits the floor—hard. Like fractured-in-half hard.

This is the power of personalization—if it’s done right. Testing personalization at scale has never been easier thanks to services like Clearbit, Optimizely and more. Cara Harshman talks about this approach in her post The Homepage is Dead: A Story of Website Personalization . A user’s IP address can tell you what company they work for or where they’re browsing from, allowing you to deliver a personalized online experience.

Native vs. Embedded Video On Social Media

Videos uploaded directly to Facebook perform better than videos uploaded to YouTube and then shared on Facebook. Videos uploaded to LinkedIn directly perform better than videos uploaded to YouTube and then shared on LinkedIn.

This is intentional. Both LinkedIn and Facebook recognize that if they send people away to YouTube, they’re losing eyeballs (and therefore advertising revenue) to Google.

Run a test where you share a video natively and via YouTube .

Meta Descriptions

When your landing pages, blog posts and other assets are ranking well, start testing ways to increase your click-through rate. One of the most underrated game-changers is the meta description. This is a meta description:

PS: Notice the cool “Volume” & “CPC” feature under the search bar? Check out this video on my favourite Chrome extensions to see how I did it. .. If Google is a highway for traffic, the title and meta description are the billboard that convinces your audience to take the next exit.

For example, if you run an ecommerce site, see how potential visitors respond when you put the price directly in the meta description. Don’t be afraid to try emojis, too.

Case Studies & Testimonials

The case studies and testimonials on your site have a real influence on potential buyers—demonstrating credibility and trust is crucial for conversion.

You can make assumptions about what you think will resonate with your audience … or you can run experiments.

Take a look at your testimonials—do you have quotes from clients in other roles or other industries? Switch things up! Can you swap out a case study for one about another company? Try it!

You’ll never know until you try.

One of the coolest case studies I’ve come across is this one from Slack:

There’s so much to unpack here. But if NASA trusts Slack, shouldn’t you?

That’s the message your own case studies and testimonials ought to send.

Traditional Advertising

today I downloaded an app after seeing a subway ad and I’m sending cosmic vibes to the marketer who will never be able to attribute it. — Kelly Eng (@boomereng) March 16, 2019

I’m a digital marketer by trade. I dream in pixels.

So do most marketers who are reading this post, I would imagine. In fact a lot of marketers have turned their heads when it comes to traditional marketing channels.

Radio ads. Television ads. Billboards. Magazine ads. Brochures.

Most of us consider these formats old-fashioned and ineffective. But how many of us have actually tried traditional channels lately?

You might discover that traditional media isn’t as lifeless as you thought. Or your campaign might fail—and that’s the point of experimentation. Either you’ll unlock a new growth channel or you’ll walk away with a lesson you can apply in the future.

Graphics For Social Sharing

People connect with people—we know that. But don’t assume you can slap any old image of a person on your content and call it a day.

In one of its many content experiments , Netflix compared how people responded to faces showing complex emotions vs. stoic or benign expressions. The results were clear: People connect with emotions.

Recapping the experiment, Netflix wrote:

Seeing a range of emotions actually compels people to watch a story more. This is likely due to the fact that complex emotions convey a wealth of information to members regarding the tone or feel of the content, but it is interesting to see how much members actually respond this way in testing…

Their testing also found that more visible, recognizable characters (especially polarizing ones) resulted in more engagement.

These results should be arresting for any marketer creating content.

We live in a world of social sharing. Millions of links are shared on Twitter, LinkedIn and Facebook every single day. When links are shared, they are typically accompanied by a graphic like this:

Experiment with the image. Use polarizing photos. Use people showing emotion. Try illustrations. Eventually you’ll figure out which graphics resonate with your audience and drive consistent engagement.

Outreach Times & Days

There’s a lot of debate online about the best time to send marketing emails .

Some people say the weekend. Some say the evening. And some say it really doesn’t matter—just share content when it makes sense for you.

What’s my suggestion?

There are literally tons of studies on this topic:

• MailChimp on send time optimization .
• Customer.io on best day to send emails .
• GetResponse on best day to send email .
• HubSpots best time to send a business email report .
• MailerMailer’s report on email marketing metrics .

And they all say different things. Some say send an email at 8AM on Tuesday. Some say send an email at 5PM on the weekend. The key is simple… Experiment. Test different time slots. Test different days. The only way to really get a clear idea of what works is to run some tests.

Wrapping This Up

The next time you think you’ve run all the marketing tests you can, think again.

There are so many opportunities for experimenting—that is, if you work in an environment where experimentation isn’t frowned upon. You need to surround yourself with people who have a growth mindset and believe that experimentation leads to breakthroughs.

But don’t just leave this post feeling inspired. Pick an experiment from the list to test out in the next few months.

Which experiment will you be running? Leave a comment or send me a tweet @TheCoolestCool —I’d love to hear from you.

PS: My latest experiment is using YouTube. Check out my channel and subscribe !

• Conversion Optimization
• Growth Marketing
• Digital Analytics
• Brand Marketing
• Digital Marketing
• Digital Psychology
• Ecommerce Marketing
• Product Marketing
• Technical Content Marketing
• Technical Marketing
• Google Analytics 4
• Browse all courses
• CXL Features
• Bottom-of-funnel SEO strategies in tough niches
• Growing AppSumo to 80m with performance marketing
• Account based marketing
• Building a growth process
• Building an innovative product
• Growth mindset: growth vs traditional marketing
• GrowthMaster Training Workshop
• Marketing strategy
• Optimizing Your Growth Process
• Partner Marketing
• Project Management for Marketers
• Retention: the most underrated growth channel
• User-centric marketing
• Data-driven influencer marketing
• Messaging strategy in public relations
• Sales Copywriting & Product Messaging
• Content marketing research
• Content recycling
• Email Marketing: Fundamentals
• Organic Social Media
• Product Marketing Content
• Scaling Content Marketing
• Content strategy and SEO for lead generation
• Growth Focused SEO testing
• On-Page, On-Site & Programmatic SEO
• SEO Link Building
• SEO-Driven Editorial Calendar
• Technical SEO
• Advanced Facebook Ads
• Advanced LinkedIn Ads
• Facebook Ads Creative
• Facebook Ads Experimentation
• Facebook Ads for Beginners
• Google Ads Experiments
• Google Ads for Beginners
• Linkedin Experimentation
• GA4 Intermediate
• Google Analytics 4 for beginners
• Preparing for Your GA4 Implementation
• Special Topics in GTM for GA4
• Attribution
• Data presentation and visualization
• Excel and Sheets for marketers
• Transactional data analysis
• Advanced Google Tag Manager
• Google Tag Manager for Beginners
• The Measurement Matrix
• Advanced Experimentation Masterclass
• CRO Agency masterclass
• Experimentation program management
• Intro to CRO and Experimentation
• Heuristic Evaluation
• Strategic Research for Experimentation
• User research
• Voice of Customer data
• A/B testing foundations
• A/B testing mastery
• CRO for Ecommerce Growth
• Good Practices
• Statistics for A/B testing
• Statistics fundamentals for testing
• Testing Strategies
• Digital psychology & behavioral design
• Intermediate statistics
• Landing Page Optimization
• People & Psychology
• Personalizing for conversion
• Brand strategy
• Positioning
• Radical differentiation
• Integrated Public Relations and SEO
• Storytelling
• Audience building
• Community building
• Community strategy
• Brand tracking 101
• Brand tracking with Momentive
• Customer storytelling and proof
• Segmentation and Persona Research
• Building a marketing agency
• Managing a remote marketing team
• Marketing Management
• Sales and customer success enablement
• Automation with Apps script
• Data collection on the web
• Data extraction
• Mobile Analytics
• Tag managers
• Python for marketers
• R for marketers
• SQL for marketers
• API Applications
• Cloud computing concepts
• Cloud services
• Machine learning applications
• Machine learning fundamentals
• Attention Basics
• Decision Making and Emotions
• Learning and Memory
• Building Habits and Loyalty
• Building Trust
• Cognitive Biases
• Nonconscious Motivation
• Principles of Persuasive Design
• Facebook Ads for ecommerce
• Google Ads for Ecommerce
• Google Shopping
• Selling on Amazon: Perfecting Traffic and Conversions
• Ecommerce Content Marketing
• Ecommerce SEO
• Email and SMS Marketing for Ecommerce
• Customer experience for ecommerce
• Customer journey for ecommerce
• Customer segmentation for ecommerce
• Retention and Customer Lifetime Value
• Ecommerce brand strategy
• Ecommerce merchandising
• Personalization for ecommerce
• Promotional events
• Selling on Marketplaces
• Ecommerce data and metrics
• Ecommerce forecasting
• Ecommerce tech stack
• Unit economics for ecommerce
• Competitive intel & market research
• Introduction to product marketing
• Positioning and company storytelling
• Pricing and packaging
• Product Analytics
• Analyst relations
• Product launches
• Hiring product marketers
• Working with the product team
• What is included in All-access
• First time here? See all resources
• Original research studies
• AB test calculator
• Conversion rate optimization guide
• Conversion optimization guide
• Ecommerce best practices
• Bounce rate guide: The foundations
• Clickthrough rate guide: The foundations
• Follow our B2B strategy podcast
• Sign up now

How to Run Marketing Experiments The Right Way

You need to run experiments. The one who runs the most experiments wins.

BUT – most marketing experiments are done wrong. What’s missing is hypothesis driven testing across all inter-business disciplines.

Nearly everything we use in our daily lives came about as a direct result of an experiment. The cars we drive, the computer or mobile device you are using to read this sentence, the Great Wall of China, Abercrombie and Fitch advertisements – all of them have gone through some form of the scientific method applied many times over a period of months, years, or even decades.

“You want experimentation. Every once in awhile, you stumble upon something that blows your mind.” – Jeremy Stoppelman, CEO of Yelp

Human beings use informal experiments all the time as a way to test-drive decisions that may have long reaching consequences. In order to decide whether we will take a certain action, we first construct a hypothesis, run an experiment, analyze the results, and then make a decision based on the available data. Most of the time, this process happens unconsciously and in only a few seconds.

Relatively recently, conversion optimization specialists have discovered the immense power of experimentation when applied to an online environment in the form of A/B Testing, Usability Testing, and more. In this course we will review the power of experimentation in the digital landscape while also examining wider reaching methods companies can use to harness it’s power.

By the end of this essay you will understand:

• How to create a structurally sound hypothesis
• Know the difference between leading and lagging metrics
• Have a better grasp on how to apply testing to web, email, and usability platforms
• Identify the key indicators of a well-designed offline experiment

Table of contents

Why do we test, what makes a good experiment, lagging vs. leading indicators, a/b testing, what are some other ways to use a/b testing, experimentation outside the digital framework, best practices of offline testing, testing non-inferior treatments, understanding the math, the basics of inference in experimentation, the case against focus groups, socializing experimentation, pre-registration.

Consider the case of Orville and Wilbur Wright, the famous American duo responsible for inventing the world’s first airplane. As you might be not surprised to learn, the Wright brothers didn’t launch a fully-built flying machine on their first attempt. They initially started with kites. The kite the brothers built in 1899 was only 5 feet wide. It was tested extensively over a period of months, constantly being refined, augmented, and tested again.

In 1901 the brothers created their own version of a wind tunnel where they experimented on over 200 different types of wings and gliders, the winning versions of which would eventually go on to form the basis for their historic powered flight in 1902.

So what’s the point? As brilliant as the Wright brothers were they understood the dangers of failure. They knew that if something went wrong during a manned flight one or both of them might not be around to make a second attempt.

Experimentation is essentially risk management. Would you rather move forward with your multi-million-dollar campaign strategy with the foundation for success resting on nothing but a hope and a prayer? Or would you feel far more confident in a massive business initiative if you’d already found enormous success within a test sample?

Experimentation as a business tool has taken root in many modern organizations as a necessity in a competition fueled workforce. Facebook has formed their own research unit (Facebook research) and Microsoft’s ExP Platform is dedicated to the continued launch, analysis, and theoretical exploration of online experiments.

The act of testing, refining, and applying is not just an essential part of business, but of life itself, rooted in the foundations of human beings as an unconscious necessity for survival.

Simply put, an experiment is a test of a hypothesis. A hypothesis (if you skipped remedial science) is a proposed explanation for a phenomenon based on limited data.

A good hypothesis must be both testable and falsifiable. For example, if I present the hypothesis that I am an occult mystic because I can see ghosts with my third eye, there is no way for others to test or disprove my assertion (unless you happen to know Dr. Strange).

Better examples of a falsifiable hypothesis might be:

• I believe my shoe business is losing money because our sneakers are priced too high and by reducing the cost we will increase sales by at least 50%
• I believe that Bruce Willis has starred in higher rated movies than Nicholas Cage because he has more experience acting in high budget films
• I believe that by changing the banner on our website to something more representative of our core customer demographic, visitors will convert more frequently

All hypotheses start with some form of “I believe.” We don’t know for a fact if our hypothesis is correct, otherwise why would we be testing on the first place?

Next comes an explanation of the change we will make to the control set (or our default data) and a statement of what we think will happen if our test goes the way we predict.

Despite what it may seem, we can never prove a hypothesis to be “true.” We must always leave some room in our mind for doubt and suspicion. By stating that the change we observe from a test is 100% true, we are in danger of falling for the problem of induction, or assuming that an empirical observation is a fact.

For example, if my hypothesis is “All swans are white,” it would nearly impossible to prove this hypothesis correct (I would need to find every single swan in the world to make that claim) but all it would take is a single black swan to completely disprove my idea.

For that reason, all hypotheses have an opposite called the null hypothesis. If our hypothesis states that we believe there is some difference between our control and treatment, the null hypothesis states its opposite: There is NO difference.

Hypothesis testing is a bit like a trial. Instead of “innocent until proven guilty,” we assume “Null until proven otherwise.” We must demonstrate that the evidence against the null result is so huge we’d be very surprised if it turned out there really was no difference at all.

Check out some examples of null hypotheses below.

• By reducing the cost of our sneakers, sales will not increase by at least 50%
• Bruce Willis has not starred in higher rated movies than Nicholas Cage
• If we change our website banner, visitors will not convert more frequently

Once we have a hypothesis (Represented as HA) and a null hypothesis (Represented as H0) we can begin considering what type of test will run, for how long, and with how many samples.

Take some time to think about interesting test ideas you might have and how you would structure those tests in the form of a hypothesis. For every hypothesis you create, write down what the corresponding null hypothesis would be.

One of the primary goals of experimentation is to understand the causal relationships between leading and lagging indicators.

A lagging indicator is an outcome. The data point. The numbers you typically show to the big boss. Revenue, downloads, and sign-ups are all examples of lagging indicators. Lagging indicators are often very difficult to change directly, especially if you have an established product or service. While it’s always necessary to keep track of lagging indicators to evaluate ongoing performance, as marketers we want to understand what causes those metrics to move either up or down. That’s where leading indicators come in.

Leading indicators are the “influencers” of lagging indicators. They predict how lagging indicators might be affected over time. The trouble is they are oftentimes very hard to pinpoint and equally hard to measure.

For example, a leading indicator of customer lifetime value might be user satisfaction. How is that measured? Is it something that can be calculated accurately? Are satisfaction scales different from one visitor to another?

Although it is difficult to measure customer satisfaction it is also much easier to influence. Think back to the last time you had a very bad experience in a store or online. How quickly did your level of satisfaction change? Oftentimes a single delightful or terrible experience is enough to alter a visitor’s perception of your brand forever.

Take a look below at some the leading indicators of major brands.

As you can see, leading metrics differ wildly depending on the brand and industry. Companies with large amounts of data (and even those without) regularly engage in statistical exercises like regression analysis in order to discover potential correlations between a leading and lagging indicator.

Do you know what lagging indicators represent your business KPI’s? What leading indicators are the clearest predictors of these success metrics?

Understanding what type of metrics are most important at driving growth helps us plan better more efficient experiments.

Before moving on to the next section, list down a few lagging indicators for your business. Common examples are Revenue and Conversions. For each lagging indicator, jot down several leading indicators you believe are strongly correlated to those metrics.

Which do you feel would be easier to test and why?

A/B Testing is the most well-known type of hypothesis testing in the mainstream marketing world. Popularized in the mid-late 2000’s by the rise of DIY website testing tools like Optimizely, Monetate, and Adobe Target, A/B Testing turned a previously painful and statistically rigorous mathematical process into a comparatively easy method of improving conversion rates.

The modus operandi of A/B Testing is straightforward: Randomize all web visitors into two or more groups, then show each group a different set of content. These content changes can be as small as a copy alteration or changing the entire website itself. When the test has enough visitors in each variation, analyze which version had the better performance by measuring key metrics during the test over time. Apply the winning version and badabing-badaboom, you’ve just made money! (Or at least, that’s the idea).

We don’t have enough space or time to do a full review of A/B Testing here but CXL has mountains of amazing resources on A/B Testing that should satiate even the most data minded marketer out there.

The important thing to remember about A/B Testing tools like Optimizely, VWO, or Google Experiments is that they are only automating a standardized testing process, the same standardized process that’s been used for nearly 100 years by scientists and researchers all over the world. Before moving on, let’s examine the steps of that methodology and how it takes shape.

• Develop Test Design: A/B Testing tools allow the test builder to create one or more digital treatments, served to unique visitors via a browser cookie and Javascript calls. This means the same user who visits a website and sees one version of a test will theoretically continue to see that same version as long as a cookie persists on their browser. This ensures samples are independent; a critical requirement split testing.
• Randomization: Randomization is also very important. By randomizing the visitors who are initiated into each test cell, A/B Testing normalizes for outlier effects or other unusual events of chance. Without randomization you could open yourself up to potentially test-ruining sampling errors.
• S tatistical Analysis: No hypothesis test could be complete without some form of statistical analysis. This analysis tells you how surprising the difference between the means of each variant might be and informs our decision to accept or reject the null hypothesis (Remember those terms from section 1?).

The beauty of experimentation is that the vast majority other types of tests follow the exact same rules. Therefore it’s important to understand not just WHAT your testing program is doing, but WHY it’s doing it. Once that’s clear, it’s easy to apply those key principles to many other areas of the business.

A/B Testing isn’t limited to standard web changes. Many tools are now beginning to offer such a solution on their platform. Let’s review a few of the more popular uses cases of A/B Testing and how you can leverage them.

Email Testing

A great use of A/B Testing is subject line and email content testing. Email tests can be deployed very quickly to a wide audience. Oftentimes these tests can return powerful results in short amounts of time. Many email delivery solutions like MailChimp, Bronto, and Campaign Monitor all have A/B functionality built into their programs.

Image source

Warning: You must be very careful before using an email delivery service to launch and evaluate A/B Tests. These tests often do not use any form of statistical analysis, meaning that declaring a winner is merely the result of pure mathematical observation! This is obviously unscientific and might be setting the foundation for a test program built on statistical phantoms.

Usability Testing

If you’re not already doing usability testing, you should. Usability testing is the process of measuring how real users interact with the interfaces and designs of your online or offline properties. While your website or mobile app may look very nice from the outside, if a visitor can’t accomplish key tasks or runs into infuriating errors they will inevitably have a poor user experience. Do not underestimate the wrath of a really pissed off user!

There are many different methods of testing usability. We can run tests on a solitary web property or prototype in order to discover baseline metrics to compare against for future testing (this process is called benchmarking). We could also run comparative tests of one design vs. a modified design, or even against a competitor to understand how intuitive our program is when compared against other industry leaders.

Pretty powerful stuff right? The downside of usability testing is that we inevitably have much smaller sample sizes than what we typically see in standard web-based A/B Testing. But don’t fret, there are plenty of statistical methods we can do to account for tests with low sample sizes.

Pro-tip: If you’re making any type of comparison that implies inference (aka: “I believe our users will continue to behave in the manner we’ve observed”) then you must engage in some type of statistical analysis.

When you conduct a survey, what is it that you’re really doing? Unless you plan to receive a response from every single person in your audience population (which is impossible if your audience is still growing) you are making a statistical inference based on a sample. That means you should be running tests on survey data too.

Want a memorable rule of thumb? “If you want to compare, without statistics beware.” This advice doesn’t just apply to the digital world. Anything that examines the relationships between two or more sets of independent data should optimally be run through a statistical test in order to determine if the observed results are due to noise or a meaningful difference.

Some things to consider before moving on to the next section: How does your organization currently handle comparative data? What does your company test and what do they not? Can you see any opportunities for testing that may have gone unexplored until now?

While the majority of optimization specialists focus on digital testing, the principles of experimentation we have discussed here can be applied to any part of the business, either internal or external. For example:

• Testing whether opening customer service lines one hour later negatively impacts customer perceptions or complaints
• Comparing checkout times in stores with automated vs. non-automated checkout
• Measuring whether using a dimmer, but more cost effective light-bulb has an impact on in-store foot traffic
• Testing whether major brand alterations (such as a name change) impact the number of localized website visitors

Some business models use experimentation to great effect already. QSR’s (Quick Service restaurants) such as Subway are prolific testers. With huge amounts of test subjects (franchises) an innovative R&D department, and a quick activation period, Subway has shown it has all the necessary ingredients to build a fantastic and effective testing program.

Other large chains like supermarkets, clothing outlets, and home décor companies have the ability to test extensively on product lines, pricing, visitor flow, and many other creative and wider reaching solutions.

Many tests can be done with the data that already exists in the business today. For example, in 1996 a relatively large supermarket chain “Dominick’s Finer Foods” discovered through experimentation that framing discounts in a bundle (“2 for $1”) instead of as single unit discounts (“1 for 50 cents”) significantly increased the number of units purchased. The same chain also discovered that imposing a coupon limit of 12 dramatically increased total unit volume over the standard coupon limit of 4.

In another example, a retailer based in the Midwest ran an experiment on musical “zoning,” or playing certain songs in different departments that most matched typical consumer demographic preferences. They found that altering the music by department had a significant effect on the amount and value of a customer’s purchase. Both of the above examples above spent minimal additional resources, yet the effects of these tests had a profound a profound impact on the bottom line.

While choosing ideas for an A/B Test is a relatively risk free exercise when done correctly (online tests can be stopped immediately, QA’d extensively, and so on) offline experimentation is not quite as simple. Each offline test requires some level of buy-in and in some cases that investment is a sizable financial figure.

Choosing what to test is not a simple process and must be undertaken after careful deliberation. There are three general guidelines to follow before proposing an offline experiment to leadership.

• Is it wanted?
• Is it feasible?
• Would it be profitable?

Is it wanted?: The first thing we should always ask before launching a new experimental business initiative is: Does the customer want this?

If the customer isn’t interested in what you’re offering then it doesn’t matter whether or not your testing program has the budget to roll out a test to 500 stores nationwide, it’s going to be a waste of money.

There are many methods that help in understanding whether a product or service is wanted or not, but for now we will just focus on two. The first is easy- Talk to your customers. Ask them what changes they would like to see or whether an additional feature would help their buying experience.

When speaking directly to customers about their preferences, it’s critical to be cautious of leading questions. In 2009 Walmart ran a survey asking if people preferred cleaner aisles. When the vast majority of survey participants said yes, they thought they had a winner.

Several months and millions of dollars invested in trimming back product lines later, Walmart discovered in-store sales had plummeted. It turns out one of the defining aspects Walmart was that store-goers could find almost anything on the shelves. Losing the “clutter” actually damaged their inherent value proposition. It was a 1.85 billion dollar mistake.

In another example, Southwest Airlines was known for refusing to add inter-line baggage, reserved seating, and food service (though they’ve since caved on some of these). The reason wasn’t because the airline was inattentive or didn’t care about its customers, but because the airline’s differentiating factor was its low price-point and on-time service. Although frequent fliers may have wanted a slew of additional features, they did not want these to come at the cost of their existing perks.

The second method of uncovering customer feedback is through pilot studies. Prior to rolling out an experiment to 10, 50, or even 100 stores, measure the responses and feedback from one or two stores. Will these results be statistically significant? Well…possibly. Will they be representative of your customer population? Probably not.

However, measuring the customer experience in the wild is far better than solely focusing on what marketers believe people want. Conducting qualitative research is a great way to take the pulse on your visitors and hear from their own mouths whether they think your idea is good or bad.

Is it feasible? It’s true that experimentation should not be confined to rigid business goals. However, it’s important to consider the budget, resources, and potential metrics that might be affected negatively by a failed test in advance.

The first consideration in feasibility is practicality. How will you accomplish this test? What sort of resources and manpower would you need to execute it on the ground?

If you are testing different opening hours for a physical location, for example, you’d need to think of how employees will be compensated for the lost time. Will it be a required change or opt-in only? Will managers be paid a stipend for participating in a test that might damage their bottom line? Will local marketing campaigns be needed to make users aware of the change, and if so how, and how much will such an advertising effort cost?

Would it be profitable? Some might say it’s a jaded way of thinking, but the end goal of any experiment is to make the company money.

Think carefully about whether or not the test concept is likely to result in an impact on the business’ bottom line. While it’s not a strictly bad thing if a test produces no results, it’s also not optimal. Each test is an investment and your goal is to make sure those investments pay off as frequently as possible.

Imagine you are adjusting a product’s price point from $5.99 to $4.99. After conducting the test, the revenue gained from the additional foot traffic to the store was nullified by increasing margins. Even though the test was a failure you learned something important: people DO respond well to cheaper prices, but not nearly as well as you might have initially believed!

Before moving to the next section jot down a list of ideas you might be interested in running at the company level. For each concept list with it the accompanying challenges (Budget? Stakeholder approval? Execution?) and how you might overcome these challenges.

Generally speaking, most conversion optimization specialists focus the majority of their energy on what is called “superiority testing.” The goal of superiority testing is to observe whether one or more variants are “superior” to the control in regards to a specific metric. Any experiment where the desired result outcome is a “lift” is a superiority test.

However, oftentimes superiority tests are incredibly difficult to run and even more difficult to validate. The smaller the difference between a control and a treatment, the larger the sample size is needed to accurately measure whether this change is real or simply noise. In digital experimentation larger businesses can detect tiny lifts because they have large amounts of traffic. For everyone else, it’s often a waste of time to invest in superiority testing that is not designed for observing much larger lifts.

An experiment that tests whether or not a treatment is no worse than the control by a certain amount is actually much easier and requires far less sample size. We won’t get too deep into the math, but check out Georgi Georgiev’s explanation here .

Non-Inferiority testing is actually i ncredibly powerful and remains a staple in clinical trials.

In 2013 Kohl’s was looking for ways to significantly cut operating costs. They decided to experiment with closing store hours earlier in a test that spanned 100 branches. After the test completed they determined that were was no significant loss of sales by decreasing the amount of operating hours, and the resulting implementation of this test was massive winner in terms of cost-saved.

Consider all the experiments that could be run as an inferiority test: Removing a low-margin menu item, staffing a store with one less employee and measuring productivity and efficiency, posting on social media less frequently, and so on.

Before moving to the next section, think of a few cost-saving experiments you could run in the field today. What sort of outcomes would you expect from such an experiment? Is it feasible? Is it wanted?

In a post on the statistics behind A/B Testing , Conductrics founder Matt Gershoff wrote that it’s important to understand “how the milk is made.” That statement could not be truer when applying experimental design theory to areas of the business outside traditional digital marketing.

It is simply not enough to have a cursory understanding of statistics in these cases. There are far too many hidden variables, potential mathematical danger zones, and terrifying pitfalls that might have a lasting impact on both the customers and the business.

Testing on essential store elements or proposing new offline functionality that might require significant resources to develop or launch can present real dangers. Making a statistical mistake on whether or not Price Point A works best over Price Point B could lose a business tens to hundreds of millions of dollars in the blink of an eye. Therefore, having a grasp of basic mathematical models is 100% essential for a proper offline experimentation investment.

When conducting tests in any form we most frequently use inferential statistics, meaning our test outcomes are predictive measures of behavior. We are not simply observing or cataloguing what is currently happening under test criteria, but also making a statement as to how we expect our entire visitor population to behave over time.

Inferential statistics require a few things in order to work properly: 1st, the correct statistical test. There are many types of hypothesis testing procedures that work best for certain types of data at certain sample sizes. This Measuring U post does a great job explaining some of the many types of tests that can be used to measure or compare different variables.

When deciding on what statistical test to use, it’s important to consider:

• The nature of your data: Are you observing a binary metric or continuous data? Will the sample sizes be the same or different? What are the independent variables? The dependent variables?
• The limitations of your test: Time? Sample Size? Testing on potentially different populations? Small or large predicted effect sizes?
• The question you are looking to answer: Are you trying to figure out which variant is superior? If they are equivalent? Testing for a correlation?

The second thing inferential statistics requires is a proper sample size . In order to understand why sample size is important, imagine you want to find out if bald-headed people are more likely to be denied bank loans than long haired people. You could run a test on one bald person and one long haired person, but would just 2 people tell you much about what might happen if we tested 10 or 100,000 people? Not really right? Usually there’s much more variation between individual people than there is between groups.

In order to discover those changes at the aggregate level we need to have a sample size that is representative of our visitor population.

Finally, you must take into account two parallel statistical concepts called “significance” and “power ”. Statistical power, simply put, is the probability of finding an actual result in your test data if there is an effect to be found, while significance is the ratio of signal to noise expressed as a numerical value.

Power and Significance are both test inputs and outputs. In order to calculate sample size you must decide on the appropriate significance and power levels. The standard significance level (also called alpha) is 5% and the typical power level (Or 1 – beta) is 80%. These two values (plus the smallest effect you are interested in detecting) are used to determine how many test samples you would need to observe the smallest detectable effect at a 95% significance level 80% of the time.

Confused yet? Don’t worry, until you get the hang of it I recommend trying your hand on one or more of the fantastic and easy to use statistical calculators below.

• https://cxl.com/ab-test-calculator/ – (One test calculator to answer all your pre and post test analysis questions)
• https://www.analytics-toolkit.com/ab-test-roi-calculator/ (Great for all statistical calculations – the most robust tool on the net)
• http://www.evanmiller.org/ab-testing/sample-size.html  (Not ready for the big guns yet? Start here for quick and dirty sample size calculations)
• http://thumbtack.github.io/abba/demo/abba.html (A straightforward significance testing calculator- to be used once an experiment concludes)
• https://abtestguide.com/bayesian/ (Interested in Bayesian analysis? Look no further than this great calculator from Online Dialogue)

Before moving to the next section, try opening up Evan Miller’s sample size calculator and playing with the data. What sort of sample size would you need to detect a 5% lift over a 20% conversion rate, with a significance level of 5% and a power level of 80%? What are you noticing about the correlation between minimal detectable effects and the proposed sample size?

Many researchers love the idea of focus groups as a form of testing. Sit a combination of diverse people together in a room and guide discussion around relevant topics to inform business decisions. While it might sound like a great idea at first (qualitative data is certainly a great place to start for test ideation) focus group sessions often fall prey to many forms of confirmation bias.

• Social Acceptability Bias : Social acceptability is the equivalent of “telling the interviewer what they want to hear.” Being face-to-face with a customer oftentimes drastically impacts how they moderate their language and behavior
• Interviewer Bias : Interviewer bias is also very real in focus groups. If the interviewer appears to be friendly and welcoming, they often get different answers compared to interviewers who are cold and indifferent. Studies in the area have shown that male and female interviewers are treated differently, as are interviewers of different races.
• The Bandwagon effect : Related to group-think, the bandwagon effect often occurs when one individual expresses an opinion and the rest of the group agrees, even if they wouldn’t have had the same opinion on their own.

For the reasons above (and many more) it’s best to avoid focus groups whenever possible. Offline observational studies, one-to-one moderated studies or online unmoderated studies using tools like UserTesting.com provide much better qualitative feedback than focus groups as long as you take steps to account for sampling error and bias.

Before moving on to the next section, take a moment to consider how you would choose test samples for an experiment you think might be interesting. Think carefully about the following questions before deciding on your final test group:

• Could the way I selected users lead to a bias? (Sampling error)
• Could the setup of my test bias users? (Leading questions)
• Do I have enough users to be representative of my population?
• Do I have enough users to find a result if it exists? (Statistical power)

A major part of experimentation is spreading awareness, sharing results, evangelizing your program, and demonstrating reliability to stakeholders. Building a culture of experimentation, especially where long established practices have taken root can be a challenging exercise and requires tact.

To begin, try to understand what fears stakeholders might have about testing. Imagine a CMO is hired specifically for their experience building a powerful loyalty program at a different company. What would happen if the existing program performed significantly better than the new version in testing? Would that person still be employed?

It’s important to realize that testing can represent a direct threat to many people’s livelihoods, but it doesn’t have to be this way. It’s incredibly important to give presentations and demonstrations so you can review the power and benefits of testing as a subject matter expert instead of the grand inquisitor of layoffs. When deciding how to socialize results within your business, ask yourself the following:

• Who knows about testing?
• Who needs to know but doesn’t?
• What do different stakeholders need to be comfortable with testing?
• Who would benefit most directly from experimentation?
• Who is most at risk from not using experimentation?
• What are the biggest problems you can help stakeholders solve?
• How are you making everyone’s life easier?
• When would be the best time or place to raise these issues to the right people?

Once you have the answer to those questions you can set out on your quest for buy-in. (If a culture of optimization already exists, then lucky you! You’re one of the chosen few)

One of the most important aspects of experimentation is developing a transparent test methodology. In order to have a program that consistently generates results without the of fear statistical errors sneaking up on us, we must abide by the rules of the model we’ve set in place. Of course that doesn’t mean we should blindly take the data at face-value, in fact the opposite is true. However, there should always be a meaningful reason for contesting or disputing the validity of an experiment and a clear methodology will show everyone exactly why and how you performed the test the way you did.

A great way to communicate test structure is through an experimental process called pre-registration. Pre-registration is simple: By recording in advance what testing methodology you use, what you will be analyzing and why, it is far less likely that you fall prey to either type I errors (false positives) or type II errors (false negatives).

In order to pre-register your study, create a document that contains the list items below:

• Hypothesis (What you expect to happen and the change to be made)
• Dependent variables (What are the test outcomes?)
• Methodology (What type of statistical test will you use on the data and why?)
• Sample size (How will you calculate sample size, and what is the expected number?)
• Analysis Plan (Which segments of data will you be analyzing? Which metrics?)
• Test Execution Plan (How will this test be run? Where and when?)
• Dependencies (What sort of resources will be needed for this test? Budget? Staff?)

Not only does pre-registration give context to your experiments, it also prevents you from engaging in exploratory analysis,  which is looking for results in the data until you find something interesting (As the saying goes, “If you torture the data long enough, it will confess to anything!).

According to Mark Zuckerberg and Jeff Bezos, experimentation is one of the key drivers of innovation and success within both Facebook and Amazon. In order to spread a culture that embraces testing as business north star, you must take the time to explicitly share your results widely within the organization. It’s not only important to share wins but to demonstrate how experimentation can answer questions, evolve, and adapt to new business challenges over time.

Be a vocal representative of testing within all parts of the organization, not just whatever branch the testing or optimization team falls under. Marketing, Analytics, Sales, Customer Service, and even Legal could all benefit from a system of tests designed to improve efficiency, effectiveness, or both.

Remember that a solid testing program will not be an overnight success. That’s the point! Successful experiments are the result of failure, refinement, failure, and THEN success. In the same way it took Orville and Wilbur Wright many years to create their history altering invention, true positive change comes about through many repeated tests, a discerning mathematical eye, and the imagination to create something new.

Happy testing!

Related Posts

As a marketer and optimizer it's only natural to want to speed up your testing…

A/B testing is highly useful, no question here. But a lot of businesses should not be…

A/B testing is fun. With so many easy-to-use tools, anyone can—and should—do it. However, there's…

The correct answer is of course that you should start testing where you have the…

Chad Sanderson

Chad Sanderson is the current Head of Product for Data Platforms at Convoy, and has worked on Experimentation teams for Microsoft, Sephora, Subway, and Oracle.

He is a passionate believer in the power of the scientific process and the value of measurement. You can find him on LinkedIn or shoot him an email at csanderson.data [at] gmail.com.

Join the conversation Add your comment

I needed to experiment different marketing strategies on my blog audience. Thanks for sharing this. Will try some of them now.

Comments are closed.

Current article:

Search posts.

• Acquisition (182)
• Brand Building (22)
• Business Building (109)
• Copywriting (42)
• CRO & Testing (316)
• Customer Stories (7)
• Digital Analytics (89)
• Marketing Tactics (50)
• Original Research (15)
• Psychology (81)
• Social Media (20)
• User Experience & Persuasive Design (179)

Subscribe to our newsletter.

Join 140,000+ marketers and get a weekly expert-led newsletter focused on helping marketing teams overcome growth challenges, punch above their weight, and crush their competition.

• Your e-mail *
• I agree to receive updates from CXL.
• Phone This field is for validation purposes and should be left unchanged.

Design of Experiments for Project Managers

Experiments are not just for scientists; they are in fact a tool project managers and engineers have used for years to better understand and refine processes. In the context of project management, an experiment is not in a secret lab with bubbling liquid in beakers; instead, the testing is done in a controlled manufacturing setting. In project management, the quality planning tool of setting up tests for a process is known as “Design of Experiments.”

On this page:

Define design of experiments

What you need to know about design of experiments for the pmp®, when are design of experiments used, how and why is design of experiments used, design of experiments benefits.

The design of experiments (DOE) is a tool for simultaneously testing multiple factors in a process to observe the results. Credited to statistician Sir Ronald A. Fisher , DOE is often used in manufacturing settings in an attempt to zero in on a region of values where the process is close to optimization . At its core, Design of Experiments is a statistical model enabling simultaneous testing rather than iterative testing of single factors.

Project Managers use the Design of Experiment tool in the Quality Planning process to determine the factors of a process, the way to test those factors, and what impact each has on the overall deliverable. Project Managers and those preparing for the Project Management Professional (PMP®) certification need to know of the DOE regardless of the industry in which they work. PMP® exam questions about the design of experiments are testing your understanding of when the tool should be used, how to set up experiments, the types of questions the test will answer, and when to use it instead of other tools.

Project Managers considering using the Design of Experiments should know that success depends on a very rigorous and strict application of it.

The experiment is more than the process is it also about the people. Project Managers using Design of Experiments in their Quality planning must have excellent communication skills. Knowing what to ask about the process (or product) being studied, knowing how to extract information from subject matter experts, and knowing how to share the results of the experiments are critical; communication is an important part of the planning and running of your experiment.

When conducting an experiment that tests only one factor, the impact of factors upon each other is missed. Design of experiment enables the project manager to learn about what happens when factors interact, thus providing a more accurate evaluation of quality. Project managers can use the data from a well-designed experiment in their Quality planning.

Knowing when to use design of experiment is as important as knowing what it is. It is a powerful tool in the Quality planning process and can be used when seeking answers for questions such as:

• What are the factors in a process that are controlled?
• What are the factors in a process that cannot be controlled?
• What are the settings for each factor?
• How do the factors impact each other?
• At what settings does the factors impact each other?
• What are the types of interactions among factors?

In the planning stage, enough time needs to be included in the overall project timeline to plan, execute, evaluate, and document the Design of Experiments. The time put into the experiments can save time later in the project and better ensure the level of quality of the final outputs.

Studying for the PMP Exam?

Project Managers who excel in planning will be able to apply that skill to the running of a Design of Experiments for their project. Specific tasks must be conducted in a certain sequence to achieve statistically relevant results.  

Note that the tool is called Design of Experiments, plural; a single experiment, even with multiple factors, will not provide enough data. One experiment may provide results which indicate a different problem to solve thus requiring the design of additional experiment ation .

A common example of the DOE tool is the baking a cake. With the diagram, you see different factors such as the equipment (oven) and ingredients. And within those factors, there are variables. For example, with an oven, is it conventional or convection? Is it powered by electricity or gas? Is the cooking rack in the bottom, middle, or top? In your design, you must capture the factors specifically so that, just as in any experiment, you can replicate them. If your cake burns on the bottom, is it the heating process (conventional / convection), is it the powering source (electrical or gas), and/or is it the placement of the racks (bottom, middle, top) within the oven? Or do all three aspects of the oven factor impact the interactions among other factors and your final result? You must accurately document each factor to better know how changes may shape the outcome.

In a manufacturing setting, the design of experiment should reflect all factors that work together in the process under study. Consider the equipment, the raw materials, the people, and the environment; each plays a role in the process and changes in some can impact all. The Design of Experiment provides a line of sight into a process so that levels of factors can be manipulated in a controlled manner to better manage the overall quality.

At the most basic level, the design of experiments, if conducted correctly, can result in reduced costs, reduced production time, and more reliable results. Benefits can include:

All PMP ® credential holders need to know about design of experiments

Although the design of experiments is associated with manufacturing, all project managers should know what it is, what it is used for, and what the potential benefits can be. It takes knowledge of the tool to determine when it will be beneficial to the project and company. Not only as part of preparing for the Project Management Professional (PMP®) certification exam, but also as a tool that can be used in the right situation. It also takes knowledge of the design of experiments tool in order to correctly answer the hypothetical and situational questions in the exam.

Upcoming PMP Certification Training – Live & Online Classes

• Megan Bell #molongui-disabled-link Creating an Effective PMP Study Plan: A Complete Guide
• Megan Bell #molongui-disabled-link What is a Project Schedule Network Diagram?
• Megan Bell #molongui-disabled-link Scheduling Methodology: Build & Control Your Project Schedule
• Megan Bell #molongui-disabled-link Schedule Baseline: How to Create, Use, and Optimize

Popular Courses

PMP Exam Preparation

PMI-ACP Exam Preparation

Lean Six Sigma Green Belt Training

CBAP Exam Preparation

Corporate Training

Project Management Training

Agile Training

Read Our Blog

Press Release

Charitable Contributions

Connect With Us

PMI, PMBOK, PMP, CAPM, PMI-ACP, PMI-RMP, PMI-SP, PMI-PBA, The PMI TALENT TRIANGLE and the PMI Talent Triangle logo, and the PMI Authorized Training Partner logo are registered marks of the Project Management Institute, Inc. | PMI ATP Provider ID #3348 | ITIL ® is a registered trademark of AXELOS Limited. The Swirl logo™ is a trademark of AXELOS Limited | IIBA ® , BABOK ® Guide and Business Analysis Body of Knowledge ® are registered trademarks owned by International Institute of Business Analysis. CBAP ® , CCBA ® , IIBA ® -AAC, IIBA ® -CBDA, and ECBA™ are registered certification marks owned by International Institute of Business Analysis. | BRMP ® is a registered trademark of Business Relationship Management Institute.

Agile marketing: A step-by-step guide

An international bank recently decided it wanted to see how customers would respond to a new email offer. They pulled together a mailing list, cleaned it up, iterated on copy and design, and checked with legal several times to get the needed approvals. Eight weeks later, they were ready to go.

In a world where people decide whether to abandon a web page after three seconds and Quicken Loans gives an answer to online mortgage applicants in less than ten minutes, eight weeks for an email test pushes a company to the boundaries of irrelevance. For many large incumbents, however, such a glacial pace is the norm.

We’ve all heard how digital technology allows marketers to engage in innovative new ways to meet customers’ needs far more effectively. But taking advantage of the new possibilities enabled by digital requires incumbents’ marketing organizations to become much nimbler and have a bias for action. In other words, they have to become agile.

Agile, in the marketing context, means using data and analytics to continuously source promising opportunities or solutions to problems in real time, deploying tests quickly, evaluating the results, and rapidly iterating. At scale, a high-functioning agile marketing organization can run hundreds of campaigns simultaneously and multiple new ideas every week. (For more on what agile is, see also “ Want to become agile? Learn from your IT team .”)

The truth is, many marketing organizations think they’re working in an agile way because they’ve adopted some agility principles, such as test and learn or reliance on cross-functional teams. But when you look below the surface, you quickly find they’re only partly agile, and they therefore only reap partial benefits. For example, marketing often doesn’t have the support of the legal department, IT, or finance, so approvals, back-end dependencies, or spend allocations are slow. Or their agency and technology partners aren’t aligned on the need for speed and can’t move quickly enough. Simply put: if you’re not agile all the way, then you’re not agile.

Would you like to learn more about our Marketing & Sales Practice ?

For companies competing in this era of disruption, this is a problem. In many companies, revenues in the segment offerings and product lines that use agile techniques have grown by as much as a factor of four. And even the most digitally savvy marketing organizations, where one typically sees limited room for improvement, have experienced revenue uplift of 20 to 40 percent. Agile also increases speed: marketing organizations that formerly took multiple weeks or even months to get a good idea translated into an offer fielded to customers find that after they adopt agile marketing practices, they can do it in less than two weeks.

Making your marketing organization agile isn’t a simple matter, but we have found a practical and effective way to get there.

Putting the agile marketing team together

There are a number of prerequisites for agile marketing to work. A marketing organization must have a clear sense of what it wants to accomplish with its agile initiative (e.g., which customer segments it wants to acquire or which customer decision journeys it wants to improve) and have sufficient data, analytics, and the right kind of marketing-technology infrastructure in place. This technology component helps marketers capture, aggregate, and manage data from disparate systems; make decisions based on advanced propensity and next-best-action models; automate the delivery of campaigns and messages across channels; and feed customer tracking and message performance back into the system. (It should be noted that the tech tools don’t have to be perfect. In fact, it can be a trap to focus on them too much. Most companies actually have a surfeit of tools.)

Another crucial prerequisite is sponsorship and stewardship of the shift to agile by senior marketing leaders. They provide key resources and crucial support when the new ways of working encounter inevitable resistance.

While these elements are crucial for success, the most important item is the people—bringing together a small team of talented people who can work together at speed. They should possess skills across multiple functions (both internal and external), be released from their “BAU” (business as usual) day jobs to work together full time, and be colocated in a “war room” (exhibit). The mission of the war-room team, as these groups are sometimes called (though companies also refer to them by other names, such as “pod” or “tribe”) is to execute a series of quick-turnaround experiments designed to create real bottom-line impact.

The exact composition of the war-room team depends on what tasks it plans to undertake. Tests that involve a lot of complex personalization will need a team weighted more heavily toward analytics. By contrast, if the agile initiative expects to run large numbers of smaller conversion-rate optimization tests, it would make more sense to load up on user-experience designers and project-management talent.

Whatever the composition of the team, the war room needs to have clear lines of communication with other groups throughout the organization and speedy processes to access them. For example, buying marketing assets often requires procurement review and legal approval. So the war-room team must have access to key people in legal and procurement to negotiate any changes. At one bank trying to establish a war room, there was significant resistance to providing representatives from legal and the controller’s office because of competing priorities. But marketing leadership knew their agile approach wouldn’t work without them, so it pushed with all relevant leaders to make it happen. Those people need to be identified ahead of time, and “service-level agreements” put in place that outline how quickly they will respond. Similar models of interaction may be needed with IT, compliance/risk and finance groups.

The team itself needs to be small enough for everyone to remain clearly accountable to one another—8 to 12 is the maximum size. Jeff Bezos famously referred to “two-pizza teams,” i.e., teams no bigger than can be fed by two pizzas.

A “scrum master,” ideally with experience in agile and often working with an assistant, leads the team. The scrum master sets priorities, defines the hypotheses, manages the backlog, identifies necessary resources, and manages “sprints” (one-to-two-week cycles of work).

Building out an agile war room will require working in new ways with external agencies, adding depth in key resource areas such as media buying, creative, and UX design, or analytics as needed. Working at the pace of agile may challenge an agency’s established workflows, but we have found that once they get into the rhythm, the performance boost justifies the change in procedures.

The marketing organization’s senior leaders will understandably need to oversee the activities of the war-room team. But they ought to interact with the team in a lightweight manner—once every three or four weeks, for example. Automated dashboards with key metrics can help provide leadership with transparency.

Reading about what war-room teams do, one might think agile practices apply only to direct-response marketing activities. But agile methods can improve the performance of product development, marketing mix, and brand marketing as well, by providing more frequent feedback, allowing for testing and iterating of ideas and communications in market, and accelerating the process for delivering impact from brand efforts.

Step-by-step overview of what an agile marketing team does

Here is how an agile team works:

Aligns with leadership and sets team expectations

Once the war-room team is assembled, it works with the leaders of the marketing organization and other key stakeholders to align everyone on the initiative’s goals. After that, the war-room team has a kickoff meeting to establish clearly that former ground rules and norms no longer apply and to articulate the agile culture and expectations: deep and continuous collaboration; speed; avoidance of “business as usual”; embracing the unexpected; striving for simplicity; data-trumping opinions; accountability—and above all, putting the customer at the center of all decisions.

Analyzes the data to identify the opportunities

By its second day, the team ought to be up and running and doing real work. That begins with developing insights based on targeted analytics. The insights should aim to identify anomalies, pain points, issues, or opportunities in the decision journeys of key customer or prospect segments. Each morning there is a daily stand-up in which each team member gives a quick report on what they accomplished the day before and what they plan to do today. This is a powerful practice for imposing accountability, since everyone makes a daily promise to their peers and must report on it the very next day.

Designs and prioritizes tests

For each identified opportunity or issue, the team develops both ideas about how to improve the experience and ways to test those ideas. For each hypothesis, the team designs a testing method and defines key performance indicators (KPIs). Once a list of potential tests has been generated, it is prioritized based on two criteria: potential business impact and ease of implementation. Prioritized ideas are bumped to the top of the queue to be tested immediately.

The keys to organizational agility

The team runs tests in one- to two-week “sprints” to validate whether the proposed approaches work—for example, does changing a call to action or an offer for a particular segment result in more customers completing a bank’s online loan application process? The team needs to operate efficiently—few meetings, and those are short and to the point—to manage an effective level of throughput, with a streamlined production and approval process. One team at a European bank ran a series of systematic weekly media tests across all categories and reallocated spending based on the findings on an ongoing basis. This effort helped lead to more than a tenfold increase in conversion rates.

Iterates the idea based on results

The team must have effective and flawless tracking mechanisms in place to quickly report on the performance of each test. The scrum master leads review sessions to go over test findings and decide how to scale the tests that yield promising results, adapt to feedback, and kill off those that aren’t working—all within a compressed timeframe.

At the end of each sprint, the war-room team debriefs to incorporate lessons learned and communicate results to key stakeholders. The scrum master resets priorities based on the results from the tests in the prior sprint and continues to work down the backlog of opportunities for the next sprint.

Scale across the organization

Getting a single war-room team up and running is good, but the ultimate goal is to have the entire marketing organization operate in an agile way. Doing this requires a willingness to invest the time and resources to make agile stick.

The first step in scaling is building credibility. As the war-room team works its way through tests, the results of agile practices will begin to propagate across the marketing organization. For each test that generates promising results, for example, the team can forecast the impact at scale and provide a brief to the marketing organization, with guidelines for establishing a series of business rules to use for activities and initiatives based on operationalizing the finding more broadly. With credibility, it’s easier to add more agile teams; one global retail company we know has scaled up its operations to include thirteen war rooms operating in parallel.

As companies add new war rooms, it’s important that each one be tightly focused on a specific goal, product, or service, based on the business goals of the company. Some companies, for example, have one team focused on customer acquisition and another on cross-/upselling to existing customers. Others have teams dedicated to different products, customers segments, or junctures in the customer journey.

We recommend adding agile teams one at a time and not adding new ones until the latest is operating effectively. As the number of teams grows and their capabilities increase, they can begin to expand their focus to assume responsibility for establishing business rules and executing against them. That systematic approach not only gives each new team intensive support as it comes online; it also allows business leaders to develop the kind of metrics dashboard it can use to track and manage performance for each team. This “control tower” helps to align resources as well, share best practices, and help break through bureaucratic issues. By scaling up in this way, the control-tower team has the opportunity to bring along all the supporting capabilities for marketing, everything from customer management to analytics to procurement, so that they operate at higher speeds as well.

A North American retailer established an agile marketing control tower and several war rooms to scale personalization across all key categories. The control tower ensured that the hundreds of tests run each year did not conflict and that the right technology was in place to collect appropriate data from the addressable audiences and to deliver a personalized experience across categories and channels. The war rooms each focused on systematically testing different media attributes and optimizing conversion on the company website across categories. After eighteen months, the retailer’s marketing-campaign throughput had grown four-fold, its customer satisfaction had increased by 30 percent, and digital sales had doubled.

As promising test findings become business rules, and as the number of war rooms grows, insights generated by agile practices will shape an ever-larger percentage of the organization’s marketing activities.

Marketing executives contemplating change often speak of the challenge associated with overcoming business as usual. By aggressively adopting agile practices, marketers can transform their organizations into fast-moving teams that continually drive growth for the business.

David Edelman is a former partner at McKinsey and current CMO at Aetna Health Insurance; Jason Heller is global leader of McKinsey’s digital marketing operations group; and Steven Spittaels  leads McKinsey’s marketing service line in Europe, Africa, and the Middle East.

Explore a career with us

Related articles.

Why agility pays

New insights for new growth: What it takes to understand your customers today

5 Marketing Experiments We Tried: The Winners, The Losers, and The Useless

One big thing that startups do differently to big companies is experimentation .

In reaction to the old corporate methods, startups are less like finely tuned money machines, and more like laboratories. That’s partly because of the culture of innovation, and partly because startups have less to lose by running a wrong experiment, but everything to gain if it is a success.

At Process Street , we’ve had our fair share of surprisingly positive experiments, as well as ones that were totally useless.

The more data you evaluate from other company’s experiences, the better you’ll get at running your own tests. So, in the spirit of experimentation and innovation, I’ve decided to share some of our A/B tests with you.

In this post, I’ll write up some of our growth hack results and then explore in-depth how to track and implement your own experiments so you can start improving conversions.

Home page headline: +5% conversion rate

A general rule we have at Process Street is that the most time and energy should be put into optimizing material that the most people see. This doesn’t stop us running experiments on everything from subject lines to content upgrade pop-ups, but it does mean we focus most of our time on our main landing page .

Out of all your site’s elements, the headline on the landing page is probably the most high-impact test you could run.

We’re still testing ours — we’re running an experiment and waiting for the statistical significance to peak — but here’s a test from the past that improved engagement and signups.

The control

This headline had been our main one since the beginning of time.

Variant #1: -9% conversion rate

We found that a lot of our users switched over from Excel, and that they were frustrated with the limitations of spreadsheets.

Variant #2: +5% conversion rate

As it turns out, the focus on automation was the right choice. Since then, we’ve directed a lot of our marketing material to sell the benefits of automation, and even written a free ebook about it .

Email marketing subject line: +30% open rate, +33% click rate

When we first started optimizing our marketing emails, we got amazing results. This isn’t really because of any complex trickery or breakthroughs. It’s mostly because the email marketing we were doing pre-test was terrible. I rip it apart in this post . Here’s an extract:

Since then, we’ve started running tests in Intercom instead because it’s linked to everything — our SaaS metrics, our users, and our support conversations.

Below are the results of one of the tests since:

It’s straight to the point, but is it too boring?

Variant #1: +30% open rate, +33% clicks

Yes, the control was too boring. The promise of avoiding nuclear war upped the open and click rates by almost a third! Including Ben Mulholland’s trademark playfulness in the subject line paid off.

Exit pop-up for Google-related posts: +196% conversion rate

A big chunk of our ranked posts are for Google-related keywords. We’ve got Gmail tips , Google Drive tips , Gmail vs. Inbox , Gmail extensions — you name it. To make better use of that traffic, we decided to add an exit pop-up. We tested the color, which bumped conversions up quite significantly, but we also changed the offer completely.

Originally, we wanted to sell the reader on a content upgrade: a list of Google Drive tips. We displayed this on any Google-related post, by setting the show rules as paths that contain google .

Variant #1: +65% conversion rate

All we did here was change the color. As silly as it can seem at first, color has featured in a particularly famous A/B test in the past and makes a proven difference in conversions.

Variant #2: +178% conversion rate

We decided to up the ante and offer a free two year subscription to premium Google Drive. As you might imagine, it sent the FREE STUFF sensors off the charts.

Variant #3: +196% conversion rate

…But a terabyte of data is much more attractive than a two year subscription. This pop-up was the clear winner.

SEO title and description: + 212% organic traffic

We use YoRocket to run tests on our SEO titles and descriptions. YoRocket helps you write better headlines and descriptions by checking the copy against a list of factors proven to convert:

This plugin is awesome for optimizing your old content, and recently we ran a series of experiments to optimize old posts that sat somewhere between position 9 and position 4 on Google. The aim was to push them higher up by the virtue of better copy.

The most staggering result was on our timeline templates post. Here’s what happened:

SEO title: Every Todo List Template You’ll Ever Need

SEO description: Need a to do list template? Check this huge list for Excel to-do list templates and Word documents, too!

Variant #1: +212% organic traffic

SEO title: Every To Do List Template You Need (The 21 Best Templates)

SEO description: Need an awesome to do list template? Check this huge list for 21 Excel to-do list templates and Word documents, too!

The main thing we added was a mention of how many templates we included in the post. Numbers are a proven conversion factor because they set expectations for the reader. And, if the number of templates is higher in our post than others, users might be more likely to click it.

How to run your own marketing experiments

Growth hacking isn’t a ‘set it and forget it’ process. It’s rooted in scientific principles, and if you’re not tracking it properly you might as well not even do it at all.

You can read all the A/B test results you want, and try to copy the winning variation…

But guess what?

The only  way to find out what works for your business is to do tests yourself, track the results, and improve upon it.

The thing is, A/B tests can be anything from a quick split test on an email subject that a few hundred people will read, to a massive change to your homepage that gets hundreds of thousands of views per month.

In this section, I’m going to share with you the growth hack tracking process we’ve developed at Process Street so you can:

• Store results and future tests all in one place
• Prioritize tests better
• Run more tests
• Get your own data
• Make smarter growth hacking decisions

Use this quick structure to easily record your experiments

No matter what tool you use (we’ll get to that later), you need a structure to record experiments, whether that’s experiments you’re already running or ones in the backlog.

For this, we use Kurt Braget’s PILLARS system .

Let me explain. PILLARS means:

•   Place (platform, e.g. Twitter)
•  Idea (a rough summary of the idea)
•  Labor (the work to be done)
•  Link (where you’re directing visitors / the CTA)
•  Audience (who you’re aiming at)
• ✅ Results (what you hope to get)
•  Spend (how much money, time, or resources you’ll need)

Here’s an example of Kurt’s implementation of PILLARS in Microsoft Excel :

Once you’ve dumped your ideas into a spreadsheet or another tool using this system, it’ll make it much easier to give each test a priority. We use a scale of 1-5, with 5 being the highest priority, highest impact tests.

Priority will vary depending on your desired outcome — if you’re looking to start more campaigns on social media, you’d give higher priority to a test for getting followers, for example.

Fill in every detail of the highest priority test

Pick a test from your list that will have the highest impact on your current goals, and then start to flesh out the details.

• What is your hypothesis?
• Additional notes or details
• What constitutes a winning test
• How long will the test run for?
• What is the control variable?
• What are the test variables?

There’s no point in filling all of this information in before you’ve done the basic prioritization with PILLARS, so go ahead and do this now for the test you’re definitely going to run .

We built this form into a Process Street template and run it for every test:

Get the test underway to start boosting conversion rate

After all the details are noted down, there’s nothing left to do but start building the test.

Send a list of instructions along with the variables to the person building it (if a developer’s needed, for example), or just build it yourself.

For example, if you were split-testing homepage headlines in Optimizely , you’d either give step-by-step instructions to a member of your team, or go into the app and set it up yourself.

Mark it as ‘in progress’ wherever you’re tracking it, and let the test run.

Here’s an important part you can’t afford to forget:

You need to do a spot check one day after the test has been deployed.

In my case, I’d check whether the headlines are displaying properly on the homepage, and that Optimizely is logging the results. I’d also take screenshots as I went along in case I wanted to write a blog post about the test later, or report an error to support.

Here’s how you properly track the success of your growth hacks

Did the experiment have any effect? Which test won?

This is some of the most important data you have at your disposal in your business.

It gives you insights into your customers, your best platforms, the language that resonates, and the design that converts best. Armed with that knowledge, you can make sure every campaign you run is better than the last, and your failures are just lessons to learn from that didn’t have a devastating effect.

Since you already set out the goals of the test, analysis is  easy:

• Was the win condition met?
• What was the control variable?
• What was the winning test variable?
• Why did it win?

Here’s a quick example for a test we ran:

Now, let’s take a look at the different tools you can implement this system with…

How you can track your own marketing experiments

In this section, I’m going to demonstrate how each of the below tools can be used to effectively track growth hacks using the methods I outlined earlier:

• Process Street
• Google Sheets

Tracking marketing experiments with Trello

Trello was made to handle different tasks being pushed through the funnel as part of a larger project, which means it lends itself to tracking tests:

As you can see, each test is its own card. As they move through the funnel, more details are filled out. I’ll explain the list names:

• Brainstorm : for pasting in links for experiments you feel could be useful for the future. Anything goes.
• Backlog : once you decide to move forward with a test, you prioritize it (screenshot below)
• Designing : getting the material ready to launch
• Running : the test is in progress! It needs checking, and you need to be recording the data of experiments in this list
• Analyzed : after the run period is up, move the card here and analyze the results. What’s the outcome? Which variable won?

Prioritization and analysis looks like this inside the card:

Trello is an easy way to track experiments as they happen, and you also end up with a list of completed tests that you can write up longer analyses on if you need.

Tracking growth experiments with Process Street

Process Street is a checklist and workflow tool that can track pretty much anything you can imagine. We created an in-house A/B testing process to help us with our own optimization, and then realized the app had a whole new use case, and can do it as well as any tailored solution could.

Here’s a more in-depth look at the process:

The idea is that you fill in as much information as is needed. If the test isn’t a priority, you only fill in the most bare-bones information. If it moves forward, then you tick a box when it’s started, fill in your analysis, etc.

Here’s what it looks like in action:

Tracking growth experiments with Google Sheets

Spreadsheets are probably the most popular way to track A/B tests. You can sort by priority, by audience, by labor, and really start to filter down on the individual elements of each proposed test. It’s more mathematical and produces more structured data than Trello, but updating progress isn’t as easy as just dragging cards into lists.

Here’s an example of the same setup as Trello translated into Sheets. As you can see, it isn’t ideal:

Off the screen, there is the same funnel as Trello. with Designing, Running, Analyzed, and a box for a long-form analysis.

These are the methods we’ve tested at Process Street, but I’m sure there must be plenty more. What are your top A/B testing tips? How do you track your experiments? Let me know in the comments.

Get our posts & product updates earlier by simply subscribing

Benjamin Brandall

Benjamin Brandall is a content marketer at Process Street .

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Save my name, email, and website in this browser for the next time I comment.

Take control of your workflows today

Main Types of Business Careers

Quiz: what business career is right for me, how to land a career in business, what business career is right for me the bottom line, what business career is right for me quiz.

• Share on Twitter Share on Twitter
• Share on Facebook Share on Facebook
• Share on LinkedIn Share on LinkedIn

Forage puts students first. Our blog articles are written independently by our editorial team. They have not been paid for or sponsored by our partners. See our full  editorial guidelines .

Table of Contents

Business careers can involve everything from leading major organizations and their strategic visions to digging deep into a company’s financial data. But if you’ve never even stepped foot into an office, how can you answer, “What business career is right for me?”

We’ll break down the main types of business careers so you can get a better idea of what each career path focuses on and the types of skills, personality types, and work environments it may align with. Then, you can take a fun, free quiz to figure out which business career is right for you.

Before you figure out which business careers are right for you, what are your options? 

Business careers are roles that focus on the entire management, organization, and development of companies or organizations. There are opportunities to get into business no matter what your skills or interests are — for example, you might want to work on developing a business’ product, or you might want to be the one selling it. 

Business careers can be a good fit for various skill sets, personality types, interests, and preferred work environments. Here are five main types of these careers.

Marketing and Sales

Marketing and sales are both business careers focused on promoting a company’s product or service. Professionals in marketing drive awareness of the product through various channels and strategies, whether that’s events, social media, email, or content. Sales professionals build relationships with potential customers and sell the product or service to them. 

>>MORE: Sales and Marketing: What’s the Difference and Which One Is Right for Me?

Business careers in marketing and sales require excellent communication and persuasion skills to convince people to use or buy the company’s product. They’re also a good mix of creative and analytical career paths — while marketing and sales professionals develop creative outreach strategies, they also need to analyze their impact to see what’s working (and what’s not).

Learn how to create an effective selling pitch that will convince a store manager to set up your company's product in the store.

Avg. Time: 2-3 hours

Skills you’ll build: Gaining influence, building relationships, acting as an owner, managing team members

Finance careers in business focus on managing a company’s money, including budgeting, investments, and risk management. Professionals in finance careers may be accountants , preparing financial statements to understand the company’s financial health. They may be more future-thinking and be financial analysts or investment bankers , assessing whether a specific investment or strategic decision can help the company make more money. 

Finance careers demand a strong foundation in mathematics and analytics. Professionals in this field must be adept at interpreting financial data, forecasting future trends, and constructing accurate financial models. Additionally, soft skills like problem-solving , attention to detail , and excellent organizational abilities are crucial for ensuring the reliability and accuracy of their analyses.

Investment Banking

Analyze a company's historical financial performance, then make a recommendation for a potential M&A target.

Avg. Time: 4-5 hours

Skills you’ll build: Financial analysis, SWOT Analysis, M&A screening, valuation, communication, presentation

Human Resources 

Human resources careers focus on ensuring employee success from recruitment to their exit interview. These professionals may work on hiring the right candidates for open roles, managing benefits and compensation structures and logistics, and helping employees grow and learn throughout their time at the company.

Human resources professionals need exceptional people skills — specifically conflict resolution , communication, and active listening — to resolve interpersonal problems in the workplace. They also need organizational and analytical skills to assess the effects and efficiency of the company’s current policies, benefits, recruitment practices, and more.

Human Resources

Gain practical human resources skills like giving feedback, creating a process map, and assessing compensation frameworks.

Avg. Time: 3-4 hours

Skills you’ll build: Feedback giving, communication skills, empowering with insights

Management and Leadership

Management and leadership careers in business focus on creating a strategic vision for a company and ensuring teams can execute it. These professionals monitor overall company performance and decide what strategies their team should use to meet goals. 

While some of these professionals are an organization’s top leaders — like a CEO, CFO, or COO — there are business careers in management and leadership down the ladder, too. For example, project managers lead project timelines, budgets, and execution, and consultants provide expert guidance to companies on strategy, operations, management practices, and more. 

Project Management

Conduct initial project planning for a global manufacturer's showroom development. 

Avg. Time: 1-2 hours

Skills you’ll build: Critical thinking, identification, risk analysis, project organization, time management, Excel

Management and leadership professionals need (not shockingly) leadership skills , including decision-making , communication, feedback, influence, and prioritization. 

Operations and Supply Chain

Operations and supply chain careers in business focus on ensuring that goods and services are produced and delivered efficiently. These behind-the-scenes professionals may manage a company’s product inventory levels, oversee production processes, coordinate transportation and logistics, and ensure that the product meets quality standards.

These professionals need excellent organization skills to track and manage the many moving parts of a business and its product, as well as communication skills to collaborate with the teams that work on and deliver the product. They also need strong analytical skills to assess the efficiency of their processes and identify where and how the company can improve.

Explore Supply Chain

Develop the technical skills needed to succeed in supply chain management, including how to use engineering data to specify applicability and capacity requirements and determine the next steps.

Skills you’ll build: Critical thinking, problem solving, interpreting engineering data, accountability

Ready to figure out what types of careers in marketing are right for you? Take the quiz! You’ll need to sign up for your results, but it’s 100% free.

You’ve figured out what business career is right for you — congratulations! So, how do you actually land a role in the field?

Become an Expert Problem-Solver

Every business’s goal is to solve some problem. Some businesses are mission-oriented, and their problems involve fighting for a social cause. Others may try to solve an everyday pain point in people’s lives. Some may be focused on helping other businesses solve their problems. 

“My top piece of advice is to start with a problem you’re passionate about solving and then build a solution around it,” says Chris Sorensen, CEO of PhoneBurner. “For students, this means getting hands-on experience in sales or a related field to truly understand the challenges. Whether it’s through internships , side projects, or even starting a small venture, immerse yourself in the industry you’re interested in. This practical experience, combined with a commitment to solving real-world problems, will not only make you stand out in job interviews but will also set the foundation for a successful career in business and technology.”

>>MORE: Get hands-on experience with Forage job simulations . In these self-paced, online programs, you’ll experience a day-in-the-life working for a top employer, building in-demand job skills along the way.

This doesn’t mean you need to solve world hunger. Instead, it’s a push to be curious about the world around you and how you might approach solving everyday problems. Companies hire people who can help them solve their problems, which means that this mindset is valuable and essential to landing any business role.

Be Ready for Anything

Business careers can be fast-paced and ever-changing. Even if you work for a more traditional company, you may have to deal with changes in leadership, goals, vision, or even your responsibilities. That’s why it’s crucial to build adaptability skills. 

“The business landscape is constantly changing, and being able to adapt to new challenges and trends is key to long-term success,” says Jonathan Goldberg, founder and CEO of Kimberfire. “Whether it’s adopting new technologies or shifting strategies, adaptability keeps your business competitive.”

Read industry news and follow relevant industry professionals on LinkedIn to stay current on what’s happening in the industries you’re interested in. 

>>MORE: Learn more about adaptability skills and how to add them to your resume.

Build Relationships

It’s cliché advice for a reason — networking works, especially in business careers.

That’s because so much of working in business involves collaborating with others, even if you’re in a role that requires some independent work. In an organization, it’s crucial to share what you do, what you’re working on, and how you’re doing it to help promote your growth and build relationships with others.

“Building a successful business requires strong relationships with clients, suppliers, and partners,” Goldberg says. “Networking opens doors to opportunities and collaborations that are essential for growth.”

Business careers cover so many different types of work, from financial analysis to helping recruit new employees. That means there’s a lot to choose from and a variety of opportunities depending on your unique skill set, work goals, personality type, and preferred work environment. 

No matter what type of business career is right for you, cultivating a problem-solving mindset and adaptability skills can help you stand out in job applications and succeed once you’re in the role. 

You can build these in-demand business skills and get real-world work experience with Forage job simulations — and more than triple your chances of landing a business career. 

Image credit: Canva

Related Posts

Computer science specializations quiz: which cs career is right for me, computer science vs. software engineering: what career path is right for me quiz, sales and marketing: what’s the difference and which one is right for me, upskill with forage.

Looking for more career quizzes?