experiment meaning business

Memberships

Business Experiments: Steps and Examples

Business Experiments: this article provides a practical explanation of various forms of Business Experiments . After reading, you’ll understand the basis of this supportive function for decision making, the various experiments, and the added value of experiments for both the organisation and the customer.

What are Business Experiments?

As a manager, but also as an entrepreneur, coming up with new ideas and executing them is a matter of course. Creating new, innovative ideas and ways to improve is essential for stimulating success within a company. Having or lacking success is not only beneficial or negative for the organisation itself, but could also have far-reaching positive or negative consequences for society.

For example, think of the development of the smartphone, cars, and aeroplanes. To some extent, these innovations were all the product of extensive testing and experimenting. It is therefore not enough to merely have ideas. The question is whether these ideas will yield results.

That’s why evidence is needed that these ideas and methods are actually effective. This evidence lies in the data obtained during Business Experiments. It should be clear that business experiments are necessary. First of all, to test and validate the ideas and methods, but also to determine the right method of implementation.

In fact, experiments are processes in which research methods and techniques are applied to assumptions or hypothesis . By testing these hypotheses and subjecting them to analysis, measurements, and validation, a conclusion can be formed based on facts. If the conclusion is positive, the decision is made to continue with the product or service.

Should the conclusion show that the results are still insufficient, the product or service is either discarded or improved and tested again.

Different Forms of Business Experiments

A business experiment, then, is a process in which different strategies, designs, and configurations are tested using a structured test-and-learn approach. Business experiments and testing revolutionary ideas are often linked to innovation. The exact goal of the experiment depends on the type of experiment.

In each type of experiment, however, data is obtained that is analysed and used in decision making. A distinction is made between, among others, the following types of experiments:

A/B testing is a testing method that is also based on a hypothesis. This experiment often makes use of two configurations of a certain set-up, such as a website. The old situation is compared to the situation after the experiment. The terms “control group” and “treatment group” are also used.

It’s a popular test method among website developers, who use it to test certain content and configurations. Using such an experiment, certain statistics such as conversion ratios and click-through rates can be optimised.

Prototype Tests

Another form of business experiment is developing a prototype. In the case of a prototype, a certain hypothesis about a product or service is tested by actually developing it. Based on the results after analysis of the prototype, it is then decided whether or not to continue developing the end product.

Two kinds of prototypes are usually distinguished. An evolutionary prototype is a prototype that is constantly adjusted and improved in order to realise a change to a product or a future product. Evolutionary prototypes are often expensive and are used in sectors that require products to be tested extremely thoroughly before they are introduced to the market. Examples include the automotive and aerospace industries. In addition, throwaway prototypes are also development and used. These are cheap and quickly designed to demonstrate a certain idea or feature. Throwaway prototypes are used in early phases of a product’s design.

Concept Testing

Concept testing is essentially a marketing experiment in which input from the potential customer is used in early phases of the design of a product or prototype. An important aspect in concept testing is gathering customer feedback through interviews, focus groups, or market research .

Mock-ups are used in production and design units within a company, and are often scale or full-size models of a design or device. The mock-up is used for promotion, design evaluation, knowledge transfer, and other ends. A mock-up can be seen as a prototype in which at least part of the original design is functional. Mock-ups are mainly used to gather feedback from users.

Thought Experiments

In a thought experiment, the consequences and effects of a certain hypothesis, procedure, or idea are evaluated. Thought experiments are developed in order to research ideas that don’t require physical experiments. Concrete thought experiments can be carried out using logic and arguments.

Abstract thought experiments often concern ideas about the future and go hand in hand with speculations. With thought experiments, various problems and hypothesis can easily be approached from various perspectives, especially when the experiment is carried out by multiple people.

A pilot, also known as feasibility study or experimental test, is a relatively small-scale experiment that tests whether a large project could also work well in practice. It provides organisations with a platform for testing certain projects, proving their worth, and uncovering shortcomings. It’s important that this be done in advance, before a considerable amount of money, energy, and time is spent on a large project.

Doing Business Experiments Yourself (Step-by-step Plan)

Doing Business Experiments Yourself - toolshero

Step 1: Formulating a Hypothesis

The process of an experiment or test always starts with formulating a verifiable, measurable hypothesis. Verifiable means it is possible to make the test or the experiment succeed or fail based on the objectives specified in the hypothesis. The details of the test are then designed, and it is determined which of the above methods, or an alternative, is suitable for the test or experiment.

Step 2: Designing the Experiment

If the hypothesis is verifiable and has been approved, the next step in the process is designing the experiment itself. This step concerns, among other things, identifying the objects, locations, or units to be tested; selecting control groups and treatment groups (the A/B tests discussed above); and defining testing and control mechanisms.

Determining which experiment is suitable for which hypothesis may be difficult. An example. The customer service department at a large multinational would like to improve their customer service.

To this end, they decide to conduct an experiment in which part of the customer service staff is taught a new approach (the treatment group), while another part (the control group) continues to do what they were doing before. By investigating the results of both groups over time, it can be determined whether the method used in the experiment is sufficiently effective or not.

Step 3: Data Analysis

After the experiment finishes, the data and results are analysed. In the experiment with the customer service, that means analysing the difference between the performances of the control group and the treatment group. Does the difference match our hypothesis? Has the quality of customer service actually increased? Have profit margins gone up? Is the outcome surprising? Are any additional experiments needed?

Step 4: Completion

After the test has been performed within the determined period at the determined location, the data obtained is analysed in order to determine the results. These results are ideally stored in a company library, and may lead to a broader roll-out of the experiment, or testing a revised hypothesis.

Step 5: Re-testing

A difficult aspect when setting up a testing method or experiment is deciding when to re-test. It’s difficult to determine when a test is outdated. This requires an experienced analyst to judge whether enough environmental factors have changed to make previous results obsolete.

For example, Netflix decided in 2006 that it was time to perform customer tests again. After five years, their user base had evolved from nothing but internet pioneers to members from all layers of society. Retailers and producers may also want to re-test pricing, for example because the prices of resources go up fast.

Business experiments: what is a Test & Learn Approach?

Test and Learn is a series of practices used by traders, banks, and other consumer-focused companies to test ideas and predict their impact. These business experiments are often performed at a small number of branches, or on a small number of customers. The process was designed to answer three questions about the testing program:

What is the impact of the pilot/test/experiment on important performance indicators?
Does the pilot/test/experiment have a greater impact on some customers or stores than on others?
Which components of the pilot/test/experiments work well? Which ones work less well?

Part of the Company Strategy

In its simplest form, the Test & Learn approach is a series of methods a company can use to experiment with new concepts and ideas. There are various reasons why Test & Learn should be added to any organisation’s business operations.

A great advantage of this method is that it allows for testing with real customers and real employees. They provide realistic data during the experiment, on the basis of which strategic decisions can eventually be made.

The Test & Learn method brings the initiative, the proposal, directly to the end user. This could be a new channel of communication, a reward structure, a new process, or something else. If it is received positively, the initiative will be taken on board for the future. If the initiative is not received positively, it may be further developed based on experience, or it can be decided to discard it.

Whatever the result, the decision is made on the basis of something that is proved by the data obtained, rather than presumptions or theories.

Practical Examples of Business Experiments

At eBay , applying changes, testing and experimenting is an important part of the company. Just like other online companies, eBay profits from the fact that it is relatively easy to carry out randomised tests of different website variations.

The online retailer’s administrators have carried out thousands of experiments. Because the site receives more than a billion visitors daily, they can carry out various experiments at the same time, while still having control groups and treatment groups in abundance. The simple A/B tests, in which two versions of a website can be compared, can be set up within a matter of days, and usually last at least a week.

Experimenting is so interesting to the company that it has built its own application called eBay Experimentation Platform. It is used by testers to keep track of what happens and which pages are tested during which times.

Besides the use of online experiments, the company also carries out offline tests, such as laboratory studies, home visits, focus groups, and website function analyses.

Business Experiments summary

Experiments are processes in which research methods are applied in order to come to a conclusion about a certain hypothesis or assumption. Business experiments come in several different forms. A well-known experiment, often used by website developers, is A/B Testing. This involves, for example, rendering a website in two configurations. Other methods are concept tests, prototypes, and pilots.

Before the experiment, a hypothesis is formulated. It’s important that the hypothesis be verifiable. Verifiable means it is possible to make the test or the experiment succeed or fail based on the objectives specified in the hypothesis.

The data gathered during the experiment is analysed afterwards. Based on this, a decision is made whether implementation should continue or the testers should go back to the drawing board.

In any case, the data obtained during the experiment is stored in the central company library. After some time, it may be necessary to test an experiment again because too many environmental factors have changed.

Now it’s your turn

What do you think? Do you recognise the explanation about Business Experiments? Are any experiments carried out in your working environment? Do you know any examples of large and successful experiments performed in the past at large companies? Do you have any tips or additional comments?

Share your experience and knowledge in the comments box below.

More information

Anderson, E. T., & Simester, D. (2011). A step-by-step guide to smart business experiments . Harvard Business Review, 89(3), 98-105.
Bocken, N. M. P., Bom, C. A., & Lemstra, H. J. (2017). Business experiments as an approach to drive sustainable consumption: The case of homie . Delft University of Technology, 8, 10.
Davenport, T. H. (2009). How to design smart business experiments . Strategic Direction .
Ganguly, A., & Euchner, J. (2018). Conducting Business Experiments: Validating New Business Models Well-designed business experiments can help validate assumptions and reduce risk associated with new business models . Research-Technology Management, 61(2), 27-36.
Website Vantazo . Retrieved 02/20/2024 from Vantazo.com

How to cite this article: Janse, B. (2020). Business Experiments . Retrieved [insert date] from Toolshero: https://www.toolshero.com/decision-making/business-experiments/

Original publication date: 24/02/2020 | Last update: 02/20/2024

Add a link to this page on your website: <a href=”https://www.toolshero.com/decision-making/business-experiments/”>Toolshero: Business Experiments</a>

Did you find this article interesting?

Your rating is more than welcome or share this article via Social media!

Average rating 5 / 5. Vote count: 14

No votes so far! Be the first to rate this post.

We are sorry that this post was not useful for you!

Let us improve this post!

Tell us how we can improve this post?

Ben Janse is a young professional working at ToolsHero as Content Manager. He is also an International Business student at Rotterdam Business School where he focusses on analyzing and developing management models. Thanks to his theoretical and practical knowledge, he knows how to distinguish main- and side issues and to make the essence of each article clearly visible.

Regression Analysis explained

B2B Marketing: the Basics, Types and Examples

Product Management explained

Buy Sell Hierarchy (Miller and Heiman)

B2C Marketing: the Definition and Strategies

STP Model in Marketing Explained: Steps & Repositioning Example

Also interesting.

Decision Tree Analysis: the Theory and an Example

What If Analysis: the Basics and an Example

Six Thinking Hats Technique explained

BOOST YOUR SKILLS

Toolshero supports people worldwide ( 10+ million visitors from 100+ countries ) to empower themselves through an easily accessible and high-quality learning platform for personal and professional development.

By making access to scientific knowledge simple and affordable, self-development becomes attainable for everyone, including you! Join our learning platform and boost your skills with Toolshero.

ABOUT TOOLSHERO

Free Toolshero e-book
Memberships & Pricing

What is Strategy?
Business Models
Developing a Strategy
Strategic Planning
Competitive Advantage
Growth Strategy
Market Strategy
Customer Strategy
Geographic Strategy
Product Strategy
Service Strategy
Pricing Strategy
Distribution Strategy
Sales Strategy
Marketing Strategy
Digital Marketing Strategy
Organizational Strategy
HR Strategy – Organizational Design
HR Strategy – Employee Journey & Culture
Process Strategy
Procurement Strategy
Cost and Capital Strategy
Business Value
Market Analysis
Problem Solving Skills
Strategic Options
Business Analytics
Strategic Decision Making
Process Improvement
Project Planning
Team Leadership
Personal Development
Leadership Maturity Model
Leadership Team Strategy
The Leadership Team
Leadership Mindset
Communication & Collaboration
Problem Solving
Decision Making
People Leadership
Strategic Execution
Executive Coaching
Strategy Coaching
Business Transformation
Strategy Workshops
Leadership Strategy Survey
Leadership Training
Who’s Joe?

EXPERIMENTATION

“There are three principal means of acquiring knowledge… observation of nature, reflection, and experimentation. Observation collects facts; reflection combines them; experimentation verifies the result of that combination.”

– Denis Diderot, 17th Century French Philosopher

Personally, dreaming up, executing, and seeing the results from experiments can be one of the most rewarding aspects of strategic leadership . One of my favorite experiments was a simple promotion to drive online reviews for a consumer goods product company. The company had a bunch of little portable speakers they were discontinuing that cost $7 but retailed for $40. The experiment was simple. For three days we would give anyone who wrote an online review of the company’s products a free portable speaker. I wanted to see how much a $40 speaker would drive people to spend 5 minutes writing a review. We posted the promotion on Facebook and emailed the customer list, and within three days the company doubled the number of reviews that took them three years to amass!

As a strategic leader, ideas are always popping up inside your head. Instead of either doing nothing or fully implementing the idea, you should often pick the middle road and run an experiment to see what happens when your idea comes to life. Experimentation is a low-cost and fast way to tweak your organization’s strategies continually.

What is experimentation?

The scientific method is alive and well in business. Experiments are structured tests to verify a hypothesis or idea and create insight into potential cause and effect. Experimentation is used a considerable amount in marketing , services, and retail to understand such things as:

• The impact of advertising on sales • How different messaging, promotions and creative in advertisements or emails creates different engagement and response rates • The interaction between pricing and demand • The customer response to different service models or call scripts • What effect edits to a website have to user behavior • How changes in a store impact consumer behavior • And about anything else you can imagine

Testing a Hypothesis with Experiment vs. Control Group

An experiment involves testing the impact of a hypothesis or idea on an experimental group. To objectively understand the cause and effect of the hypothesis, the results from the experimental group are compared to a control group, which is a group that is similar to the composition of the experimental group but didn’t have the stimulus of the experiment group.

The idea behind experimental versus control groups is the two groups had the same set of variables and conditions throughout the timeframe of the experiment, with only one difference, the variable(s) tied to the hypothesis. This way, once you understand the difference in the output between the experimental group and the control group, you can attribute the difference to the experimental hypothesis.

At a retailer I worked with, there was an entire team devoted to experimentation focused on dreaming up and testing improvements to the loyalty program, online advertising, email, promotions, ecommerce, pricing, service, store hours, store layout and remodels. One of my favorite experiments was a Google Adwords saturation test to understand online advertising’s effect on online and in-store sales. The digital marketing team picked six categories (e.g., Soccer, Baseball, Running Shoes) and eight markets and spent a considerable amount of money to place millions of Sports Authority ads every time someone searched a term within the six categories in the eight markets. After the test, we calculated online and in-store sales in those six categories in the eight markets and compared them to the online and in-store sales of the control markets. The results were fascinating. There was a significant lift in ROI on team-specific sports, but not on more generic categories such as running shoes. The interpretation of the results was customers responded better to advertising in categories where the retailer had a higher market share and not as much competition , such as team sports like soccer and baseball. On the other hand, while running shoes were a big business for the retailer, the actual market share was low, hence a low ROI on the advertising experiment.

Why is experimentation important?

It is a blend of art and science to develop an effective marketing strategy throughout the customer journey. Continuous experimentation with incremental and innovative ideas is the science behind optimizing the customer journey. Experimentation is a low-cost, low-risk, and empirical way for organizations to test new ideas . Strategic leaders that embrace the discipline of experimentation continually grow, evolve, and build upon the fundamental truths of how their customers respond to ideas and changes. Some companies built their empire off the back of experimentation. Capital One, a leader in credit cards, created a $40 billion business by executing hundreds of thousands of controlled experiments optimizing credit card designs, offers, and messaging.

How do you conduct experiments?

At a high level, the process of experimentation involves building a hypothesis, designing an experiment around the hypothesis, executing the experiment, and analyzing the results. In conducting experiments, here are some important elements to consider:

Building Hypotheses

What are some of your best ideas to improve your organization’s customer journey? If you are unsure of their potential effect on behavior and want to or need to understand their effect empirically, then you should turn your ideas into hypotheses , which are theories that should be proven or disproven through experimentation. Transform these hypotheses into one or two variables you can test through an experiment. In building out variables to test, make sure they are meaningful and different enough from the status quo to make you think there will be a difference in customer behavior. It happens a lot when someone might test two different email subject lines, but the difference is so minute, that the results are meaningless. As an illustrative example:

Poor hypothesis – Adding the word All to email subject lines will improve the email open rate

Control Ad: Save at Least 25% off Baseball Bats Test Ad: Save at Least 25% off All Baseball Bats

Good Hypothesis – Changing promotion messaging from at least 25% to 25-60% will improve the email open rate

Control Ad: Save at Least 25% off Baseball Bats Test Ad: Save 25-60% off Baseball Bats

Designing an Experiment

Most experiments change only one variable in a system to isolate the impact. You can test changing multiple variables, but the conditions and analysis get much more complex. In designing an experiment you need to understand what variable(s) you are going to change and the corresponding result you expect (e.g., increase sales, conversion rate). You also need to determine how long the test will last. Most experiments necessitate an experimental group and a control group to be able to compare the impact of changing one variable. In creating a control group, you should probably use 10% of your population to get a statistically significant read on the results of an experiment. The control group also needs to be random to ensure there isn’t bias or commonality in the control group that could affect the results. There is some excellent, although very expensive test & learn software from APT, that picks control groups, runs all of the statistics , and produces insights related to an experiment. Another option for designing experiments is a factorial design. Factorial design can test more than one variable at a time. Factorial designs are a much more sophisticated experimental designs typically necessitating the guidance of an expert.

Executing an Experiment

A critical aspect of executing an experiment is ensuring there isn’t much variance and movement in the variables not being tested. To have a valid experiment the conditions between the control group and the experimental group need to be pretty much equal, except for the change in the tested variable(s).

The communication of an experiment can also be a bit tricky. When communicating an experiment, you want to do it in a way that doesn’t change behavior between the control and experiment groups. It is also important to ensure the experiment time is long enough to be meaningful. If you are trying to measure the effect of advertising on sales, you have to take into account the typical length of a sales cycle of a customer. If you are trying to understand the immediate response, like a click-through response to an ad experiment, then you should be fine conducting shorter experiments.

Understanding Results

Once an experiment is complete, hopefully, you’re able to compare the results of the experimental group to the results of the control group. We won’t go too much into statistics, but the larger the difference in results the more you can be sure there the effect was driven because of the change in the test variable. If there is a small difference in the results, that could have been simply because of noise or a few people doing one thing versus another. You can base the accuracy and the validity of the results on a few drivers, including the magnitude of the difference in the results, the size of the control group as a percentage of the entire group, and the level of confidence you are looking for (e.g., +/- 5%). You don’t need a course in statistics to better understand how you interpret results. This sample size calculator can help determine sample sizes for the control group: http://www.surveysystem.com/sscalc.htm. Below is a straightforward analysis that plots out the conversion rates of the control group versus the experimental group during a website test

A/B Testing

A/B testing is one of the most used controlled experimentation methodologies, especially in marketing. A/B Testing is typically used to understand the incremental effects of an ad, webpage, email subject lines, direct mail, and other marketing vehicles. You take two versions that are identical except for one variable (e.g., a different message, offer, image) and compare their performance (e.g., click-through rate, conversion rate) against each other. A/B Testing is a strategy used to continuously improve websites, digital advertising, email campaigns, and direct response ads.

Pre/Post Experiments

Sometimes you don’t have the luxury to create control groups and conduct a controlled experiment. In these cases you can still conduct a simple pre/post-experiment, it just might not be as refined and accurate as a controlled experiment. A pre/post-experiment is simply making a change to a system and calculating the lift in performance before and after the experiment. Pre/post experiments introduce a lot of potential bias and noise to results, since there may be very different conditions before and during the experiment. In practice, most organizations rely on pre/post experiments for most of their experimentation.

NEXT SECTION: BENCHMARKING

DOWNLOAD STRATEGY PRESENTATION TEMPLATES

THE $150 VALUE PACK - 600 SLIDES 168-PAGE COMPENDIUM OF STRATEGY FRAMEWORKS & TEMPLATES 186-PAGE HR & ORG STRATEGY PRESENTATION 100-PAGE SALES PLAN PRESENTATION 121-PAGE STRATEGIC PLAN & COMPANY OVERVIEW PRESENTATION 114-PAGE MARKET & COMPETITIVE ANALYSIS PRESENTATION 18-PAGE BUSINESS MODEL TEMPLATE

JOE NEWSUM COACHING

EXECUTIVE COACHING STRATEGY COACHING ELEVATE360 BUSINESS TRANSFORMATION STRATEGY WORKSHOPS LEADERSHIP STRATEGY SURVEY & WORKSHOP STRATEGY & LEADERSHIP TRAINING

THE LEADERSHIP MATURITY MODEL

Explore other types of strategy.

BIG PICTURE WHAT IS STRATEGY? BUSINESS MODEL COMP. ADVANTAGE GROWTH

TARGETS MARKET CUSTOMER GEOGRAPHIC

VALUE PROPOSITION PRODUCT SERVICE PRICING

GO TO MARKET DISTRIBUTION SALES MARKETING

ORGANIZATIONAL ORG DESIGN HR & CULTURE PROCESS PARTNER

EXPLORE THE TOP 100 STRATEGIC LEADERSHIP COMPETENCIES

TYPES OF VALUE MARKET ANALYSIS PROBLEM SOLVING

OPTION CREATION ANALYTICS DECISION MAKING PROCESS TOOLS

PLANNING & PROJECTS PEOPLE LEADERSHIP PERSONAL DEVELOPMENT

Starting a Business
Growing a Business
Small Business Guide
Business News
Science & Technology
Money & Finance
For Subscribers
Write for Entrepreneur
Tips White Papers
Entrepreneur Store
United States
Asia Pacific
Middle East
United Kingdom
South Africa

The Basics of Experimentation and Why It's Key to Your Startup's Growth In this three-part series, we will discuss experimentation in detail. This article focuses on the "why" and "what" of experimentation.

By Piyanka Jain Edited by Chelsea Brown Jul 22, 2022

Opinions expressed by Entrepreneur contributors are their own.

Some people say give the customers what they want, but that's not my approach. Our job is to figure out what they're going to want before they do. - Steve Jobs

Understanding customer preferences is critical in driving business growth, but customers typically don't know what they want. As a business, it is beneficial to understand customers better and deduce what they want from you. This is where experiments help!

Startups with limited historical data typically get a better understanding of customer needs by running business experiments over a feature rather than from running analyses over historical data. You can make experimentation the key to growth for your startup if you can establish the right strategy, approach and methodologies. Let us first discuss what experimentation is.

Related: Experimentation and A/B Testing: A Must-Use E-commerce Growth Strategy

What is experimentation?

Experimentation, or what is sometimes loosely referred to as A/B testing , is a method where a business hypothesis is practically tested on consumers. Often an organization might not have historical data to analyze the business decisions. Similarly, they might be looking at some decisions where one cannot have historical data, like testing a new pricing strategy they have never tried. In these scenarios, we conduct an experiment.

We follow a methodology where we take a small sample from the entire population that will be affected by the decision and introduce them to a new feature. We compare the results observed from this test group to a control group that wasn't introduced to the feature and understand if that particular feature can be beneficial to the consumers and the business. Using this specific methodology, we evaluate which of our hypotheses will be more valuable and implement the same. That's experimentation in a nutshell for you.

This three-part series of articles will cover what experimentation is, why one should adopt it, elements of a successful experiment , the common reasons for experiment failure and some behavioral biases affecting experiment outcomes.

Related: Transform Your Business by Encouraging Experimentation and Change

Why experiment?

The two obvious reasons for conducting experiments are hypothesis testing and proving causality:

Hypothesis testing:

Typically, humans make decisions based on gut feelings and intuitions. Data analytics is an anti-thesis that supports data-based decision-making. But not all data are the same. You will find yourself in situations where you believe specific changes in the feature can increase your primary metric (such as growth or revenue). The hypothesis might sound reasonable to you and your colleagues, but it's not guaranteed success as you do not have any backing data. In such a situation, experimentation is that friend who can provide you with a data-backed answer that can validate (or nullify) your hypothesis.

To prove causality:

Correlation vs. causality is a living issue in data analysis. Two or more variables are considered related in a statistical context if the values of one variable increase or decrease as the value of one variable changes. This change can have two cases:

Correlation is a statistical measure (expressed as a value between -1 and 1) that describes the magnitude and direction of a relationship between two or more variables. However, a correlation between variables does not automatically mean that the change in one variable is the cause of the difference in the value of the other variable.

Causation indicates that the change in one variable results from the changes of another variable, i.e., a cause-and-effect relationship exists between the two variables.

Theoretically, the difference between correlation and causation is easy to identify. However, it does not remain easy in practice. Randomized experiments help differentiate between these two realities to find truly causal effects. Randomized experiments are the norm in the real world to understand if a specific change can create a difference in the outcome. For example, a randomized controlled clinical experiment establishing a pill's effectiveness helps to confirm that the effect is a result of the intervention and not anything else.

Not resource-intensive like real-world experiments:

Digital experiments are not resource-intensive as compared to offline experiments. It doesn't need any additional funding or arrangements required for real-world experiments. You don't need to recruit participants or tell users they are part of an experiment! So, how exactly is it different from data analytics?

Related: To Stay Successful, You Must Continuously Evolve Your Business

What makes experiments different from analytics?

The data source for the analysis is the fundamental element differentiating experimentation from analytics. Typically, there are two ways to get data for quantitative analysis:

Historical : Historical data includes data stored by the company in their data warehouses about what has happened in the past, which helps understand how users behave on your platform. Historical data helps run various analyses, including user behavior and identifying customer segments.

Experimental : Experiments help you validate business hypotheses as a new change idea will not have essential historical data to validate the change. One could conduct experiments to observe user responses to an app change or feature addition and compare that to the control group's behavior.

Experimentation can be your friend and a business enabler, which is a widely discussed and commonly used process but not usually executed without fallacies. The next post in this series will discuss the key elements that define a successful experiment and the four common reasons for experiment failures .

Entrepreneur Leadership Network® Contributor

CEO of Aryng

Want to be an Entrepreneur Leadership Network contributor? Apply now to join.

Most Popular Red Arrow

'let it go': a couple has spent $400k suing disney after being banned from the park's exclusive 33 club. social media reactions have not been g-rated..

After getting banned from the exclusive members-only club for alleged bad behavior, a California couple has spent a fortune trying to get back to paling around with Mickey.

How a History Student Turned Her Side Hustle Into a Startup That's Raised $7 Million: "People Always Tell Me, 'I Thought I Had to Major In Computer Science to Be an Entrepreneur'"

Audrey Wisch never imagined she'd be one of those kids who dropped out of college to grow her side hustle into a startup. But with the help of AI, her "human-centric" service is scaling up.

The 7-Step ChatGPT Formula for Peak Productivity and Profit

With this powerful solution, you can take your productivity and profitability to new heights with ease.

This NFL Team Is Owned By the Fans — Here's How Much Shares Cost and What Ownership Gets You

The Cheeseheads even own their hat maker.

Taco Bell's New Mountain Dew Baja Blast Gelato Is Causing a Frenzy — But Fans Have One Big Complaint

The company released the dessert to mark the 20th anniversary of the iconic Mountain Dew Baja Blast, which has garnered a cult-like following since its debut in 2004.

5 Strategies That Helped Me Achieve 10x Returns on My Marketing Efforts

These five marketing tactics have delivered remarkable returns for my business.

Successfully copied link

Harvard Business School →
Faculty & Research →

Experimentation Works: The Surprising Power of Business Experiments

Format: Print

Supplemental Information

Find it at Harvard

About The Author

Stefan H. Thomke

Capture New Value from Your Existing Tech Infrastructure

Harvard Business Review

Act Like a Scientist: Great Leaders Challenge Assumptions, Run Experiments, and Follow the Evidence

Porsche By: Stefan Thomke and Daniela Beyersdorfer
Capture New Value from Your Existing Tech Infrastructure By: Stefan Thomke and Anthony Rodrigo
Act Like a Scientist: Great Leaders Challenge Assumptions, Run Experiments, and Follow the Evidence By: Stefan Thomke and Gary W. Loveman

Experimenting: What Makes a Good Business Experiment?

First Online: 12 August 2021

Cite this chapter

Stefan Thomke 3

2103 Accesses

Running good business experiments has become a competitive advantage but many organizations have considerable difficulty executing them. Although the process of experimentation seems straightforward, it is surprisingly hard in practice, owing to myriad organizational, management, and technical challenges. Most tests of new business initiatives are too informal, as they are not based on proven scientific and statistical methods. Based on a study of best practices in leading companies, we will see how you can run good experiments—and make better decisions—by answering seven important questions (and follow scientific principles). The chapter ends with a detailed checklist for good business experiments.

Our success at Amazon is a function of how many experiments we do per year, per month, per week, per day.

—Jeff Bezos, CEO, Amazon

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save.

Get 10 units per month
Download Article/Chapter or eBook
1 Unit = 1 Article or 1 Chapter
Cancel anytime
Available as EPUB and PDF
Read on any device
Instant download
Own it forever
Compact, lightweight edition
Dispatched in 3 to 5 business days
Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

The Use of Experiments in Business Research

Design of Experiments: A Key to Successful Innovation

Sorts and Types of Experiment

Anand B, Rukstad M, Page C (2000) “Capital one financial corporation,” Harvard Business School case no. 700–124. Harvard Business School Publishing, Boston

Google Scholar

Anderson C (2008) The end of theory: the data deluge makes the scientific method obsolete, Wired

Calaprice A (2005) The new quotable Einstein. Princeton University Press, Princeton, NJ

Cross R, Dixit A (2005) Customer-centric pricing: the surprising secret of profitability. Bus Horiz 48:483–491

Article Google Scholar

http://www.tylervigen.com/view_correlation?id=2956 (Accessed April 4, 2018). For a list of more spurious (and fun) correlations, see http://www.tylervigen.com/spurious-correlations

Ionnidis JPA (2005 July) Contradicted and initially stronger effects in highly cited clinical research. J Am Med Assoc 294(2):218–228

Jesdanun J (2017) Amazon deal from whole foods could bring retails experiments. Washington Post, June 16

Kohavi R, Thomke S (2017) The surprising power of online experiments. Harvard Bus Rev, 95(5), 74–82

Lazer D et al (2014, March 14) The parable of Google Flu: traps in big data analysis. Science 343(6176):1203–1205

Manzi J (2012) Uncontrolled: the surprising payoff of trial-and-error for business, politics, and society. Basic Books, New York, pp 132–141

Mayer-Schönberger V, Cukier K (2013) Big data: a revolution that will transform how we live, work, and think. Houghton Mifflin Harcourt, Boston

McCann D (2010) Big retailers put testing to the test. CFOcom , November 8

Pearl J, Mackenzie D (2018) The book of why: the new science of cause and effect. Basic Books, New York

Personal interview with Scott Cook (2018) February 29

Ramachandran S, Flint J (2018) At Netflix, who wins when It’s Hollywood vs. the algorithm? Wall Street J, November 10

SEC archives (2016) 2015 Letter to Amazon Shareholders from CEO Bezos

Thomke S, Manzi J (2014) The discipline of business experimentation. Harvard Business Review, December 2014. The chapter draws extensively from text, concepts, and examples presented in this article. Unless otherwise noted, the examples came from the authors’ interviews with managers and were approved by company designates

Thomke S, Nimgade A (2002a) “Bank of America (A),” Harvard Business School Case No. 603–022. Harvard Business School Publishing, Boston

Thomke S, Nimgade A (2002b) “Bank of America (B),” Harvard Business School Case No. 603–023. Harvard Business School Publishing, Boston

Thomson W (1891) Popular lectures and addresses, vol 1. MacMillan, London, p 80

Download references

Author information

Authors and affiliations.

Harvard Business School, Technology and Operations Management, Boston, USA

Stefan Thomke

You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Stefan Thomke .

Editor information

Editors and affiliations.

ITEM-HSG, University of St. Gallen, St. Gallen, Switzerland

Oliver Gassmann

Zühlke Engineering, Zühlke Group, Eschborn, Germany

Fabrizio Ferrandina

Rights and permissions

Reprints and permissions

Copyright information

About this chapter

Thomke, S. (2021). Experimenting: What Makes a Good Business Experiment?. In: Gassmann, O., Ferrandina, F. (eds) Connected Business. Springer, Cham. https://doi.org/10.1007/978-3-030-76897-3_10

Download citation

DOI : https://doi.org/10.1007/978-3-030-76897-3_10

Published : 12 August 2021

Publisher Name : Springer, Cham

Print ISBN : 978-3-030-76896-6

Online ISBN : 978-3-030-76897-3

eBook Packages : Business and Management Business and Management (R0)

Share this chapter

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Publish with us

Policies and ethics

Find a journal
Track your research

Thinking future and innovation: the power of experimentation in business

Things to consider before employing experiments, the science behind business experimentation, lessons to learn from corporate behemoths: how the big ones do it, experimentation changes the decision making process.

In the world of science, experiments are the norm. It’s a reliable way to see if your ideas work and to check if an innovation is a true breakthrough or something that works well only on paper. Nevertheless, it is always a vision that comes first, then a string of trials and failures until we get the result we want. The same is true for business. Experimentation in business has exploded in recent years, promising concrete results and eliminating guesswork. Sticking to only one method without keeping a watchful eye on the market’s pulse or considering whether your customers actually want a new or improved product or service can cost a company its life. And while most of us like our Coca-Cola in the same container, with a recognisable logo, we also appreciate new and exciting ads. That’s the key to success – experimentation. However, knowing what to change isn’t always clear, but there are several key elements future-focused entrepreneurs may want to consider before starting their experiments.

Great ideas don’t always yield great results. For one, most businesses rely on data from previous experiences. They rely on knowing how their customers react based on already tried and tested products or services. But figuring out how they might respond to a new service you plan to implement is largely based on guesswork, and great ideas don’t always translate into success. Launching new products or services based on not much more than a hunch could cause irreparable damage to your brand. Running experiments, however, can provide valuable insights into your target market and ideas or projects worth investing in.

Keep in mind, though, that experimentation in business isn’t simple. Having a great idea is a great starting point, but the idea alone isn’t enough. To make running business experiments a bit easier, you can outline a checklist with as many steps and details as possible to help you gather the maximum amount of data. It’s always smart, though, to take into consideration what could hypothetically go wrong – and add potential pitfalls to your checklist.

Firstly, it’s important to set the right parameters – meaning, decide exactly what can be measured and at what cost. For example, when you’re running an ad it’s important that you have a clear vision of what you’d like to achieve and then set corresponding parameters. These can include choosing your target market using geographic, demographic, and behaviouristic characteristics, the media you’d use, the time frame during which the ad would be active, to name a few. In short, leave out as little as possible. The more detailed the preparation, the better results. Once the results have been generated, you’ll need a great data analyst to translate all the data into actionable information your product development and marketing team can use. Before running ‘real’ experiments that can be costly, consider using a scientific A/B test model.

A/B testing means that you work with two groups, each treated differently. If you want to see, for instance, how your cold email campaign would work, you can run a test within your organisation. The first step could include choosing two sets of emails and sending them out to your colleagues. One group would receive emails written in your recognisable tone and style, while the other would receive a completely different set of emails sent out over a certain period. You could also include your business partners, as their opinion is valuable, too. Then, run the campaign, collect data, and employ an in-house analyst to help you get actionable information. Bear in mind, however, that the results most likely won’t tell you much as you’re dealing with small groups, but you’d most definitely gain valuable experience from this exercise. To find out which innovation is most effective, it’s essential to run a number of carefully planned experiments. Take Microsoft for example. It discovered that “only one-third of its experiments prove effective, one-third have neutral results, and one-third have negative results.”

The online accommodation platform Booking.com also conducts experiments within their organisation, which can serve as pilot for conducting experiments in a larger ecosystem. As much as 80 per cent of the product development teams is conducting experiments, ranging from customer facing platform experiments to customer service and marketing to partner oriented ones. Given that about 1,500 employees are involved in experimentation, the gathered data is qualitatively as well as quantitatively valuable. Once finished, results are saved in a centralised repository allowing transparency and granting anyone on a team access to see outcomes of previous experiments. Naturally, engaging more employees in the experiments results in getting valid experimental evidence that can later be used to make product-related decisions.

Online accommodation rental giant Airbnb also runs experiments to solve serious challenges like racial discrimination. Airbnb establishment owners/managers as well as customers have been found to have been subjected to racial discrimination. Customer complaints of being rejected because of their skin colour and African-American accommodation hosts earning less money were the reason to run an experiment and determine the extent of the racial discrimination on the platform. Michael Luca and Max Bazerman, authors of the book The Power of Experiments: Decision-Making in a Data Driven World , described running an audit experiment in which 16 per cent of inquiries with distinctively African American sounding names were less likely to rent accommodation than those signed with “white-sounding names”, despite the content of the inquiry being identical. Reportedly, the company gathered a team together to study the case and find a solution. “Research is scheduled to begin in September 2020, and all hosts and guests will have an opportunity to opt out. Starting June 30, we shared details about how the process works and how you can opt out – and everyone will receive at least 30 days’ notice to opt out should they choose not to participate,” stated the company. “We’ll use our partner’s perceptions, for example, to figure out whether the reservations of those seen as a certain race are declined more often than others, which will help us create new features and policies to address any difference. We’ve partnered with civil rights and privacy organizations to make sure we do this work in a way that’s both thoughtful and respectful of everyone’s privacy.”

Knowing what to measure and the right timeframe are also valuable parameters in business experimentation. Take StubHub, the American ticket exchange and resale company, for example. They run experiments to help them decide if they should mention transaction fees upfront or shroud them until the final checkout screen. Their study showed that not mentioning them upfront increased revenue as customers were more likely to purchase when they only saw the transaction costs right at the end of the proces, at the checkout. However, it’s worth noting that this revenue increase might only reflect short term profits. The company tracked customers for a year to find that “these customers were less likely to come back in the next few months, but this is dominated by the revenue effect of increased short term sales.” Still, even a one-year measurement isn’t sufficient to give a clear picture of long term reputation effects of their decision to move away from their ‘no surprise fees’ policy.

Introducing a new product is a challenge for small and large companies alike. Customers are often hesitant to try new products, which can deter companies from rolling out new products or services. This is where running an experiment could help, too. In 2018, Uber tested out a new service called Express Pool, a cheaper alternative to their UberPool. “Uber says these Express Pool trips will be up to 50 percent cheaper than Uber Pool and 70 percent cheaper than Uber X,” reported Engadge t. The service would save passengers a few coins, but they are asked to walk a short distance to meet their ride. The service was introduced in six larger markets, and the gathered data not only showed how their customers reacted to Express Pool, they also managed to compare its impact on existing services. As the results were positive, the company decided to introduce the new service to its major markets.

Experimentation in business makes it possible to test every assumption or parameter in order to make the best possible data-backed decision. Single experiments run on a small scale and within an organisation won’t necessarily generate actionable information, but it will arm you with experience by pointing out potential pain points. This makes it easier to outline detailed experiments, decide which parameters should be measured, and determine the right timeframe. Supporting your teams to experiment with new ideas is important, and a scientific approach to making data-driven decisions is the key to success.

Generic talks

Trends 2030
AI revolution
AI for managers
The company of the future
The future of leadership

Topic talks

The future of crime and security
The future of marketing
The future of work & HRM

Industry talks

The future of agriculture
The future of construction
The future of education
The future of finance
The future of healthcare
The future of retail
The future of logistics
The future of smart cities
The future of smart governments

Renowned keynote speaker Richard van Hooijdonk offers inspiring lectures on world trends, technology, and marketing.

Terms & conditions

More From Forbes

Fostering business innovation with experimentation.

Share to Facebook
Share to Twitter
Share to Linkedin

By Maria Thimothy, Sr. Consultant at OneIMS, helps businesses grow by creating and capturing demand and managing and nurturing relationships.

Starting and running a successful business can be challenging, with lots of decisions to be made and risks to take. So, it is no surprise that when things are running according to plan, most businesses are reluctant to deviate from the beaten path once it has proven successful. After all, there is that adage: If it ain’t broke, don’t fix it.

However, sticking to only one approach can lead to rigid tunnel vision and consequently prevent the innovations required to keep a business relevant and profitable. Innovation is the key to the evolution of business and requires an open mind and willingness to experiment.

While testing new business ideas can seem intimidating, you can mitigate the risk by understanding how to develop innovations and make calculated decisions on where and when to experiment.

Adopt An Environment Of Innovation

Most innovation is not the result of a "lightbulb" moment by a single individual. Instead, innovation is a process that happens best in an open and collaborative environment where people are encouraged to share ideas as part of a team. If you are more inclined to think, "But this is the way we have always done it," rather than, "Sure, let’s give that a shot," when you hear new ideas, you are probably at risk of suppressing innovation rather than fostering it.

Of course, not all new ideas are good, but the more new ideas you encourage your team to share and develop, the more opportunities and choices you have to evolve your business. Set aside time for brainstorming sessions and keep an open-door policy when it comes to new ideas and suggestions. Once you hit on a promising idea, you can take it from the sandbox to the real world.

Seek Proof Of Concept

Proof of concept is the process of gathering evidence that establishes if a new approach is potentially viable and provides feedback about what the benefits or pitfalls may be. It is testing to see if the new idea is feasible on a small scale before developing it further. This could be anything from presenting a new landing page to a select group of testers and getting their feedback to trying a new marketing strategy on a social media platform to sending out free samples of a product to customers.

Whatever the concept, the idea is to generate enough feedback to determine if it is worth rolling out on a bigger scale. Often proof of concept is right in front of us, and we do not even realize it. For example, social media marketing works for millions of businesses around the world, so strong data already supports at least trying it before writing it off as "not for us."

Ask A Professional

If you have a good idea and evidence to suggest that it is worth exploring further but aren’t sure how, you should seek professional advice. Often the reluctance to innovate and experiment is the result of a lack of in-house expertise. This is especially true for small businesses who do not have a large staff and are therefore less likely to have dedicated team members responsible for things like website development, marketing or social media.

Professional marketing firms or consultancies work with a wide range of businesses and have experience in experimenting and implementing innovation and are therefore well placed to help you test your ideas and interpret the feedback metrics. Once you have more data, these firms can then help you with long-term implementation.

The Next Big Idea

Innovation doesn’t always mean radical changes that come with a big price tag. Sometimes just considering a different protocol or approach in one area of the business can lead to unexpected positive effects in other areas. For example, many companies have reported increased productivity during the pandemic despite employees working from home and are now considering saving money by doing away with office space altogether.

Sometimes you are forced to innovate, and other times it is voluntary, but putting the concept to the test is key to success. At worst, you may discover that a particular change is unnecessary, but at best, you could hit upon the next big idea to take your business to the next level.

Editorial Standards
Reprints & Permissions

A Harvard Business School professor on how companies like Google and Amazon use experimentation to innovate, grow, and improve

Stefan H. Thomke is the William Barclay Harding professor of business administration at Harvard Business School.
The following is an excerpt from his book, " EXPERIMENTATION WORKS: The Surprising Power of Business Experiments ."
In it, he writes best practices in business experimentation and illustrates how these practices work at leading companies, including Amazon, Google, and Nike.
The scientific method helps businesses determine what works and what doesn't, ultimately producing valuable products and ideas.
Visit Business Insider's homepage for more stories .

Four hundred years ago, in 1620, Francis Bacon published "Novum Organum," the classical formulation of a new instrument for building and organizing knowledge: the scientiﬁc method. Thinking and acting scientiﬁcally has had an enormous impact on the world. For centuries, we've built and organized scientiﬁc and technological knowledge through testable explanations and predictions. These, in turn, have given us modern medicine, food, energy, transportation, communication, and so much more. The engine that has powered the scientiﬁc method is the humble experiment.

And today, companies as varied as Google, Booking.com, Nike, Kohl's, State Farm Insurance, and the BBC are running experiments to fuel innovation — to roll out new products, improve customer experiences, and try new business models. These organizations have discovered that an "everything is a test" mentality yields surprisingly large payoffs and competitive benefits, and may even help stock performance. I've spent over 25 years studying experimentation in businesses and, along the way, beneﬁted tremendously from the work of many scholars and practitioners. I think that they would all agree with me: experimentation works.

Why? Consider the cautionary tale of Ron Johnson. Soon after he left Apple to become the CEO of JC Penney in 2011, he led his team to implement a bold new plan. Under his leadership, the company eliminated coupons and clearance racks, ﬁlled stores with branded boutiques, and used technology to eliminate cashiers, cash registers, and checkout counters. Yet after just 17 months, sales had plunged, losses had soared, and Johnson had lost his job.

How could Penney have gone so wrong? Didn't it have lots of transaction data revealing customers' tastes and preferences? What about Johnson's experience in creating Apple's highly successful store concept, which redeﬁned the customer in-store experience with innovations like the Genius Bar and cashier-free checkout? Those innovations led to the highest average retail sales per square foot of any retailer worldwide with stores and more visitors than Disney's theme parks. The Penney board must have hoped that Johnson would repeat Apple's retail success at the old department store chain, with its more than one thousand United States locations. Why didn't that happen?

For one thing, most managers operate in a world where they lack sufﬁcient data or relevant experience to inform their innovation decisions. That is, there may be transaction data, but that information provides clues only about past behavior, not about how customers might react to future changes. Oftentimes, too, managers rely on their intuition — but ideas that are truly innovative typically go against experience. In fact, most ideas don't work. Whether it's improving customer experiences, trying out new business models, or developing new products and services, even the most experienced business leaders are often wrong.

Not all is lost, however. The good news is that managers can discover whether a change in product, service, or business model will succeed. They can do that by subjecting it to a rigorous experiment. Think of it this way: A pharmaceutical company would never introduce a drug without ﬁrst conducting a round of experiments based on established scientiﬁc protocols (in fact, the US Food and Drug Administration requires extensive clinical trials). Yet that is essentially what many companies do when they roll out new business models and other novel changes. Had Penney run rigorous experiments on its CEO's proposed innovations, the company might have discovered that, notwithstanding the success of these innovations at Apple, Penney customers would probably reject them. Such a rejection would have not been surprising, given the long odds against any innovation. In fact, Microsoft has found that only one-third of its experiments prove effective, one-third have neutral results, and one-third have negative results.

Had Penney tested extensively, it would have found itself in good company. Google employs extensive experimentation in its ongoing quest for the best customer experience. Even its experts get it wrong most of the time. Eric Schmidt, its former CEO, disclosed the odds in a 2011 Senate testimony:

To give you a sense of the scale of the changes that Google considers, in 2010 we conducted 13,311 precision evaluations to see whether proposed algorithm changes improved the quality of its search results, 8,157 side-by-side experiments where it presented two sets of search results to a panel of human testers and had the evaluators rank which set of results was better, and 2,800 click evaluations to see how a small sample of real-life Google users responded to the change. Ultimately, the process resulted in 516 changes that were determined to be useful to users based on the data and, therefore, were made to Google's algorithm. Most of these changes are imperceptible to users and aﬀect a very small percentage of websites, but each one of them is implemented only if we believe the change will beneﬁt our users.

In other words, Google's experts missed their mark 96.1% of the time. The low (3.9%) success rate includes less rigorous tests, such as click evaluations. At Google and Bing, about 10% to 20% of controlled experiments generate positive results. But it's precisely that capability — to test what does and does not work at a huge scale — that has given the company an advantage against its competitors. Scott Cook, the cofounder of Intuit and a former Amazon director, recalled former Yahoo executives saying as much: "'[Google] just outran us,' they said. 'We didn't have that experimentation engine.'" Even Yahoo's highly publicized project Panama — launched in 2007 as an eﬀort to close the wide gap with Google in the race for advertising dollars — couldn't erase the advantage of Google's ferocious experimentation, which was the company's system of continuous improvement.

A company's ability to create and reﬁne its products, customer experiences, processes, and business models — in other words, to compete — is deeply aﬀected by its ability to experiment.

The rationale behind experimentation is the pursuit of knowledge about cause and eﬀect; all experiments yield information through understanding what does, and does not, work. For centuries, scientists and engineers have relied on experiments, guided by their insight and intuition, to learn new information and advance knowledge. Experiments have been conducted to characterize naturally occurring processes, to decide among competing scientiﬁc hypotheses, to ﬁnd hidden mechanisms of known eﬀects, and to simulate what is diﬃcult or impossible to research through observation — in short, to inductively establish scientiﬁc laws.

In the business world, experiments have led to the discovery of both technical solutions and new markets. A classic example of both is the discovery of 3M's Post-it Note. The story begins in 1964, when 3M chemist Spencer Silver started a series of experiments aimed at developing polymer-based glues. As Silver recalled: "The key to the Post-it adhesive was doing the experiment. If I had sat down and factored it out beforehand, and thought about it, I wouldn't have done the experiment. If I had limited my thinking only to what the literature said, I would have stopped. The literature was full of examples that said that you can't do this."

Although Silver discovered a new glue with unique properties — a high level of "tack" but low adhesion — it would take 3M at least another ﬁve years to ﬁnd a market. Silver kept trying to sell his glue to other departments at 3M, but they were focused on ﬁnding a stronger glue that formed an unbreakable bond, not a weaker glue that only supported a piece of paper. Market tests with diﬀerent concepts (such as a sticky bulletin board) were telling 3M that the Post-it concept was hopeless — the adhesive just didn't solve any known customer problems — until Silver met Arthur Fry. Fry, a chemist and choir director, observed that members of his choir would frequently drop bookmarks when switching between songs. "Gee," wondered Fry, "if I had a little adhesive on these bookmarks, that would be just the ticket." This "Eureka moment" launched a series of experiments with the new glue that broadened its applicability and ultimately led to a paper product that could be attached and removed without damaging the original surface. In other words, repeated experimentation was instrumental in ﬁnding the now-obvious solution to a frustrating customer problem once the Eureka moment occurred.

While such Eureka moments make for memorable stories, they do not give a complete account of the various experimentation strategies, tools, processes, and histories that lead to innovative solutions. After all, such moments are usually the result of many failed experiments and accumulated learning that prepare the experimenter to take advantage of the unexpected. "Failure and invention," notes Amazon's CEO Jeﬀ Bezos, "are inseparable twins. If you already know it's going to work, it's not an experiment." Consider what the authors of a careful study of Thomas Edison's invention of the electric light bulb concluded:

This invention [the electric light], like most inventions, was the accomplishment of men guided largely by their common sense and their past experience, taking advantage of whatever knowledge and news should come their way, willing to try many things that didn't work, but knowing just how to learn from failures to build up gradually the base of facts, observations, and insights that allow the occasional lucky guess — some would call it inspiration — to eﬀect success.

When management aims for big results, however, they cannot rely on lucky guesses, experience, or intuition alone. Their companies' business experiments must be disciplined, organizationally aligned, supported by an infrastructure, and culturally embraced; that is, running experiments should be as normal as running the numbers. At the same time, the serendipitous breakthroughs may be more likely to occur when managers are clear that understanding what does not work is as important as learning what does.

Reprinted by permission of Harvard Business Review Press. Excerpted from EXPERIMENTATION WORKS: The Surprising Power of Business Experiments by Stefan H. Thomke. Copyright 2020 Stefan H. Thomke. All rights reserved.

Watch: This Facebook exec cofounded and then got fired from Pets.com. Here's why she is no longer hiding from this failure.

Main content

Pollfish School
Market Research
Survey Guides
Get started
What is Experimental Research & How is it Significant for Your Business

What is Experimental Research & How is it Significant for Your Business

Experimental research uses a scientific method for conducting research, employing the most methodical research design. Known as the gold standard, it involves performing experiments to reach conclusions and can be conducted based on some of the findings from previous forms of research.

Logically, it would follow correlational research, which studies the relationships between variables. It can also follow causal research , a kind of experimental research in itself, as it establishes cause and effect relationships between previously studied variables.

Experimental research is typically used in psychology, physical and social sciences, along with education. However, it too can be applied to business.

This article expounds on experimental research, how it is conducted, how it differs from other forms of research, its key aspects and how survey studies can complement it.

Defining Experimental Research

Experimental research is a kind of study that rigidly follows a scientific research design. It involves testing or attempting to prove a hypothesis by way of experimentation . As such, it uses one or more independent variables, manipulating them and then using them on one or more dependent variables .

In this process, the researchers can measure the effect of the independent variable(s) on the dependent variable(s). This kind of study is performed over some time, so that researchers can form a corroborated conclusion about the two variables.

The experimental research design must be carried out in a controlled environment .

Throughout the experiment, the researcher collects data that can support or refute a hypothesis, thus, this research is also referred to as hypothesis testing or a deductive research method.

The Key Aspects of Experimental Research

There are various attributes that are formative of and unique to experimental research in addition to its main purpose. Understanding these is key to understanding this kind of research in-depth and what to expect when performing it.

The following enumerates the defining characteristics of this kind of research:

It includes a hypothesis, a variable that will be manipulated by the researcher along with the variable that will be measured and compared .
The data in this research must be able to be quantified.
The observation of the subjects, however, must be executed qualitatively.
The latter is rarer, as it is difficult to manipulate treatments and to control external occurrences in a live setting.
It relies on making comparisons between two or more groups (the variables).
Some variables are given an experimental stimulus called a treatment; this is the treatment group.
The variables that do not receive a stimulus are known as the control group.
First, researchers must consider how the variables are related and only afterward can they move on to making predictions that can be tested.
Time is a crucial component when putting forth a cause-and-effect relationship.
Pre-experimental research design
True experimental research design
Quasi-experimental research design

The Three Types of Experimental Research

Experimental research encompasses three subtypes that researchers can implement. They all fall under experimental research, differing in how the subjects are classified. They can be classified based on their conditions or groups.

Pre-experimental research design:

This entails a group or several groups to be observed after factors of cause and effect are implemented.

Researchers implement this research design when they need to learn whether further investigation is required for these particular groups.
One-shot Case Study Research Design
One-group Pretest-posttest Research Design
Static-group Comparison

Quasi-experimental Research Design

Representing half or pseudo, the moniker “quasi” is used to allude to resembling true experimental research, but not entirely.

The participants are not randomly assigned, rather they are used when randomization is impossible or impractical.
Quasi-experimental research is typically used in the education field.
Examples include: the time series, no equivalent control group design, and the counterbalanced design.

True Experimental Research Design

This kind of experimental research design studies statistical analysis to confirm or debunk a hypothesis.

It is regarded as the most accurate form of research.
True experimental research can produce a cause-effect relationship within a group.
A control group (unaltered) and an experimental group (to undergo changes in variables)
Random distribution
Variables can be manipulated

Why Your Business Needs Experimental Research

There are various benefits to conducting experimental research for businesses. Firstly, this form of research can help businesses test a new strategy before fully engaging in/ launching it.

The strategy can involve anything from content marketing strategy, to a new product launch. This is especially useful for technology companies, which conduct experimentation frequently. In fact, this kind of research is essential to an R & D (research and development) department.

This makes experimental research a much-needed effort when it comes to spurring innovation. Whether it involves a slight rebranding or an upgrade of products, experimental research guides these campaigns in a science-backed manner.

Secondly, a business must excel in meeting customer needs. Customer experience is an overwhelmingly important side of any business, as customers are willing to make on-the-stop purchases and pay more for a good CX .

As such, each product addition and change in a customer journey must be carried out wisely. Businesses ought to avoid creating unwanted services, or those that cause any aversion within customers. Instead, they should only invest in the most profitable services, products and experiences, a feat that cannot be accomplished solely on guesswork.

Experimenting allows brands to understand customer preferences and changes in their behaviors , as the experiments create stimuli and changes in independent variables.

Additionally, experimental research grants companions an understanding of their business environment. In turn, this helps them predict outcomes, or create hypotheses about outcomes to guide them in further research, if need be. For example, a business may consider testing the reactions of its competitors should it raise its costs on various offers.

Aside from discovering if this yields a profitable change, it can discover how companies in the same niche respond and if those responses drive more sales, etc.

Key Independent Variables

Digital user experience (DX) such as new site features
Advertisements
Marketing activity (SEO, SEM, social media announcements, retargeting, etc.)
Inventory (new products or upgrades)
Interactions with sales agents

Key Dependent variables

VoC feedback (whether positive or negative)
Site traffic
In-store visits
Time spent on a website, bounce rates, etc.

An Example of Experimental Research for Business

Market researchers can apply experimental research to a wide breadth of testing needs. Virtually anything that requires proof, confirmation, or is clouded by uncertainty can put experimentation into practice.

The following is an example of how a business can use this research:

A product manager needs to convince the higher-ups in a denim company to launch a new product line at a particular department store. The objective of this launch is to increase sales, expand the company’s floor presence and widen the offerings.

The manager has to prove that this line is needed in order for the company to pitch the idea to the department store. The product manager can then conduct experimental research to provide a strong case for their theory, that a new line can raise sales.

The product manager performs experimental research by executing a test in a few stores, in which the new line of denim is sold. These stores are varied in location to signify the target market sales before and after the launch. The test runs for a month to determine if the hypothesis (the new line resulting in increased attention and sales) can be proven.

This represents a field experiment. The product manager must heed the sales and foot traffic of the new product line, paying attention to spikes in revenue and overall sales to justify the new line.

Experimental Research Survey Examples

Survey research runs contrary to experimental research, unlike the other main forms of research such as exploratory, descriptive and correlational research. This is because the nature of surveys is observational, while experimental research, as its name signifies, relies on experimentations, that is testing out changes and studying the reactions to the changes.

Despite the contrast of survey research to experimental research, they are not completely at odds. In fact, surveys are a potent method to gain further insight into an existing experiment or understand variables before conducting an experiment in the first place.

As such, businesses can adopt a wide variety of surveys to complement their experimental research. Here are some of the key forms of surveys that work in tandem with experimentation:

Discovers the aspects of statistical significance within variables.
Helpful in that causal research is quantitative in essence.
Delves into past events, occurrences and attitudes in regards to the variables.
Shows whether the variables changed and how so.
Can find causative elements between variables over a period of time.
Useful for formulating hypotheses.
Helps businesses zero in on variables that contribute to or result from certain kinds of customer experiences.
Allows businesses to test CX in relation to the responses from this survey.
Measures various matters critical in a business or organization; surveys employees.
Deployed more frequently, so variables can always be continually tracked.
Helps answer the what, why and how with open-ended questions.
Extracts key high-level information in depth.

How Experimental Research Differs from Correlational, Exploratory, Descriptive and Causal Research

Experimental research differs from exploratory, descriptive and correlational research in self-evident ways. It is, however, often conflated with causal research. However, they too have notable differences.

Causal research involves finding the cause-and-effect relationships between variables. Thus, it too employs experimentation. However, this means that causal research is a form of experimental research, not the other way around.

Experimental research, on the other hand, is fully science and experiment-based, as it chiefly seeks to prove or disprove a hypothesis. While this largely involves studying independent and dependent variables, as it does in causal research, it is not solely based on these aspects. Instead, it can introduce a new variable without knowing the dependent variable or experiment on an entirely new idea (as in the example used in the previous selection).

Causal research looks into the comparison of variable relationships to find a cause and effect, while experimental research states an expected relationship between variables and is bent on testing a hypothesis.

As far as comparisons to correlational research go, while experimental research also studies the relationships between variables, it functions far beyond this by manipulating the variables and virtually all subjects involved in experiments .

On the contrary, correlational research does not apply any alterations or conditioning to variables. Instead, it is a purely observational research method. As such, it merely detects whether there is a correlation between only 2 variables. In contrast, experimental research studies and experiments with several at a time.

Exploratory research is vastly different from experimental research, as it forms the very foundation of a research problem and establishes a hypothesis for further research. As such, it is conducted as the very first kind of research around a new topic and does not fixate on variables.

Descriptive research , like exploratory research and unlike experimental research, is conducted early in the full research process, following exploratory research. Like exploratory research, it seeks to paint a picture of a problem or phenomenon , as it zeros in an already-established issue and delves further, in pursuit of all the details and conditions surrounding it.

Thus, unlike experimental research, it only observes; it does not manipulate variables in any capacity or setting.

The Advantages and Disadvantages of Experimental Research

Experimental research offers several benefits for researchers and businesses. However, as with all other research methods, it too carries a few disadvantages that researchers should be aware of.

The Advantages

Researchers have a full level of control in an experiment.
It can be used in a wide variety of fields and verticals.
The results are specific and conclusive.
The results allow researchers to apply their findings to similar phenomena or contexts.
It can determine the validity of a hypothesis, or disprove one.
Researchers can manipulate variables and use them in as many variations as they desire without tarnishing the validity of the research.
It discovers the cause and effect among variables.
Researchers can further analyze relationships through testing.
It helps researchers understand a specific environment fully.
The studies can be replicated so that the researchers can repeat their experiments to test other variables or confirm the results again.

The Disadvantages

It involves a lot of resources, time and money, as such, it is not easy to conduct.
It can form artificial environments when researchers unwittingly over-manipulate variables as a means of duplicating real-world instances.
It is vulnerable to flaws in the methodology, along with other mistakes that can’t always be predicted.
Flawed experiments may require researchers to start their experiments anew to avoid false calculations, measuring results from artificial scenarios or other mistakes.
Some variables cannot be manipulated and some forms of research experiments are too impractical to conduct.

How to Conduct Experimental Research

Experimental research is often the final form of research conducted in the research process and is considered conclusive research. The following explains the general steps required to successfully complete experimental research.

Form a specific research question.
Gather all available literature and other resources around the subject.
Conduct secondary research around the subject and primary research via surveys .
Consider how they relate to your question and how they line up with the secondary research you conducted.
After your initial studies, form a hypothesis.
First, decide which variable(s) is dependent/ independent (if it doesn’t involve experimenting).
Decide how far to vary the independent variable.
In the experiment, manipulate the independent variable(s).
Measure the dependent variable(s) while you study the independent variable(s) alongside.
Make sure to control potential confounding variables.
Keep the study size in mind; a larger study pool creates statistical findings.
Assign your subjects to “treatment” groups randomly, with each to receive a different level of “treatment.”
Use a control group, which receives no manipulation. This shows you the test subjects as they appear/behave without any experimental intervention.
Completely randomized design: every subject gets randomly assigned to a treatment.
Randomized block design : aka stratified random design, subjects get first grouped based on a shared characteristic, then assigned to treatments within their groups at random.
Independent measure : subjects receive only one of the possible levels of an experimental treatment.
Repeated measures design : every subject gets each of the experimental treatments consecutively, as their responses are measured. It also refers to measuring the effect of an emerging effect over time.
Continue experimenting on variables as needed, take measurements and take notes.
Based on your experiment(s), put together a logical conclusion. It is possible that it may need testing over time.

Using Experimental Research and Going Further

Although experimental research can be very complex, this research method is the most conclusive. Using a scientific approach, it can help you form tests on various business matters. While it is critical for understanding your target market’s and customers’ existing behaviors, it can also be used to experiment on a wide variety of other matters.

Before launching a new product, or an updated one, for example, you can conduct an experiment to understand the product in action. This helps you avoid any glitches or undesirable qualities that will incur problems for your customs and a bad reputation for your brand.

Experimental research is not for every business, yet if you decide to implement this form of research, consider using surveys in tandem. An online survey platform can help you establish and distribute your surveys to a wide network via organic sampling to avoid biases.

Although it isn’t a requirement, in today’s age of excelling in customer experience (CX), it is of the essence to have as much data on your target market as possible. An online survey tool makes this possible.

Do you want to distribute your survey? Pollfish offers you access to millions of targeted consumers to get survey responses from $0.95 per complete. Launch your survey today.

Privacy Preference Center

Privacy preferences.

Business Management
Career development
Communication & Skills
Finance & Accounting
Marketing & Sales
Self introduction
Strategy & Innovation
Business Tools

What is Experimentation, and Why Is It Crucial for Businesses Today?

Disclaimer : We sometimes use affiliate links in our content. For more information, visit our Disclaimer Page .

In the business world, experimentation is vital. You never know what will work until you try it. This is especially true in the digital age, where things are constantly changing and evolving. To stay ahead of the competition, you need to be willing to experiment with new strategies and tactics.

This blog post will discuss what experimentation is and why it is crucial for today’s businesses. We will also provide some tips for how you can start experimenting with your own business!

What is Experimentation?

Experimentation is the process of conducting controlled tests or trials to explore a hypothesis. This process can be used to answer questions about the natural world or to develop new technologies. In addition, experimentation can help scientists and engineers understand how things work and improve their designs.

One common type of experimentation is the scientific experiment, which follows a rigorous experimental protocol to ensure the validity of the results. Other types of experimentation include product testing, market research, and usability testing.

Experimentation is an integral part of the scientific method, and it can be used to test new theories, hypotheses, and ideas. In addition, by conducting controlled tests and trials, scientists can better understand how the world works and how new technologies can be used.

10 Great Benefits of Experimentation

Experimentation is essential for businesses because it tests new ideas and strategies. Without experimentation, companies would be stuck using the same old methods, eventually stagnating. Experimentation will enable businesses to find new ways of doing things and stay ahead of the competition.

There are many benefits of experimentation, but some of the most important ones are listed below.

1. Leads to discoveries and innovations

Often, experimentation is the key to making discoveries and innovations. We can learn about the world around us and how it works by trying new things. This trial and error process can help us find new ways of doing things or improving existing methods.

In many cases, it is only through experimentation that we can unlock the potential of new technologies or ideas. We can see what works and doesn’t and find ways to make things better by testing different possibilities.

This is particularly important in the world of science and technology. Scientists often have to try out new ideas and theories to make progress and see what happens. This process of experimentation is essential to scientific progress and has led to some of the most important discoveries in history.

2. Helps us learn about the world around us

One of the best ways to learn about the world around us is through experimentation. We can understand how things work and why they happen by testing and observing the results.

Experimentation also allows us to see the world in new and different ways. We can open our eyes to new possibilities and ways of thinking by trying new things.

So why not get started on your experiments today? See what you can learn about the world around you! Who knows, you might just be surprised at what you discover.

3. Experimentation can help us solve problems

Experimentation is a critical scientific process that helps us to better understand the world around us. By designing and conducting experiments, we can test hypotheses and collect data that can be used to support or refute our ideas about how things work.

In many cases, experimentation is the only way to gain reliable knowledge about something. For example, it would be impossible to know whether or not a new medication is effective without first testing it on people. Similarly, we would never have developed today’s technology without first experimenting with different designs and materials.

Experimentation can be used to solve all sorts of problems, both big and small. For instance, trying different methods for growing crops can help farmers find the most efficient and sustainable way to produce food. Likewise, testing different ways to treat an illness can lead to discovering new, more effective treatments.

4. Help us improve products and services

The best way to improve our products and services is through experimentation. By experimenting, we can see what works and what doesn’t and then make changes accordingly. This allows us to constantly improve our offerings, increasing customer loyalty, and satisfaction.

In addition, experimentation can also help us develop new ideas for products and services. We can see what customers want and need by trying out different things and then creating new offerings that meet those needs. This helps us stay ahead of the competition and maintain a leadership position in our industry.

Thus, experimentation is essential to improving our products and services. OVerall helps us understand what consumers want and require, and it allows us to improve our offerings constantly. This, in turn, leads to increased customer satisfaction and a stronger competitive position.

5. Experimentation can help us learn more about ourselves

Experimentation can be used in many different ways to help us learn more about ourselves. For example, we can experiment with new foods to see how our bodies react or experiment with different ways of approaching relationships to see what works best for us.

Ultimately, experimentation can help us learn more about who we are and what we want. In addition, by taking the time to experiment, we can better understand our strengths and weaknesses, which can help us make more informed decisions in the future.

6. Experimentation can help us improve our relationships

We all want to improve our relationships, but sometimes it’s hard to know where to start. Experimentation can be a great way to explore new things and find out what works best for you and your partner. By trying new things and communicating openly, you can learn more about each other and yourself and build a stronger relationship.

Here are a few ideas for experiments you can try in your relationship:

Try a new activity together.
Talk about something you’ve never talked about before.
Set a goal together and work towards it.
Do something nice for your partner without them expecting it.
Express your feelings to your partner in a new way.
Try something that’s usually considered “taboo” within your relationship.

The sky’s the limit for experimentation, so get creative and have fun! The important thing is to be open and communicate honestly with each other. With a bit of effort, you can use experimentation to help take your relationship to the next level.

7. Can help us better understand customers and target market

A big part of the marketing team in the business is to run experiments. We can learn a lot about our customers and target market by experimenting with different tactics and strategies. For example, we might try other pricing models to see what works best or test out other marketing messages to see which ones resonate the most with our audience.

We can better understand what our customers want and how they think by experimenting. We can also learn what doesn’t work and what to avoid in the future. Overall, experimentation is a vital tool that can help us better understand our customers and target market.

8. Experimentation can help us reduce risk

In today’s business world, the ability to take risks is essential for success. However, not all risks are created equal. Some risks can lead to disastrous results, while others can be highly beneficial. Therefore, knowing which risks to take and avoid is critical for any business owner or manager .

One tool that can help us reduce the risk of taking unnecessary risks is experimentation. By experimenting with different options, we can better understand what works and what doesn’t. This allows us to make more informed decisions, which leads to fewer risky gambles.

9. Increase chances of success

In business, as in life, experimentation is the key to success. By trying new things, you increase your chances of finding what works for you. This is especially important when it comes to marketing and sales. You never know what will resonate with your audience until you try something new.

One way to experiment with your marketing is to test different strategies. For example, you can try running ads on other platforms or using different types of content. You can also experiment with your pricing strategies and see which ones generate the most sales.

Another way to experiment is to try new methods of reaching your audience. For example, you might want to try using social media to promote your product or reach out to bloggers to get them to write about your product. You can also experiment with how you deliver your message and see which methods are most effective.

The key to experimentation is to be willing to try new things. Keep an open mind, and be prepared to fail sometimes. But if you’re eager to experiment, you’ll increase your chances of finding a winning strategy. And that’s how you can increase your chances of success.

10. Experimentation is fun!

People often think of experimentation as a dry, scientific process carried out in a lab. However, experimentation can be enjoyable! Here are just a few reasons why experimentation is so much fun:

Allows you to explore new things:

It helps you learn more about the world around you.
Great way to bond with friends or family members.
It can be a fun way to relieve stress.
Also, a great way to challenge yourself.

So, next time you’re bored or uninspired, why not try experimenting? You may find that you enjoy it!

Related: Benefits of Research and Development

What are the three types of Experiments?

There are three main types of experiments listed below: controlled, field, and natural experiments.

1. Controlled Experiment

A controlled experiment is a scientific procedure in which one or more variables are manipulated to observe the effects on another variable. It is important to note that the controlled experiment is the gold standard of scientific research – it allows researchers to isolate and identify the effects of a specific variable while ruling out all other potential factors.

To be considered valid, a controlled experiment must be conducted in a completely controlled environment, meaning that all other variables must be kept constant. This allows the researchers to draw accurate conclusions about the effects of the investigated variable.

One of the key benefits of a controlled experiment is that it allows researchers to control for confounding variables – factors that may affect the results but are not explicitly investigated. This is done by holding all other variables constant, which allows the researcher to isolate the effects of the variable they are interested in.

There are many different types of controlled experiments, but all follow the same basic structure:

First, a research question is defined, and a hypothesis is formulated.
Then, an experiment is designed to test the hypothesis.
Finally, the results of the experiment are analyzed and interpreted.

2. Field Experiments

Field experiments are a type of research that scientists use to study the natural world. This type of experiment is done outdoors, in a real-world setting. Field experiments can be used to learn anything from the behavior of animals to the effects of climate change.

One advantage of field experiments is that they can provide more accurate results than lab experiments. This is because field experiments occur in a real-world setting, where many variables can affect the results.

Another advantage of field experiments is that they can be used to study rare events. For example, if a scientist wants to study a natural disaster, such as a hurricane, it would be complicated to do this in a lab setting. However, scientists can study the hurricane in its natural environment by doing a field experiment.

Field experiments have been used for many years, and they continue to be a valuable tool for scientists. For example, there has been an increase in field experiments for climate change research in recent years. This is because field experiments can help scientists understand the complex effects of climate change on the natural world.

Overall , Field experiments are a valuable tool for scientists, and they will continue to play an essential role in research.

3. Natural Experiments

Natural experiments are a type of scientific study that uses naturally occurring events to answer questions about the effects of variables on specific outcomes. These studies can assess the efficacy of policies or interventions and understand the underlying mechanisms at work.

One of the advantages of using natural experiments is that they avoid any potential biases in other types of studies. For example, experiments that use human subjects may be biased by the expectations of the researchers or the participants. In addition, natural experiments are often less expensive and time-consuming than traditional experiments.

Despite these benefits, natural experiments are not always possible to execute, and they can be challenging to interpret. Additionally, the results of natural experiments may not be generalizable to other settings.

What is Experimentation in Business with Examples?

There is no single answer to this question, as experimentation in business can take many different forms. However, at its core, experimentation in business means trying new things to find better or more efficient ways of doing things. This could involve anything from testing new marketing strategies to experimenting with production processes.

Some notable examples of businesses that have embraced experimentation include Google, famous for its use of data-driven experimentation, and Amazon, which constantly uses A/B testing to test new features on its website. By continually experimenting, these companies have achieved exceptional levels of growth and success.

The randomized controlled trial (RCT) has been the scientific gold standard for causality since 1948. However, in our modern era of digital business, the experimentation process is becoming increasingly crucial for companies looking to innovate and stay ahead of the curve.

At its core, experimentation constantly tests new hypotheses and ideas to learn what works and what doesn’t. By running experiments, businesses can validate or invalidate their assumptions quickly and cheaply, without making major bets on untested ideas.

In the past, experimentation was mostly used in product development and marketing. However, with the rise of data-driven decision-making, experimentation is now being applied in nearly every corner of the business.

The key to a successful experimentation program has a well-defined process in place. This process should include clearly defined goals, a well-thought-out experiment design, and a rigorous analysis of the results.

Businesses that embrace experimentation will be better positioned to survive and thrive in today’s competitive landscape.

Best Tool To Start Experimentation in Business

Businesses need to be nimble and quick to adopt new methods and technologies to stay ahead of the competition in an ever-changing world. This is especially true in the field of sales and marketing, where adopting new technologies can give businesses a competitive edge.

The best tool for experimentation in 2022 is Zia AI , a sales assistant that uses data mining and machine learning to deliver key business information such as sales predictions, suggestions, and alerts. With Zia AI, businesses can understand their customer’s behavior and deliver personalized marketing campaigns and other services that can help them improve sales.

Zia AI is based on what goes into the system, but she even learns how each salesperson uses Zoho CRM from all her analyses. This gives businesses an edge over their competition, as they can personalize their sales and marketing strategies to suit their customers’ needs better.

With such a powerful tool at their disposal, businesses can confidently experiment with new methods and technologies to stay ahead of the curve and improve their sales. So if you’re looking for the best tool to start experimentation in 2022, look no further than Zia AI.

13. Ways To Foster A Culture of Experimentation in Business

Experimentation is essential for any business that wants to stay ahead of the competition and continue to grow. By promoting a culture of experimentation, you will give your employees the freedom to develop new ideas and solutions to help the company thrive. Below are just a few of how you can foster a business environment of experimentation within your company.

1. Encourage a diversity of perspectives

A culture of experimentation cannot thrive if only one perspective is represented. Diversity of thought is essential to generating new ideas and testing them out. Encourage employees to share their unique perspectives and different welcome points of view.

2. Promote risk-taking

For experimentation to occur, there must be some element of risk involved. Promote an environment where employees feel comfortable taking risks and trying new things. Encourage them to view failure as a learning opportunity rather than a setback.

3. Encourage idea generation

A key part of experimentation is generating new ideas. Create an environment where employees feel comfortable sharing their ideas and brainstorming. Encouraging idea generation will help to foster a culture of experimentation.

4. Foster a learning mentality

To experiment, employees need to be willing to learn from their mistakes. Foster a learning mentality within your entire company by encouraging employees to ask questions and explore new ideas. Help them to see failure as a way to learn and grow.

5. Celebrate failure

One of the biggest inhibitors of experimentation is the fear of failure. To overcome this, it’s important to celebrate failure. When employees know that failing is not bad, they will be more willing to experiment. So show your appreciation for employees who take risks and experiment.

6. Encourage collaboration

Collaboration is essential to experimentation. When employees work together, they can share ideas and help each other to test new concepts. Encourage cooperation between your team members and promote a culture of teamwork.

7. Promote creativity

Creativity is another critical element of experimentation. Enable employees to be creative and think outside the box. If you want to foster a culture of experimentation, it’s essential to promote creativity.

8. Lead by example

Best business leaders lead by example. They show employees that you’re willing to take risks and experiment. Let them see that you’re open to new ideas and ready to change course if necessary. Leading by example will help to foster a culture of experimentation in your company.

9. Promote Brainstorming Sessions

One of the most significant ways to foster a culture of experimentation is to promote brainstorming sessions. By encouraging employees to brainstorm, you encourage them to think outside the box and develop innovative ideas . Brainstorming sessions also promote communication and collaboration, essential for any successful business.

10. Allow for autonomy

Another way to foster a culture of experimentation is to allow for autonomy. This means giving employees the freedom to experiment with new ideas and methods without getting approval from management first. This will enable employees to take ownership of their work and feel more invested in the company’s success.

11. Encourage outside-the-box thinking

To foster a culture of experimentation, it is vital to encourage outside-the-box thinking. This means that you should create an environment where employees feel comfortable proposing new ideas and solutions, even if they are not necessarily in line with the company’s current way of doing things.

By encouraging this type of thinking, you will give employees the freedom to business experiment and develop new, innovative ways to do things that can help the company grow.

12. Give employees the resources they need to experiment

For employees to experiment and come up with new ideas, they need to have the resources they need to do so. This means providing them with access to the necessary tools, equipment, and financial resources they need to experiment without worrying about whether or not their ideas will pan out. By giving employees the resources they need to experiment, you empower them to develop new solutions that can benefit the company.

13. Set aside time for experimentation

To foster a culture of experimentation, it is important to set aside time for employees to experiment. This means creating dedicated periods where employees are free to experiment with new ideas and solutions without worrying about meeting deadlines or completing tasks.

Related: Benefits of Research and Development in Business Competitive

What is another word for experimentation?

There is no one-size-fits-all answer to this question, as “experimentation” can mean different things to different people. However, some possible alternative words or phrases that could be used in place of “experimentation” include “exploration,” “investigation,” “testing,” and “trial and error.”

Each of these words or phrases captures a slightly different nuance of the concept of experimentation and can be helpful depending on the context in which they are used. Ultimately, it is up to each individual to decide which word or phrase best suits their needs and intentions when engaging in the experimental activity.

What is Strategic Experimentation?

Strategic experimentation is trying new strategies to learn about their effectiveness and improve upon them. This process usually involves some degree of trial and error, as different methods are tested and refined. Strategic experimentation aims to find the best possible strategy for achieving the desired outcome.

Many different approaches can be taken to strategic experimentation. For example, some organizations may choose to experiment with new strategies on a small scale. In contrast, others may opt for a more aggressive approach that involves trying out new strategies in a live environment.

Whatever approach is taken, it is crucial to have a clear understanding of the goals and objectives of the experiment. This will help to ensure that the right strategies are tested and that the results can be effectively analyzed and interpreted.

What are productivity and innovation?

Productivity is a measure of how efficiently resources are used to produce outputs. It is often described as a ratio of output (e.g., goods or services) to input (e.g., labor, capital, or land). Productivity growth can come from improvements in technology, changes in production organization, or increases in the quality of inputs.

Innovation is the introduction of new products, processes, or services. It can also be described as applying new knowledge to create value. It often involves creativity and risk-taking. It can be disruptive, making existing products or services obsolete. Or it can be incremental, improving upon existing products or services.

Productivity and innovation are essential drivers of economic growth. They help businesses improve their competitiveness and create jobs. They also enhance the quality of life for consumers by providing them with better products and services.

Related: How do Companies Encourage Innovation?

How does innovation improve productivity?

Innovation has always been a critical driver of productivity growth. There are many ways in which innovation can improve productivity. For example, new and improved products can help businesses be more efficient and reduce costs.

Process improvements can also increase productivity by reducing waste and increasing output quality. In addition, innovation can help businesses tap into new markets and create new growth opportunities.

In short, innovation is a critical ingredient for productivity growth. Businesses can improve their competitiveness and position themselves for long-term success by investing in innovation.

Related: Benefits and Risks of Innovation

Final Thoughts

Overall, experimentation is a vital part of innovation and growth. You can create an environment where new ideas are fostered, and innovative solutions are found by encouraging employees to experiment.

Give employees the resources they need to experiment and set aside time to do so. Be clear about the goals and objectives of any experiments conducted, and use the results to improve future strategies. Ultimately, experimentation can help your company find new ways to grow and succeed.

We hope this article helped understand what experimentation is and why it is important. If you have any questions or comments, please feel free to leave them below.

Thanks For Reading!

Frequently Asked Questions

What is a control group.

A control group is a set of subjects in an experiment who are not exposed to the experimental treatment. Control groups are used to compare the experimental group’s results with a known standard or determine if there is any treatment effect. They may be exposed to a placebo, sham treatment, or no treatment. Control groups are essential in scientific experiments to help ensure that the results are not due to chance.

What are statistical methods?

Statistical methods are mathematical formulae, models, and procedures used to analyze raw research data in statistical research. The use of statistical techniques extracts information from research data and provides several methods for assessing the validity of research findings. Statistical methods can analyze the entire customer journey, from initial contact to purchase. As a result, businesses can develop more effective marketing and sales strategies by understanding customer behavior at each journey stage. Statistical methods can also generate valuable insights into business operations and performance. No matter what type of business you run, statistical methods can be valuable for understanding your customers and improving your bottom line.

What is Circular business model experimentation?

Circular business model experimentation (CBME) is the process of trialing new ways of doing business to achieve a circular economy. It involves trying out new circular business models, seeing how they work in practice, and making changes as necessary. This helps firms learn what works and doesn’t and how to best transition their operations towards a more sustainable model.

What are ethical review boards?

An ethical review board, sometimes known as an institutional review board or research ethics board, is a committee that reviews the ethical implications of research proposals. The board is responsible for ensuring that research projects are conducted following ethical principles, such as respect for human dignity and autonomy.

Who is Stefan Thomke?

The William Barclay Harding, Professor of Business Administration at Harvard Business School, is Stefan Thomke. He earned his Ph.D. from MIT in 1995 and joined the Technology and Operations Management section at Harvard Business School that year. Thomke is best known for his publication “Experimentation Works: The Surprising Power of Business Experiments .” This publication was highlighted by well-known media outlets, including The wall street journal. Thomke argued that businesses should embrace experimentation as a critical driver of innovation and growth.

What is an independent variable in experimental research?

The independent variable is the factor that the researcher manipulates in an experiment. It is the intentionally changed variable to see its effect on the dependent variable. For example, in a study on the effects of sleep deprivation on cognitive performance, the independent variable would be the amount of sleep that participants get each night. In contrast, the dependent variable would be cognitive performance.

How To Use Product Innovation Strategy at Your Company
Top 10 Costco Competitors – [All about Costco]-2022
Southwest Airlines Competitors: The Top 10 Alternatives [In-Depth Analysis]
9 Top Etsy Competitors [Etsy Alternatives in 2022] – Complete List

Sternberg´s Triarchic Theory of Intelligence: What Is It and How Can We Use It?

4 best sales tracking software for small business, related posts.

6 Innovative Strategies for Staffing Firms to Improve Profitability

Incremental Innovation: Real-world Examples and Benefits for Businesses

Amazing Examples of Disruptive Technology in 2023

Navigate Site

Privacy and Policy
Terms and Conditions

Welcome Back!

Remember Me

Retrieve your password

Please enter your username or email address to reset your password.

Overview of Business Experimentation

Experimentation is the quantification of doubt. Where doubt is present, risk is present. Deliver more of your projects successfully, using experimentation to reduce business risk.

Introduction

Developing new products and services in large enterprises can be challenging.

Project delivery typically takes months or years, with capital investment running into many millions of dollars.

Even with inordinate amounts of planning and governance, the likelihood of projects underperforming is still very high.

This delivery model is not geared to discovery and exploration. Many small, iterative in-market tests are required to unearth what customers value, before proceeding to a mass-market launch.

Using a big-bang approach to execution carries significant risk. Product and market assumptions are only validated with customers when the new product is launched in the market.

Businesses need to approach development of new products and services differently.

Organisations require new ways of working and tools which support iterative product development, whereby new products can be discovered through fast, low-cost experimentation.

Overview of business experimentation

Business leadership is changing

Nowadays, the answers to many business problems are unknown. Customers, competitors and market conditions can be highly unpredictable.

Business leaders are required to operate in a world with insufficient data to make decisions. As such, decision-making can often rely on past experiences, intuition and legacy thinking.

The challenge with using this decision-making approach is that new developments and innovations fly in the face of existing experience and wisdom.

Boeing would never launch a new jet engine to market without conducting extensive testing and experimentation based on sound engineering and scientific principles. The stakes are too high.

Yet, many companies still launch new business models, products and features without a thorough testing and validation process with customers.

If businesses embark on a disciplined experimentation process prior to launching new ideas, organ rejection from customers can be discovered before launch, at low cost.

“Part of establishing a culture of data-driven experimentation, is having the humility of not knowing the right answer, or maybe not knowing at all”

Learning is a natural part of the entrepreneurial process.

The role of the business leader is transitioning from business manager to CEO , the Chief Experimentation Officer.

Experimentation creates more options

Through a disciplined process of discovery and exploration, experimentation helps to create more options.

By running many fast and low-cost experiments, it’s possible to adjust effort and directionality after discovering what does and, doesn’t work.

Experimentation allows businesses to try new strategies and ideas at small scale – a small audience subset, one geography, variants of ideas.

Strategies and hypotheses can then be refined on the run, rather than waiting for quarterly or annual planning cycles.

“Experimentation forces a mindset shift to think in terms of hypotheses, rather than ideas”

It’s impossible to know in advance which hypotheses will be correct. Consumer reactions to new concepts are always surprising – both positively and negatively.

Experimentation enables business leaders to predict with a higher level of confidence how their new idea will perform at scale.

Make better decisions & business investments

Experimentation helps to point you in the right direction when:

Answers aren’t obvious

People have conflicting opinions

There’s uncertainty about the value or merit of an idea

Experimentation reduces wasted effort – capital, time and resources.

Many organisations spend a considerable amount of time and resources developing solutions that generate little business impact.

An experimentation led approach helps businesses to weed out initiatives that have no merit, quickly understanding which ideas show promise.

Experimentation facilitates gathering of high-quality, relevant, real-world data from customers.

“Business leaders are able to connect the dots between strategy, risk and investment faster to make better quality decisions”

When performed in a systematic and industrialised manner, experimentation enables organisations to innovate more efficiently, build new products and services that meet the needs of their customers, and develop solutions to previously unanswered questions.

If you want to be a more effective and successful business leader, experimentation is the answer.

Experimentation is still challenging for many businesses

Many businesses still have difficulty executing experiments.

“The process of running experiments is relatively easy, however, organisational constraints can make executing experiments hard”

Existing business processes, governance frameworks and funding models are typically not geared for fast, low-cost experimentation. These operating models support predictability and certainty, rather than iterative learning approaches.

Experiments can also struggle for prioritisation when competing against other high priority business projects or customer campaigns.

To succeed in large organisations, experimentation requires dedicated funding, resourcing, processes and tools and infrastructure.

Make it easy for teams to run experiments and analyse results.

What are the challenges?

Human beings regularly suffer from optimism bias. We overestimate how good our ideas are, thinking that every opportunity will succeed.

We completely underestimate the likelihood of negative events occurring, where our ideas are actually bad.

However, failure is omnipresent. It’s around us every day, we just choose to ignore it, thinking that we’re averse from failure happening to us too. It only happens to everyone else right?

“It doesn’t matter how beautiful your theory is, it doesn’t matter how smart you are. If it doesn’t agree with the experiment, it’s wrong” – Richard Feynman

Market failure statistics are consistent, year on year:

A study by AC Nielsen suggested that up to 85% of 24,543 new consumer goods products launched to market in one year failed. The products either underperformed or failed relative to business case expectations

In the United States, 20% of small businesses fail in Year One, with a further 30% failing in Year Two. Only 50% of small businesses make it beyond Year Five

Harvard Business School lecturer Shikhar Ghosh published a Wall Street Journal article suggesting that 75% of venture-backed companies never provide a ROI to investors

At Microsoft, only 33% of experiments generate positive commercial outcomes, with two thirds yielding a neutral or negative outcome

The Startup Genome in their 2019 Global Startup Ecosystem Report suggest that only 1 in 12 entrepreneurs succeed

The odds are against us before we even start. We just choose not to acknowledge it.

What are the complicating factors?

So, the next obvious question, why do so many new product launches underperform in the market?

Delivering new products and services is thought to work as a one-way production line.

An idea is input at the beginning of the conveyor belt, transformation activities occur along the production line to “manufacture” a new offering, with a pot of money spit out the other end.

Customers and markets are unpredictable and, developing new products and services is a non-linear event. Remember the famous innovation squiggle?

One of the primary reasons that so many new ventures fail is that businesses skip the customer validation phase in the “manufacturing” process, jumping straight from “I’ve got a great idea” to “Let’s start building”.

“Failing to understand market demand, customer behaviours and preferences prior to launching a new product or service leads to massive organisational blind spots”

The longer these blind spots persist, the greater the inherent business risk in the project delivery process.

Ultimately, if the blind spots and assumptions remain unchecked, things can end up terminal … you realise that customers aren’t interested in your offer and you’ve wasted significant time, capital and resources building something that customers don’t want.

When is it best to run experiments?

There are two types of questions in business, those that you already know the answer, and those that you don’t know the answer.

For the questions that you already know the answer (system upgrades, continuous improvement, regulatory, compliance) you should work to understand cost-benefit and risk, prepare a business case, and get moving.

You’re not asking, “should we do it”, you just need to do it. Business experimentation is not as beneficial in these types of scenarios.

For business questions where you don’t know the answer – the “should we do it” or “how do we do it” questions – experimentation is a powerful way gather data to inform better quality decision making.

The latter scenario represents the majority of circumstances in the modern business.

What is experimentation?

Experimentation is underpinned by the scientific method, being more science than art.

It’s an iterative learning approach, shifting between inductive thinking (theory, guess) and deductive thinking (data, facts) to test business hypotheses.

“Experimentation is the quantification of doubt. Where doubt is present, risk is present”

It’s a continuous cycle of learning and exploration – hypothesise, experiment, learnings, repeat.

In the same way that the scientific method helped humanity accelerate many important developments, a scientific method of experimentation can also help business to solve complex problems much faster.

Experimentation - hypothesise, experiment, learn, repeat

An experiment represents an individual learning event with customers. It’s an opportunity to learn something new, something that you previously didn’t know.

Experiments are highly targeted, short duration, fast and low-cost. Each experiment is designed to test underlying product and market assumptions.

“If assumptions are left untested prior to the launch of a new product, service or experience, they present significant risk in the project delivery process”

Experimentation helps you to shift the odds more in your favour by testing and developing new initiatives in-market with customer prior to scaled launch.

Experimentation is all about being able to learn faster and make decisions faster.

If you can outlearn your competitors, you have a natural competitive advantage.

Experimentation example - AirBnB rejects social proof statements

In this experimentation example, AirBnB was seeking to understand if adding a form of social proof on their host signup landing page would encourage hosts to list their property.

The experiment consisted of a Control (Experiment A) without a social proof statement, and two Variants (Experiment B) and (Experiment C), containing different social proof statements.

Screen Shot 2021-04-16 at 2.21.26 pm.png

Without engaging with the customers (hosts) that participated in this experiment to conduct additional research to understand user behaviours and decision-making drivers, it’s difficult to understand why Experiment B and Experiment C performed negatively without speculating.

For the purposes of the example, we’ll make some inferences:

Experiment A - Control

Experiment A is the control. There is no social proof statement.

Experiment B - Message 1 - Toronto is popular!

In Experiment B, this social proof statement suggests that, “39% of the listings in the local area of the host are fully booked next week”. This social proof statement is trying to influence hosts to list their property, given a level of quantifiable customer demand. The only problem is … 39% is not a very big number. For example, if 84% of listings in the local area were fully booked next week, it’s a much more compelling proposition for hosts, and maybe the experiment would have performed differently.

Experiment C - Message 2 - Host the world in Toronto! For Experiment C, the social proof statement suggests that, “travellers from 180 countries searched for places to stay in your area last week”. Given that there are only 249 countries in the world, this social proof statement could give hosts the impression that the whole world is descending on their doorstep, raising a raft of other objections or fears?

The results Experiment A (Control) was implemented. Experiment B and Experiment C were rejected after producing a negative outcome relative to the control experiment.

How do you conduct an experiment?

The process of experimentation involves the following key steps:

Developing a hypothesis

Designing an experiment

Executing the experiment

Analysing the results

Let’s discuss each of these steps in a little more detail.

Step 1 - Developing a hypothesis

A hypothesis states your predictions about what your research will find. It is a tentative answer to your research question that has not yet been tested.

For some research projects, you might have to write several hypotheses that address different aspects of your research question.

“A hypothesis is not just a guess — it should be based on existing theories and knowledge”

It has to be specific, testable and falsifiable, which means you can support or refute it through scientific research methods (experiments, observations and statistical analysis of data).

Your hypothesis must be able to be proven wrong.

Poor hypothesis:

Lightning is caused by angry ghosts.

Good hypothesis:

Lightning is caused by electrical charges moving from the earth to the clouds.

Don’t forget to set the key success metrics for your experiment before you launch, otherwise every experiment is a winner.

Step 2 - Designing an experiment

Good experimental design consists of an experimental group, and a control group to be able to compare the impact of changing a variable.

Experiments generally only change one variable at a time to isolate the impact. Otherwise, if you’re changing multiple variables simultaneously it can be difficult to pinpoint what’s caused any change.

When designing your experiment, you need to determine which variable you are going to change and the expected outcome from the experiment (I.e. increased conversion rate)

Split Testing (A/B Testing) is one of the most popular experimentation methodologies. For this method, you take two versions that are identical except for one variable (image, headline, discount) and compare their performance (click through rate, conversion rate).

Other factors to consider are the duration of the experiment, and your path to customer (E.g. Paid Advertising, EDM, Website/Landing Page etc.) and customer sample size.

Screen Shot 2021-04-14 at 5.05.20 pm.png

Netflix has been iterating on showing additional fields upfront on their homepage. Previous experiments had demonstrated that displaying an email address field upfront increased customer conversion rates.

This experiment is the next step (Experiment B), to understand what impact adding in the password field upfront has on customer conversion rates.

Step 3 - Executing the experiment The key to executing a good experiment is ensuring that variance is limited.

To have a valid experiment, the conditions between the experimental group and the control group need to be pretty much identical, except for the change in the tested variable.

Communication of your experiment is critical. You never want to announce to your target customers that your feature, product or offer is in fact an experiment.

That would immediately invalidate the experiment, and your results, as it would produce a change in consumer behaviour.

“It’s better to operate with intuition than incorrect data”

Experiment artefacts should always have the appearance of being real and available, without deliberately tricking customers.

Remember, you’re always experimenting with your customers, not on your customers.

Step 4 - Executing the experiment

Once the experiment has been concluded, it’s now time to compare the results of your experimental group against the results of the control group.

A degree in statistics is not required to do this.

You’re more looking to compare trends over time and the relative performance of different variables against one another.

It’s really important to try and analyse experimentation results as objectively as possible. We all bring our own beliefs, biases, opinions and assumptions to the table.

Be careful to not let decision-making be clouded by “group think”.

It’s very easy to read meaning into the data that isn’t there, or to create a counter narrative to the data that ensures an idea continues to be resuscitated, when it should be taken off life support.

Screen Shot 2021-04-15 at 10.51.11 am.png

The result of this experiment suggests a negative outcome, as Netflix reverted back to the Control version (Experiment A) - without the visible password field.

What are the benefits of experimentation?

Experimentation has the following key benefits:

Reduce business risk

Decrease time

Better decisions

More options

Reduce business risk:

Experimentation decreases business risk by enabling business leaders and project teams to understand how an offer performs in-market at small-scale, prior to mass-market launch.

Experimentation provides strong indicators of customer demand and interest earlier in the development process. If experimentation results indicate strong signals of customer demand, it gives teams the confidence to keep incrementally investing.

Experimentation decreases the likelihood of False Negatives (you should’ve built it, but you didn’t) and False Positives (you build it, but you shouldn’t have).

Save costs:

Save capital by not developing new products or services that underperform in-market or provide little business impact.

Weeding out ideas that have no customer value or business impact early in the development process produces quantifiable CAPEX savings that can then be redistributed to pursue other business opportunities.

Decrease time:

Experimentation enables you to test and validate your product and market assumptions with real customers in terms of hours and days, not months and years.

As such, this allows businesses to accelerate to market faster, while ensuring that resources are prioritised to initiatives that have demonstrated business impact and customer value.

Better decisions:

Experimentation elevates the role of customer data and voice of customer in business decision-making, shifting the conversation from opinions and beliefs, to facts and data. This produces higher quality decisions, resulting in better business investments.

More options:

Good strategy is about creating options. Experimentation facilitates the generation of many options and possibilities, allowing business leaders to find the optimal path to reach their goals.

With experimentation being fast and low-cost, it enables businesses to pursue many more growth and expansion opportunities.

Who should be using experimentation?

Data-driven experimentation isn’t just for large, platform businesses like Amazon, Google and Netflix.

Experimentation is for all companies, whether you’re an early-stage startup, or a large, megacorp.

When is it best to use experimentation? From my perspective, all the time, at any opportunity.

Common applications of experimentation include (non exhaustive):

- New product development / growth opportunities

- Product line extensions / tactical offers

- New product features

- Impacts of pricing on customer demand

- New customer experiences

- Membership / Loyalty / Referral programs

- Customer retention and engagement

- Impacts of website design on customer conversion rates

- Impacts of Marketing campaigns / messaging on customer conversion rates

- etc. etc.

There is a myriad of opportunities. Let your mind run wild with possibility.

Organisations need to approach the development of new products and services differently.

Traditional big-bang models of project delivery carry significant risk. Product and market assumptions are only validated with customers when the new product is launched in the market.

Year on year, market failure statistics demonstrate that the majority of new consumer products will underperform in the market, failing to meet business expectations.

Experimentation helps business leaders to connect the dots between strategy, risk and business investment faster.

Good strategy is about creating options. Experimentation facilitates the generation of many options, allowing business leaders to find the optimal path to reach their goals.

Project teams can understand very early in the delivery process, which ideas deliver value for customers, and which don’t. Business resources can then be allocated to initiatives that show the most promise and deliver the greatest business impact.

Reduce business risk, save costs and decrease time with experimentation.

If you want to deliver more projects successfully, experimentation is the answer.

References:

Startup Genome , Wall Street Journal , Entrepreneur.com , Joe Newsum , Harvard Business Review , Harvard Business Review ,

Before you finish...

Did you find this article helpful? Before you go, it would be great if you could help us by completing the actions below.

By joining the First Principles community, you’ll never miss an article.

Help make our longform articles stronger. It only takes a minute to complete the NPS survey for this article. Your feedback helps me make each article better.

Or share this article by simply copy and pasting the URL from your browser.

Be Hypothesis Driven, Not Idea Led

Strengthen your strategy with experimentation.

All Subjects

In Intro to Business

An experiment in the context of market segmentation is a methodical process businesses use to test different marketing strategies or product features among specific segments of their market to determine what best meets the customers' needs. It involves creating a controlled environment to observe, measure, and analyze the effects of variations on consumer behavior or preferences.

congrats on reading the definition of experiment . now let's actually learn it.

Related terms

Market Segmentation : The process of dividing a broad consumer or business market into sub-groups of consumers based on some type of shared characteristics.

Target Market : A specific group of consumers identified as the recipients of a particular marketing message, who are likely to purchase a company's products or services.

Consumer Behavior : The study of individuals, groups, or organizations and the processes they use to select, secure, use, and dispose of products, services, experiences, or ideas to satisfy needs and the impacts that these processes have on the consumer and society

" Experiment " also found in:

Subjects ( 24 ).

AP Psychology
AP Statistics
AP US Government
Algebra and Trigonometry
College Algebra
College Introductory Statistics
College Physics: Mechanics, Sound, Oscillations, and Waves
Contemporary Mathematics for Non-Math Majors
Epidemiology
Foundations of Lower Division Mathematics
History of Science
Honors Biology
Honors Physics
Honors Statistics
Intro to Business Statistics
Intro to Sociology
Introduction to Epidemiology
Introduction to Political Research
Introduction to Probability
Mathematical Probability Theory
Music Psychology
Physical Science
Science Education
The Modern Period

Ap® and sat® are trademarks registered by the college board, which is not affiliated with, and does not endorse this website..

More from M-W
To save this word, you'll need to log in. Log In

Definition of experiment

(Entry 1 of 2)

Definition of experiment (Entry 2 of 2)

intransitive verb

experimentation

Examples of experiment in a Sentence

These examples are programmatically compiled from various online sources to illustrate current usage of the word 'experiment.' Any opinions expressed in the examples do not represent those of Merriam-Webster or its editors. Send us feedback about these examples.

Word History

Middle English, "testing, proof, remedy," borrowed from Anglo-French esperiment, borrowed from Latin experīmentum "testing, experience, proof," from experīrī "to put to the test, attempt, have experience of, undergo" + -mentum -ment — more at experience entry 1

verbal derivative of experiment entry 1

14th century, in the meaning defined at sense 1a

1787, in the meaning defined above

Phrases Containing experiment

control experiment
controlled experiment
experiment station
pre - experiment
thought experiment

Articles Related to experiment

This is the Difference Between a...

This is the Difference Between a Hypothesis and a Theory

In scientific reasoning, they're two completely different things

Dictionary Entries Near experiment

experiential time

experimental

Cite this Entry

“Experiment.” Merriam-Webster.com Dictionary , Merriam-Webster, https://www.merriam-webster.com/dictionary/experiment. Accessed 11 Sep. 2024.

Kids Definition

Kids definition of experiment.

Kids Definition of experiment (Entry 2 of 2)

Medical Definition

Medical definition of experiment.

Medical Definition of experiment (Entry 2 of 2)

More from Merriam-Webster on experiment

Nglish: Translation of experiment for Spanish Speakers

Britannica English: Translation of experiment for Arabic Speakers

Subscribe to America's largest dictionary and get thousands more definitions and advanced search—ad free!

Play Quordle: Guess all four words in a limited number of tries. Each of your guesses must be a real 5-letter word.

Can you solve 4 words at once?

Word of the day, chockablock.

See Definitions and Examples »

Get Word of the Day daily email!

Popular in Grammar & Usage

Plural and possessive names: a guide, 31 useful rhetorical devices, more commonly misspelled words, absent letters that are heard anyway, how to use accents and diacritical marks, popular in wordplay, 8 words for lesser-known musical instruments, it's a scorcher words for the summer heat, 7 shakespearean insults to make life more interesting, 10 words from taylor swift songs (merriam's version), 9 superb owl words, games & quizzes.

Play Blossom: Solve today's spelling word game by finding as many words as you can using just 7 letters. Longer words score more points.

SUGGESTED TOPICS
The Magazine
Newsletters
Managing Yourself
Managing Teams
Work-life Balance
The Big Idea
Data & Visuals
Reading Lists
Case Selections
HBR Learning
Topic Feeds
Account Settings
Email Preferences

Building a Culture of Experimentation

Stefan Thomke

Online experiments can be a game changer when it comes to marketing and innovation. By running some 25,000 tests a year, for instance, Booking.com has transformed itself from a small start-up to the world’s largest accommodation platform. Today scaling up an organization’s experimentation capabilities is critical, but many firms struggle to do it—not because of technology but because of culture.

To break down cultural barriers, companies need to create an environment where curiosity is nurtured, data trumps opinions, any employee can launch tests, all experiments are ethical, and a new more-democratic model of leadership prevails. Ultimately, executives have to be able to confront the possibility that they are wrong daily and willing to give their people greater autonomy.

It takes more than good tools. It takes a complete change of attitude.

Idea in Brief

The opportunity.

In an increasingly digital world, randomized, controlled A/B experiments are an extremely valuable way to create or improve online experiences.

The Obstacle

Culture—not tools and technology—prevents companies from conducting the hundreds, even thousands, of tests they should be doing annually and then applying the results.

Create an environment in which curiosity is nurtured, data trumps opinion, anyone can conduct a test, all experiments are done ethically, and managers embrace a new model of leadership.

In December 2017, just before the busy holiday travel season, Booking.com’s director of design proposed a radical experiment: testing an entirely new layout for the company’s home page. Instead of offering lots of options for hotels, vacation rentals, and travel deals, as the existing home page did, the new one would just feature a small window asking where the customer was going, the dates, and the number of people in the party, and present three simple options: “accommodations,” “flights,” and “rental cars.” All the content and design elements—pictures, text, buttons, and messages—that Booking.com had spent years optimizing would be eliminated.

Make sure your experiments recognize customers’ varying needs.

Stefan Thomke is the William Barclay Harding Professor of Business Administration at Harvard Business School. He is a leading authority on the management of business experimentation and innovation and has worked with many global companies on product, process, and technology development. He is the author of Experimentation Works: The Surprising Power of Business Experiments (HBR Press, 2020).

Partner Center

Cambridge Dictionary +Plus

Meaning of experiment in English

Your browser doesn't support HTML5 audio

Don't forget to try out the equipment before setting up the experiment.
In the second experiment they obtained a very clear result .
For the experiment to be valid , it is essential to record the data accurately .
The experiments were conducted by scientists in New York.
Our experiment worked better than we could have expected , and soon the baby was happy to sleep in her own bed .
as an experiment
background check
experimental
experimentally
experimentation
experimenter
put someone/something through their/its paces idiom
put something to the test idiom
reinspection
try something out
uncheckable
welfare check
The young film-makers were given free rein to experiment with new themes and techniques .
I like to experiment with different light filters on my camera .
For a while the poet experimented with the idea of chanting his poems to music .
The artist experimented with different pigments and mediums , often with disastrous results .
I'd be very nervous about letting a trainee hairdresser experiment with my hair .

Related word

Experiment | intermediate english, examples of experiment, collocations with experiment.

These are words often used in combination with experiment .

Click on a collocation to see more examples of it.

Translations of experiment

Get a quick, free translation!

Word of the Day

to put your arms around someone and hold them in a loving way, or (of two people) to hold each other close to show love or for comfort

Treasure troves and endless supplies (Words and phrases meaning ‘source’)

Learn more with +Plus

Recent and Recommended {{#preferredDictionaries}} {{name}} {{/preferredDictionaries}}
Definitions Clear explanations of natural written and spoken English English Learner’s Dictionary Essential British English Essential American English
Grammar and thesaurus Usage explanations of natural written and spoken English Grammar Thesaurus
Pronunciation British and American pronunciations with audio English Pronunciation
English–Chinese (Simplified) Chinese (Simplified)–English
English–Chinese (Traditional) Chinese (Traditional)–English
English–Dutch Dutch–English
English–French French–English
English–German German–English
English–Indonesian Indonesian–English
English–Italian Italian–English
English–Japanese Japanese–English
English–Norwegian Norwegian–English
English–Polish Polish–English
English–Portuguese Portuguese–English
English–Spanish Spanish–English
English–Swedish Swedish–English
Dictionary +Plus Word Lists
English Noun Verb
Collocations
Translations
All translations

To add experiment to a word list please sign up or log in.

Add experiment to one of your lists below, or create a new one.

Something went wrong.

There was a problem sending your report.

IMAGES

What is Experimental Research & How is it Significant for Your Business
Business Experiments: Steps and Examples
What is Experimental Research & How is it Significant for Your Business
Tips for Retail Business Experimentation
What is Experimental Research & How is it Significant for Your Business
The Power of Fast Experimentation for Product Managers

VIDEO

Testing Business Ideas: 8 Steps to Avoid Experiment Traps Webinar
An experiment in experiments
soldering iron banaaiye| Mini Experiment Sai🙄 💯🔥
Experiment
The Experiment Full Movie Facts & Review in English / Adrien Brody / Forest Whitaker
The Experiment (2010) Explained in English with Subtitles

COMMENTS

A Step-by-Step Guide to Smart Business Experiments
Specifically, they need to embrace the "test and learn" approach: Take one action with one group of customers, a different action (or no action at all) with a control group of customers, and ...
Business Experiments
Business experiments help you to test your ideas and gather information before you commit significant resources to a large project. They can be anything from very basic tests to complex projects involving prototype products or services. To conduct an effective business experiment, do the following: Create a hypothesis.
Business Experiments: Steps and Examples
Step 4: Completion. After the test has been performed within the determined period at the determined location, the data obtained is analysed in order to determine the results. These results are ideally stored in a company library, and may lead to a broader roll-out of the experiment, or testing a revised hypothesis.
The Discipline of Business Experimentation
The Discipline of Business Experimentation. Increase your chances of success with innovation test-drives. Summary. The data you already have can't tell you how customers will react to ...
How to Design (and Analyze) a Business Experiment
How to Design (and Analyze) a Business Experiment. by. Oliver Hauser. and. Michael Luca. October 29, 2015. Jimmy Musto. The rise of experimental evaluations within organizations — or what ...
Experimentation the McKinsey Way
The scientific method is alive and well in business. Experiments are structured tests to verify a hypothesis or idea and create insight into potential cause and effect. Experimentation is used a considerable amount in marketing, services, and retail to understand such things as: • The impact of advertising on sales • How different messaging, promotions and creative in advertisements or ...
The Basics of Experimentation and Why It's Key to Your ...
Experimental: Experiments help you validate business hypotheses as a new change idea will not have essential historical data to validate the change. One could conduct experiments to observe user ...
Experimentation Works: The Surprising Power of Business Experiments
See how the power of experiments works for you. When it comes to improving customer experiences, trying out new business models, or developing new products, even the most experienced managers often get it wrong. They discover that intuition, experience and big data alone don't work. What does work? Running disciplined business experiments.
Experimenting: What Makes a Good Business Experiment?
Abstract. Running good business experiments has become a competitive advantage but many organizations have considerable difficulty executing them. Although the process of experimentation seems straightforward, it is surprisingly hard in practice, owing to myriad organizational, management, and technical challenges.
The power of experimentation in business
Thinking future and innovation: the power of experimentation in business. In the world of science, experiments are the norm. It's a reliable way to see if your ideas work and to check if an innovation is a true breakthrough or something that works well only on paper. Nevertheless, it is always a vision that comes first, then a string of ...
Fostering Business Innovation With Experimentation
Innovation is the key to the evolution of business and requires an open mind and willingness to experiment. While testing new business ideas can seem intimidating, you can mitigate the risk by ...
Harvard Business Professor on Companies Using ...
In the business world, experiments have led to the discovery of both technical solutions and new markets. A classic example of both is the discovery of 3M's Post-it Note. The story begins in 1964 ...
What is Experimental Research & How is it Significant for Your Business
Experimental research is a kind of study that rigidly follows a scientific research design. It involves testing or attempting to prove a hypothesis by way of experimentation. As such, it uses one or more independent variables, manipulating them and then using them on one or more dependent variables.
Business experimentation lessons from the top-tier
Elevate your impact, reignite your ambition and challenge your thinking with a programme designed to take highly accomplished senior executives to the next level. Select up to 4 programmes to compare. Select one more to compare. How to keep experimenting so you and your organisation can continue to find new opportunities rather than drifting ...
How to Design Smart Business Experiments
How to Design Smart Business Experiments. Managers now have the tools to conduct small-scale tests and gain real insight. But too many "experiments" don't prove much of anything. Summary ...
What is Experimentation, and Why Is It Crucial for Businesses Today?
Experimentation will enable businesses to find new ways of doing things and stay ahead of the competition. There are many benefits of experimentation, but some of the most important ones are listed below. 1. Leads to discoveries and innovations. Often, experimentation is the key to making discoveries and innovations.
Overview of Business Experimentation
Experimentation enables business leaders to predict with a higher level of confidence how their new idea will perform at scale. Experimentation helps to point you in the right direction when: There's uncertainty about the value or merit of an idea. Experimentation reduces wasted effort - capital, time and resources.
Experiment
An experiment in the context of market segmentation is a methodical process businesses use to test different marketing strategies or product features among specific segments of their market to determine what best meets the customers' needs. It involves creating a controlled environment to observe, measure, and analyze the effects of variations on consumer behavior or preferences.
Why Business Schools Need to Teach Experimentation
Experiments are difficult to do well. Some challenges include special statistical knowledge, clear problem definition, and interpretation of the results. And it's not enough to have the skillset. Experiments should ideally be done iteratively, building on prior knowledge and working toward deeper understanding of the question at hand.
Experiment Definition & Meaning
The meaning of EXPERIMENT is test, trial. How to use experiment in a sentence.
Building a Culture of Experimentation
Online experiments can be a game changer when it comes to marketing and innovation. By running some 25,000 tests a year, for instance, Booking.com has transformed itself from a small start-up to ...
EXPERIMENT
EXPERIMENT definition: 1. a test done in order to learn something or to discover if something works or is true: 2. to try…. Learn more.
EXPERIMENT
EXPERIMENT meaning: 1. a test done in order to learn something or to discover if something works or is true: 2. to try…. Learn more.

Business Experiments: Steps and Examples

What are Business Experiments?

Different Forms of Business Experiments

Prototype Tests

Concept Testing

Thought Experiments

Doing Business Experiments Yourself (Step-by-step Plan)

Step 1: Formulating a Hypothesis

Step 2: Designing the Experiment

Step 3: Data Analysis

Step 4: Completion

Step 5: Re-testing

Business experiments: what is a Test & Learn Approach?

Part of the Company Strategy

Practical Examples of Business Experiments

Business Experiments summary

Now it’s your turn

More information

Related ARTICLES

Regression Analysis explained

B2B Marketing: the Basics, Types and Examples

Product Management explained

Buy Sell Hierarchy (Miller and Heiman)

B2C Marketing: the Definition and Strategies

STP Model in Marketing Explained: Steps & Repositioning Example

Decision Tree Analysis: the Theory and an Example

What If Analysis: the Basics and an Example

Six Thinking Hats Technique explained

BOOST YOUR SKILLS

POPULAR TOPICS

ABOUT TOOLSHERO

EXPERIMENTATION

What is experimentation?

Testing a Hypothesis with Experiment vs. Control Group

Why is experimentation important?

How do you conduct experiments?

Building Hypotheses

Designing an Experiment

Executing an Experiment

Understanding Results

A/B Testing

Pre/Post Experiments

DOWNLOAD STRATEGY PRESENTATION TEMPLATES

JOE NEWSUM COACHING

THE LEADERSHIP MATURITY MODEL

EXPLORE THE TOP 100 STRATEGIC LEADERSHIP COMPETENCIES

The Basics of Experimentation and Why It's Key to Your Startup's Growth In this three-part series, we will discuss experimentation in detail. This article focuses on the "why" and "what" of experimentation.

What is experimentation?

Why experiment?

What makes experiments different from analytics?

Editor's Pick Red Arrow

Most Popular Red Arrow

How a History Student Turned Her Side Hustle Into a Startup That's Raised $7 Million: "People Always Tell Me, 'I Thought I Had to Major In Computer Science to Be an Entrepreneur'"

The 7-Step ChatGPT Formula for Peak Productivity and Profit

This NFL Team Is Owned By the Fans — Here's How Much Shares Cost and What Ownership Gets You

Taco Bell's New Mountain Dew Baja Blast Gelato Is Causing a Frenzy — But Fans Have One Big Complaint

5 Strategies That Helped Me Achieve 10x Returns on My Marketing Efforts

Experimentation Works: The Surprising Power of Business Experiments

Supplemental Information

About The Author

Stefan H. Thomke

Capture New Value from Your Existing Tech Infrastructure

Act Like a Scientist: Great Leaders Challenge Assumptions, Run Experiments, and Follow the Evidence

Experimenting: What Makes a Good Business Experiment?

Cite this chapter

Access this chapter

Similar content being viewed by others

The Use of Experiments in Business Research

Design of Experiments: A Key to Successful Innovation

Sorts and Types of Experiment

Author information

Corresponding author

Editor information

Rights and permissions

Copyright information

About this chapter

Download citation

Share this chapter

Thinking future and innovation: the power of experimentation in business

Generic talks