hypothesis python vs pytest

Pytest with Eric

How to Use Hypothesis and Pytest for Robust Property-Based Testing in Python

There will always be cases you didn’t consider, making this an ongoing maintenance job. Unit testing solves only some of these issues.

Example-Based Testing vs Property-Based Testing

Project set up, getting started, prerequisites.

Simple Example

Source code.

Simple Example — Unit Tests

Example-based testing.

Running The Unit Test

Property-based testing.

Complex Example

Source code.

Complex Example — Unit Tests

Discover Bugs With Hypothesis

Define your own hypothesis strategies.

Model-Based Testing in Hypothesis

Additional reading.

Accelerate your workflow with Semaphore – fast, reliable, and scalable for your team's success.

• Talk to a developer
• Start building for free
• System Status
• Semaphore Cloud
• Semaphore On-Premise
• Semaphore Hybrid
• Premium support
• Docker & Kubernetes
• vs GitHub Actions
• vs Travis CI
• vs Bitbucket
• CI/CD Learning Tool
• Write with us
• Get started

Getting Started With Property-Based Testing in Python With Hypothesis and Pytest

This tutorial will be your gentle guide to property-based testing. Property-based testing is a testing philosophy; a way of approaching testing, much like unit testing is a testing philosophy in which we write tests that verify individual components of your code.

By going through this tutorial, you will:

• learn what property-based testing is;
• understand the key benefits of using property-based testing;
• see how to create property-based tests with Hypothesis;
• attempt a small challenge to understand how to write good property-based tests; and
• Explore several situations in which you can use property-based testing with zero overhead.

What is Property-Based Testing?

In the most common types of testing, you write a test by running your code and then checking if the result you got matches the reference result you expected. This is in contrast with property-based testing , where you write tests that check that the results satisfy certain properties . This shift in perspective makes property-based testing (with Hypothesis) a great tool for a variety of scenarios, like fuzzing or testing roundtripping.

In this tutorial, we will be learning about the concepts behind property-based testing, and then we will put those concepts to practice. In order to do that, we will use three tools: Python, pytest, and Hypothesis.

• Python will be the programming language in which we will write both our functions that need testing and our tests.
• pytest will be the testing framework.
• Hypothesis will be the framework that will enable property-based testing.

Both Python and pytest are simple enough that, even if you are not a Python programmer or a pytest user, you should be able to follow along and get benefits from learning about property-based testing.

Setting up your environment to follow along

If you want to follow along with this tutorial and run the snippets of code and the tests yourself – which is highly recommendable – here is how you set up your environment.

Installing Python and pip

Start by making sure you have a recent version of Python installed. Head to the Python downloads page and grab the most recent version for yourself. Then, make sure your Python installation also has pip installed. [ pip ] is the package installer for Python and you can check if you have it on your machine by running the following command:

(This assumes python is the command to run Python on your machine.) If pip is not installed, follow their installation instructions .

Installing pytest and Hypothesis

pytest, the Python testing framework, and Hypothesis, the property-based testing framework, are easy to install after you have pip. All you have to do is run this command:

This tells pip to install pytest and Hypothesis and additionally it tells pip to update to newer versions if any of the packages are already installed.

To make sure pytest has been properly installed, you can run the following command:

The output on your machine may show a different version, depending on the exact version of pytest you have installed.

To ensure Hypothesis has been installed correctly, you have to open your Python REPL by running the following:

and then, within the REPL, type import hypothesis . If Hypothesis was properly installed, it should look like nothing happened. Immediately after, you can check for the version you have installed with hypothesis.__version__ . Thus, your REPL session would look something like this:

Your first property-based test

In this section, we will write our very first property-based test for a small function. This will show how to write basic tests with Hypothesis.

The function to test

Suppose we implemented a function gcd(n, m) that computes the greatest common divisor of two integers. (The greatest common divisor of n and m is the largest integer d that divides evenly into n and m .) What’s more, suppose that our implementation handles positive and negative integers. Here is what this implementation could look like:

If you save that into a file, say gcd.py , and then run it with:

you will enter an interactive REPL with your function already defined. This allows you to play with it a bit:

Now that the function is running and looks about right, we will test it with Hypothesis.

The property test

A property-based test isn’t wildly different from a standard (pytest) test, but there are some key differences. For example, instead of writing inputs to the function gcd , we let Hypothesis generate arbitrary inputs. Then, instead of hardcoding the expected outputs, we write assertions that ensure that the solution satisfies the properties that it should satisfy.

Thus, to write a property-based test, you need to determine the properties that your answer should satisfy.

Thankfully for us, we already know the properties that the result of gcd must satisfy:

“[…] the greatest common divisor (GCD) of two or more integers […] is the largest positive integer that divides each of the integers.”

So, from that Wikipedia quote, we know that if d is the result of gcd(n, m) , then:

• d is positive;
• d divides n ;
• d divides m ; and
• no other number larger than d divides both n and m .

To turn these properties into a test, we start by writing the signature of a test_ function that accepts the same inputs as the function gcd :

(The prefix test_ is not significant for Hypothesis. We are using Hypothesis with pytest and pytest looks for functions that start with test_ , so that is why our function is called test_gcd .)

The arguments n and m , which are also the arguments of gcd , will be filled in by Hypothesis. For now, we will just assume that they are available.

If n and m are arguments that are available and for which we want to test the function gcd , we have to start by calling gcd with n and m and then saving the result. It is after calling gcd with the supplied arguments and getting the answer that we get to test the answer against the four properties listed above.

Taking the four properties into account, our test function could look like this:

Go ahead and put this test function next to the function gcd in the file gcd.py . Typically, tests live in a different file from the code being tested but this is such a small example that we can have everything in the same file.

Plugging in Hypothesis

We have written the test function but we still haven’t used Hypothesis to power the test. Let’s go ahead and use Hypothesis’ magic to generate a bunch of arguments n and m for our function gcd. In order to do that, we need to figure out what are all the legal inputs that our function gcd should handle.

For our function gcd , the valid inputs are all integers, so we need to tell Hypothesis to generate integers and feed them into test_gcd . To do that, we need to import a couple of things:

given is what we will use to tell Hypothesis that a test function needs to be given data. The submodule strategies is the module that contains lots of tools that know how to generate data.

With these two imports, we can annotate our test:

You can read the decorator @given(st.integers(), st.integers()) as “the test function needs to be given one integer, and then another integer”. To run the test, you can just use pytest :

(Note: depending on your operating system and the way you have things configured, pytest may not end up in your path, and the command pytest gcd.py may not work. If that is the case for you, you can use the command python -m pytest gcd.py instead.)

As soon as you do so, Hypothesis will scream an error message at you, saying that you got a ZeroDivisionError . Let us try to understand what Hypothesis is telling us by looking at the bottom of the output of running the tests:

This shows that the tests failed with a ZeroDivisionError , and the line that reads “Falsifying example: …” contains information about the test case that blew our test up. In our case, this was n = 0 and m = 0 . So, Hypothesis is telling us that when the arguments are both zero, our function fails because it raises a ZeroDivisionError .

The problem lies in the usage of the modulo operator % , which does not accept a right argument of zero. The right argument of % is zero if n is zero, in which case the result should be m . Adding an if statement is a possible fix for this:

However, Hypothesis still won’t be happy. If you run your test again, with pytest gcd.py , you get this output:

This time, the issue is with the very first property that should be satisfied. We can know this because Hypothesis tells us which assertion failed while also telling us which arguments led to that failure. In fact, if we look further up the output, this is what we see:

This time, the issue isn’t really our fault. The greatest common divisor is not defined when both arguments are zero, so it is ok for our function to not know how to handle this case. Thankfully, Hypothesis lets us customise the strategies used to generate arguments. In particular, we can say that we only want to generate integers between a minimum and a maximum value.

The code below changes the test so that it only runs with integers between 1 and 100 for the first argument ( n ) and between -500 and 500 for the second argument ( m ):

That is it! This was your very first property-based test.

Why bother with Property-Based Testing?

To write good property-based tests you need to analyse your problem carefully to be able to write down all the properties that are relevant. This may look quite cumbersome. However, using a tool like Hypothesis has very practical benefits:

• Hypothesis can generate dozens or hundreds of tests for you, while you would typically only write a couple of them;
• tests you write by hand will typically only cover the edge cases you have already thought of, whereas Hypothesis will not have that bias; and
• thinking about your solution to figure out its properties can give you deeper insights into the problem, leading to even better solutions.

These are just some of the advantages of using property-based testing.

Using Hypothesis for free

There are some scenarios in which you can use property-based testing essentially for free (that is, without needing to spend your precious brain power), because you don’t even need to think about properties. Let’s look at two such scenarios.

Testing Roundtripping

Hypothesis is a great tool to test roundtripping. For example, the built-in functions int and str in Python should roundtrip. That is, if x is an integer, then int(str(x)) should still be x . In other words, converting x to a string and then to an integer again should not change its value.

We can write a simple property-based test for this, leveraging the fact that Hypothesis generates dozens of tests for us. Save this in a Python file:

Now, run this file with pytest. Your test should pass!

Did you notice that, in our gcd example above, the very first time we ran Hypothesis we got a ZeroDivisionError ? The test failed, not because of an assert, but simply because our function crashed.

Hypothesis can be used for tests like this. You do not need to write a single property because you are just using Hypothesis to see if your function can deal with different inputs. Of course, even a buggy function can pass a fuzzing test like this, but this helps catch some types of bugs in your code.

Comparing against a gold standard

Sometimes, you want to test a function f that computes something that could be computed by some other function f_alternative . You know this other function is correct (that is why you call it a “gold standard”), but you cannot use it in production because it is very slow, or it consumes a lot of resources, or for some other combination of reasons.

Provided it is ok to use the function f_alternative in a testing environment, a suitable test would be something like the following:

When possible, this type of test is very powerful because it directly tests if your solution is correct for a series of different arguments.

For example, if you refactored an old piece of code, perhaps to simplify its logic or to make it more performant, Hypothesis will give you confidence that your new function will work as it should.

The importance of property completeness

In this section you will learn about the importance of being thorough when listing the properties that are relevant. To illustrate the point, we will reason about property-based tests for a function called my_sort , which is your implementation of a sorting function that accepts lists of integers.

The results are sorted

When thinking about the properties that the result of my_sort satisfies, you come up with the obvious thing: the result of my_sort must be sorted.

So, you set out to assert this property is satisfied:

Now, the only thing missing is the appropriate strategy to generate lists of integers. Thankfully, Hypothesis knows a strategy to generate lists, which is called lists . All you need to do is give it a strategy that generates the elements of the list.

Now that the test has been written, here is a challenge. Copy this code into a file called my_sort.py . Between the import and the test, define a function my_sort that is wrong (that is, write a function that does not sort lists of integers) and yet passes the test if you run it with pytest my_sort.py . (Keep reading when you are ready for spoilers.)

Notice that the only property that we are testing is “all elements of the result are sorted”, so we can return whatever result we want , as long as it is sorted. Here is my fake implementation of my_sort :

This passes our property test and yet is clearly wrong because we always return an empty list. So, are we missing a property? Perhaps.

The lengths are the same

We can try to add another obvious property, which is that the input and the output should have the same length, obviously. This means that our test becomes:

Now that the test has been improved, here is a challenge. Write a new version of my_sort that passes this test and is still wrong. (Keep reading when you are ready for spoilers.)

Notice that we are only testing for the length of the result and whether or not its elements are sorted, but we don’t test which elements are contained in the result. Thus, this fake implementation of my_sort would work:

Use the right numbers

To fix this, we can add the obvious property that the result should only contain numbers from the original list. With sets, this is easy to test:

Now that our test has been improved, I have yet another challenge. Can you write a fake version of my_sort that passes this test? (Keep reading when you are ready for spoilers).

Here is a fake version of my_sort that passes the test above:

The issue here is that we were not precise enough with our new property. In fact, set(result) <= set(int_list) ensures that we only use numbers that were available in the original list, but it doesn’t ensure that we use all of them. What is more, we can’t fix it by simply replacing the <= with == . Can you see why?I will give you a hint. If you just replace the <= with a == , so that the test becomes:

then you can write this passing version of my_sort that is still wrong:

This version is wrong because it reuses the largest element of the original list without respecting the number of times each integer should be used. For example, for the input list [1, 1, 2, 2, 3, 3] the result should be unchanged, whereas this version of my_sort returns [1, 2, 3, 3, 3, 3] .

The final test

A test that is correct and complete would have to take into account how many times each number appears in the original list, which is something the built-in set is not prepared to do. Instead, one could use the collections.Counter from the standard library:

So, at this point, your test function test_my_sort is complete. At this point, it is no longer possible to fool the test! That is, the only way the test will pass is if my_sort is a real sorting function.

Use properties and specific examples

This section showed that the properties that you test should be well thought-through and you should strive to come up with a set of properties that are as specific as possible. When in doubt, it is better to have properties that may look redundant over having too few.

Another strategy that you can follow to help mitigate the danger of having come up with an insufficient set of properties is to mix property-based testing with other forms of testing, which is perfectly reasonable.

For example, on top of having the property-based test test_my_sort , you could add the following test:

This article covered two examples of functions to which we added property-based tests. We only covered the basics of using Hypothesis to run property-based tests but, more importantly, we covered the fundamental concepts that enable a developer to reason about and write complete property-based tests.

Property-based testing isn’t a one-size-fits-all solution that means you will never have to write any other type of test, but it does have characteristics that you should take advantage of whenever possible. In particular, we saw that property-based testing with Hypothesis was beneficial in that:

This article also went over a couple of common gotchas when writing property-based tests and listed scenarios in which property-based testing can be used with no overhead.

If you are interested in learning more about Hypothesis and property-based testing, we recommend you take a look at the Hypothesis docs and, in particular, to the page “What you can generate and how” .

CI/CD Weekly Newsletter 🔔

Semaphore Uncut Podcast 🎙️

Learn ci/cd.

Level up your developer skills to use CI/CD at its max.

5 thoughts on “ Getting Started With Property-Based Testing in Python With Hypothesis and Pytest ”

Awesome intro to property based testing for Python. Thank you, Dan and Rodrigo!

Greeting! Unfortunately, I don’t understand due to translation difficulties. PyCharm writes error messages and does not run the codes. The installation was done fine, check ok. I created a virtual environment. I would like a single good, usable, complete code, an example of what to write in gcd.py and what in test_gcd.py, which the development environment runs without errors. Thanks!

Thanks for article!

“it is better to have properties that may look redundant over having too few” Isn’t it the case with: assert len(result) == len(int_list) and: assert Counter(result) == Counter(int_list) ? I mean: is it possible to satisfy the second condition without satisfying the first ?

Yes. One case could be if result = [0,1], int_list = [0,1,1], and the implementation of Counter returns unique count.

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

CI/CD Weekly Newsletter

🔔 Get notified when the new articles and interviews are out.

hypothesis 6.112.0

pip install hypothesis Copy PIP instructions

Released: Sep 5, 2024

A library for property-based testing

Verified details  (What is this?)

Maintainers.

Unverified details

Project links.

• Documentation
• License: Mozilla Public License 2.0 (MPL 2.0) (MPL-2.0)
• Author: David R. MacIver and Zac Hatfield-Dodds
• Tags python, testing, fuzzing, property-based-testing
• Requires: Python >=3.8
• Provides-Extra: all , cli , codemods , crosshair , dateutil , django , dpcontracts , ghostwriter , lark , numpy , pandas , pytest , pytz , redis , zoneinfo

Classifiers

• 5 - Production/Stable
• OSI Approved :: Mozilla Public License 2.0 (MPL 2.0)
• Microsoft :: Windows
• Python :: 3
• Python :: 3 :: Only
• Python :: 3.8
• Python :: 3.9
• Python :: 3.10
• Python :: 3.11
• Python :: 3.12
• Python :: Implementation :: CPython
• Python :: Implementation :: PyPy
• Education :: Testing
• Software Development :: Testing

Project description

Hypothesis is an advanced testing library for Python. It lets you write tests which are parametrized by a source of examples, and then generates simple and comprehensible examples that make your tests fail. This lets you find more bugs in your code with less work.

Hypothesis is extremely practical and advances the state of the art of unit testing by some way. It’s easy to use, stable, and powerful. If you’re not using Hypothesis to test your project then you’re missing out.

Quick Start/Installation

If you just want to get started:

Links of interest

The main Hypothesis site is at hypothesis.works , and contains a lot of good introductory and explanatory material.

Extensive documentation and examples of usage are available at readthedocs .

If you want to talk to people about using Hypothesis, we have both an IRC channel and a mailing list .

If you want to receive occasional updates about Hypothesis, including useful tips and tricks, there’s a TinyLetter mailing list to sign up for them .

If you want to contribute to Hypothesis, instructions are here .

If you want to hear from people who are already using Hypothesis, some of them have written about it .

If you want to create a downstream package of Hypothesis, please read these guidelines for packagers .

Project details

Release history release notifications | rss feed.

Sep 5, 2024

Aug 24, 2024

Aug 15, 2024

Aug 11, 2024

Aug 8, 2024

Aug 7, 2024

Aug 6, 2024

Aug 5, 2024

Aug 4, 2024

Jul 28, 2024

Jul 22, 2024

Jul 15, 2024

Jul 14, 2024

Jul 13, 2024

Jul 12, 2024

Jul 7, 2024

Jul 4, 2024

Jun 29, 2024

Jun 25, 2024

Jun 24, 2024

Jun 14, 2024

Jun 5, 2024

May 29, 2024

May 23, 2024

May 22, 2024

May 15, 2024

May 13, 2024

May 12, 2024

May 10, 2024

May 6, 2024

May 5, 2024

May 4, 2024

Apr 28, 2024

Apr 8, 2024

Mar 31, 2024

Mar 24, 2024

Mar 23, 2024

Mar 20, 2024

Mar 19, 2024

Mar 18, 2024

Mar 14, 2024

Mar 12, 2024

Mar 11, 2024

Mar 10, 2024

Mar 9, 2024

Mar 4, 2024

Feb 29, 2024

Feb 27, 2024

Feb 25, 2024

Feb 24, 2024

Feb 22, 2024

Feb 20, 2024

Feb 18, 2024

Feb 15, 2024

Feb 14, 2024

Feb 12, 2024

Feb 8, 2024

Feb 5, 2024

Feb 4, 2024

Feb 3, 2024

Jan 31, 2024

Jan 30, 2024

Jan 27, 2024

Jan 25, 2024

Jan 23, 2024

Jan 22, 2024

Jan 21, 2024

Jan 18, 2024

Jan 17, 2024

Jan 16, 2024

Jan 15, 2024

Jan 13, 2024

Jan 12, 2024

Jan 11, 2024

Jan 10, 2024

Jan 8, 2024

Dec 27, 2023

Dec 16, 2023

Dec 10, 2023

Dec 8, 2023

Nov 27, 2023

Nov 20, 2023

Nov 19, 2023

Nov 16, 2023

Nov 13, 2023

Nov 5, 2023

Oct 16, 2023

Oct 15, 2023

Oct 12, 2023

Oct 6, 2023

Oct 1, 2023

Sep 25, 2023

Sep 18, 2023

Sep 17, 2023

Sep 16, 2023

Sep 10, 2023

Sep 6, 2023

Sep 5, 2023

Sep 4, 2023

Sep 3, 2023

Sep 1, 2023

Aug 28, 2023

Aug 20, 2023

Aug 18, 2023

Aug 12, 2023

Aug 8, 2023

Aug 6, 2023

Aug 5, 2023

Jul 20, 2023

Jul 15, 2023

Jul 11, 2023

Jul 10, 2023

Jul 6, 2023

Jun 27, 2023

Jun 26, 2023

Jun 22, 2023

Jun 19, 2023

Jun 17, 2023

Jun 15, 2023

Jun 13, 2023

Jun 12, 2023

Jun 11, 2023

Jun 9, 2023

Jun 4, 2023

May 31, 2023

May 30, 2023

May 27, 2023

May 26, 2023

May 14, 2023

May 4, 2023

Apr 30, 2023

Apr 28, 2023

Apr 26, 2023

Apr 27, 2023

Apr 25, 2023

Apr 24, 2023

Apr 19, 2023

Apr 16, 2023

Apr 7, 2023

Apr 3, 2023

Mar 27, 2023

Mar 16, 2023

Mar 15, 2023

Feb 17, 2023

Feb 12, 2023

Feb 9, 2023

Feb 5, 2023

Feb 4, 2023

Feb 3, 2023

Feb 2, 2023

Jan 27, 2023

Jan 26, 2023

Jan 24, 2023

Jan 23, 2023

Jan 20, 2023

Jan 14, 2023

Jan 8, 2023

Jan 7, 2023

Jan 6, 2023

Dec 11, 2022

Dec 4, 2022

Dec 2, 2022

Nov 30, 2022

Nov 26, 2022

Nov 19, 2022

Nov 14, 2022

Oct 28, 2022

Oct 17, 2022

Oct 10, 2022

Oct 5, 2022

Oct 2, 2022

Sep 29, 2022

Sep 18, 2022

Sep 5, 2022

Aug 20, 2022

Aug 12, 2022

Aug 10, 2022

Aug 2, 2022

Jul 25, 2022

Jul 22, 2022

Jul 19, 2022

Jul 18, 2022

Jul 17, 2022

Jul 9, 2022

Jul 5, 2022

Jul 4, 2022

Jul 3, 2022

Jun 29, 2022

Jun 27, 2022

Jun 25, 2022

Jun 23, 2022

Jun 15, 2022

Jun 12, 2022

Jun 10, 2022

Jun 7, 2022

Jun 2, 2022

Jun 1, 2022

May 25, 2022

May 19, 2022

May 18, 2022

May 15, 2022

May 11, 2022

May 3, 2022

May 1, 2022

Apr 30, 2022

Apr 29, 2022

Apr 27, 2022

Apr 22, 2022

Apr 21, 2022

Apr 18, 2022

Apr 16, 2022

Apr 13, 2022

Apr 12, 2022

Apr 10, 2022

Apr 9, 2022

Apr 1, 2022

Mar 29, 2022

Mar 27, 2022

Mar 26, 2022

Mar 17, 2022

Mar 7, 2022

Mar 3, 2022

Mar 1, 2022

Feb 26, 2022

Feb 21, 2022

Feb 18, 2022

Feb 13, 2022

Jan 31, 2022

Jan 19, 2022

Jan 17, 2022

Jan 8, 2022

Jan 5, 2022

Dec 31, 2021

Dec 30, 2021

Dec 23, 2021

Dec 15, 2021

Dec 14, 2021

Dec 11, 2021

Dec 10, 2021

Dec 9, 2021

Dec 5, 2021

Dec 3, 2021

Dec 2, 2021

Nov 29, 2021

Nov 28, 2021

Nov 26, 2021

Nov 22, 2021

Nov 21, 2021

Nov 19, 2021

Nov 18, 2021

Nov 16, 2021

Nov 15, 2021

Nov 13, 2021

Nov 5, 2021

Nov 1, 2021

Oct 23, 2021

Oct 20, 2021

Oct 18, 2021

Oct 8, 2021

Sep 29, 2021

Sep 26, 2021

Sep 24, 2021

Sep 19, 2021

Sep 16, 2021

Sep 15, 2021

Sep 13, 2021

Sep 11, 2021

Sep 10, 2021

Sep 9, 2021

Sep 8, 2021

Sep 6, 2021

Aug 31, 2021

Aug 30, 2021

Aug 29, 2021

Aug 27, 2021

Aug 22, 2021

Aug 20, 2021

Aug 16, 2021

Aug 14, 2021

Aug 7, 2021

Jul 27, 2021

Jul 26, 2021

Jul 18, 2021

Jul 12, 2021

Jul 2, 2021

Jun 9, 2021

Jun 4, 2021

Jun 3, 2021

Jun 2, 2021

May 30, 2021

May 28, 2021

May 27, 2021

May 26, 2021

May 24, 2021

May 23, 2021

May 20, 2021

May 18, 2021

May 17, 2021

May 6, 2021

Apr 26, 2021

Apr 17, 2021

Apr 15, 2021

Apr 12, 2021

Apr 11, 2021

Apr 7, 2021

Apr 6, 2021

Apr 5, 2021

Apr 1, 2021

Mar 28, 2021

Mar 27, 2021

Mar 14, 2021

Mar 11, 2021

Mar 10, 2021

Mar 9, 2021

Mar 7, 2021

Mar 4, 2021

Mar 2, 2021

Feb 28, 2021

Feb 26, 2021

Feb 25, 2021

Feb 24, 2021

Feb 20, 2021

Feb 12, 2021

Jan 31, 2021

Jan 29, 2021

Jan 27, 2021

Jan 23, 2021

Jan 14, 2021

Jan 13, 2021

Jan 8, 2021

Jan 7, 2021

Jan 6, 2021

Jan 5, 2021

Jan 4, 2021

Jan 3, 2021

Jan 2, 2021

Jan 1, 2021

Dec 24, 2020

Dec 11, 2020

Dec 10, 2020

Dec 9, 2020

Dec 5, 2020

Nov 28, 2020

Nov 18, 2020

Nov 8, 2020

Nov 3, 2020

Oct 30, 2020

Oct 26, 2020

Oct 24, 2020

Oct 20, 2020

Oct 15, 2020

Oct 14, 2020

Oct 7, 2020

Oct 3, 2020

Oct 2, 2020

Sep 25, 2020

Sep 24, 2020

Sep 21, 2020

Sep 15, 2020

Sep 14, 2020

Sep 11, 2020

Sep 9, 2020

Sep 7, 2020

Sep 6, 2020

Sep 4, 2020

Aug 30, 2020

Aug 28, 2020

Aug 27, 2020

Aug 24, 2020

Aug 20, 2020

Aug 19, 2020

Aug 17, 2020

Aug 16, 2020

Aug 14, 2020

Aug 13, 2020

Aug 12, 2020

Aug 10, 2020

Aug 4, 2020

Aug 3, 2020

Jul 31, 2020

Jul 29, 2020

Jul 27, 2020

Jul 26, 2020

Jul 25, 2020

Jul 23, 2020

Jul 21, 2020

Jul 18, 2020

Jul 17, 2020

Jul 15, 2020

Jul 13, 2020

Jul 12, 2020

Jun 30, 2020

Jun 27, 2020

Jun 26, 2020

Jun 25, 2020

Jun 22, 2020

Jun 21, 2020

Jun 19, 2020

Jun 10, 2020

May 27, 2020

May 21, 2020

May 19, 2020

May 13, 2020

May 12, 2020

May 10, 2020

May 7, 2020

May 4, 2020

Apr 24, 2020

Apr 22, 2020

Apr 19, 2020

Apr 18, 2020

Apr 16, 2020

Apr 15, 2020

Apr 14, 2020

Apr 12, 2020

Mar 24, 2020

Mar 23, 2020

Mar 19, 2020

Mar 18, 2020

Feb 29, 2020

Feb 16, 2020

Feb 14, 2020

Feb 13, 2020

Feb 7, 2020

Feb 6, 2020

Feb 1, 2020

Jan 30, 2020

Jan 26, 2020

Jan 21, 2020

Jan 19, 2020

Jan 12, 2020

Jan 11, 2020

Jan 9, 2020

Jan 6, 2020

Jan 3, 2020

Jan 1, 2020

Dec 29, 2019

Dec 28, 2019

Dec 22, 2019

Dec 21, 2019

Dec 19, 2019

Dec 18, 2019

Dec 17, 2019

Dec 16, 2019

Dec 15, 2019

Dec 11, 2019

Dec 9, 2019

Dec 7, 2019

Dec 5, 2019

Dec 2, 2019

Dec 1, 2019

Nov 29, 2019

Nov 28, 2019

Nov 27, 2019

Nov 26, 2019

Nov 25, 2019

Nov 24, 2019

Nov 23, 2019

Nov 22, 2019

Nov 20, 2019

Nov 12, 2019

Nov 11, 2019

Nov 8, 2019

Nov 7, 2019

Nov 6, 2019

Nov 5, 2019

Nov 4, 2019

Nov 3, 2019

Nov 2, 2019

Nov 1, 2019

Oct 30, 2019

Oct 27, 2019

Oct 21, 2019

Oct 17, 2019

Oct 16, 2019

Oct 14, 2019

Oct 9, 2019

Oct 7, 2019

Oct 4, 2019

Oct 2, 2019

Oct 1, 2019

Sep 28, 2019

Sep 20, 2019

Sep 17, 2019

Sep 9, 2019

Sep 4, 2019

Aug 23, 2019

Aug 21, 2019

Aug 20, 2019

Aug 5, 2019

Jul 30, 2019

Jul 29, 2019

Jul 28, 2019

Jul 24, 2019

Jul 14, 2019

Jul 12, 2019

Jul 11, 2019

Jul 8, 2019

Jul 7, 2019

Jul 5, 2019

Jul 4, 2019

Jul 3, 2019

Jun 26, 2019

Jun 23, 2019

Jun 21, 2019

Jun 7, 2019

Jun 6, 2019

Jun 4, 2019

May 29, 2019

May 28, 2019

May 26, 2019

May 19, 2019

May 16, 2019

May 9, 2019

May 8, 2019

May 7, 2019

May 6, 2019

May 5, 2019

Apr 30, 2019

Apr 29, 2019

Apr 24, 2019

Apr 19, 2019

Apr 16, 2019

Apr 12, 2019

Apr 9, 2019

Apr 7, 2019

Apr 5, 2019

Apr 3, 2019

Mar 31, 2019

Mar 30, 2019

Mar 19, 2019

Mar 18, 2019

Mar 15, 2019

Mar 13, 2019

Mar 12, 2019

Mar 11, 2019

Mar 9, 2019

Mar 6, 2019

Mar 4, 2019

Mar 3, 2019

Mar 1, 2019

Feb 28, 2019

Feb 27, 2019

Feb 25, 2019

Feb 24, 2019

Feb 23, 2019

Feb 22, 2019

Feb 21, 2019

Feb 19, 2019

Feb 18, 2019

Feb 15, 2019

Feb 14, 2019

Feb 12, 2019

Feb 11, 2019

Feb 10, 2019

Feb 8, 2019

Feb 6, 2019

Feb 5, 2019

Feb 3, 2019

Feb 2, 2019

Jan 25, 2019

Jan 24, 2019

Jan 23, 2019

Jan 22, 2019

Jan 16, 2019

Jan 14, 2019

Jan 11, 2019

Jan 10, 2019

Jan 9, 2019

Jan 8, 2019

Jan 7, 2019

Jan 6, 2019

Jan 4, 2019

Jan 3, 2019

Jan 2, 2019

Dec 31, 2018

Dec 30, 2018

Dec 29, 2018

Dec 28, 2018

Dec 21, 2018

Dec 20, 2018

Dec 19, 2018

Dec 18, 2018

Dec 17, 2018

Dec 13, 2018

Dec 12, 2018

Dec 11, 2018

Dec 8, 2018

Oct 29, 2018

Oct 27, 2018

Oct 25, 2018

Oct 23, 2018

Oct 22, 2018

Oct 18, 2018

Oct 16, 2018

Oct 11, 2018

Oct 10, 2018

Oct 9, 2018

Oct 8, 2018

Oct 3, 2018

Oct 1, 2018

Sep 30, 2018

Sep 27, 2018

Sep 26, 2018

Sep 25, 2018

Sep 24, 2018

Sep 18, 2018

Sep 17, 2018

Sep 16, 2018

Sep 15, 2018

Sep 14, 2018

Sep 9, 2018

Sep 8, 2018

Sep 3, 2018

Sep 1, 2018

Aug 30, 2018

Aug 29, 2018

Aug 28, 2018

Aug 27, 2018

Aug 23, 2018

Aug 21, 2018

Aug 20, 2018

Aug 19, 2018

Aug 18, 2018

Aug 15, 2018

Aug 14, 2018

Aug 10, 2018

Aug 9, 2018

Aug 8, 2018

Aug 6, 2018

Aug 5, 2018

Aug 3, 2018

Aug 2, 2018

Aug 1, 2018

Jul 31, 2018

Jul 30, 2018

Jul 28, 2018

Jul 26, 2018

Jul 24, 2018

Jul 23, 2018

Jul 22, 2018

Jul 20, 2018

Jul 19, 2018

Jul 8, 2018

Jul 5, 2018

Jul 4, 2018

Jul 3, 2018

Jun 30, 2018

Jun 27, 2018

Jun 26, 2018

Jun 24, 2018

Jun 20, 2018

Jun 19, 2018

Jun 18, 2018

Jun 16, 2018

Jun 14, 2018

Jun 13, 2018

May 20, 2018

May 16, 2018

May 11, 2018

May 10, 2018

May 9, 2018

Apr 22, 2018

Apr 21, 2018

Apr 20, 2018

Apr 17, 2018

Apr 14, 2018

Apr 13, 2018

Apr 12, 2018

Apr 11, 2018

Apr 6, 2018

Apr 5, 2018

Apr 4, 2018

Apr 1, 2018

Mar 30, 2018

Mar 29, 2018

Mar 24, 2018

Mar 20, 2018

Mar 19, 2018

Mar 15, 2018

Mar 12, 2018

Mar 5, 2018

Mar 2, 2018

Mar 1, 2018

Feb 26, 2018

Feb 25, 2018

Feb 23, 2018

Feb 18, 2018

Feb 17, 2018

Feb 13, 2018

Feb 5, 2018

Jan 27, 2018

Jan 24, 2018

Jan 23, 2018

Jan 22, 2018

Jan 21, 2018

Jan 20, 2018

Jan 13, 2018

Jan 8, 2018

Jan 7, 2018

Jan 6, 2018

Jan 4, 2018

Jan 2, 2018

Dec 23, 2017

Dec 21, 2017

Dec 20, 2017

Dec 17, 2017

Dec 12, 2017

Dec 10, 2017

Dec 9, 2017

Dec 6, 2017

Dec 4, 2017

Dec 2, 2017

Dec 1, 2017

Nov 29, 2017

Nov 28, 2017

Nov 23, 2017

Nov 22, 2017

Nov 21, 2017

Nov 18, 2017

Nov 12, 2017

Nov 10, 2017

Nov 6, 2017

Nov 2, 2017

Nov 1, 2017

Oct 16, 2017

Oct 15, 2017

Oct 13, 2017

Oct 9, 2017

Oct 8, 2017

Oct 6, 2017

Sep 30, 2017

Sep 29, 2017

Sep 27, 2017

Sep 25, 2017

Sep 24, 2017

Sep 22, 2017

Sep 19, 2017

Sep 18, 2017

Sep 16, 2017

Sep 15, 2017

Sep 14, 2017

Sep 13, 2017

Sep 12, 2017

Sep 11, 2017

Sep 6, 2017

Sep 5, 2017

Sep 1, 2017

Aug 31, 2017

Aug 29, 2017

Aug 28, 2017

Aug 26, 2017

Aug 25, 2017

Aug 24, 2017

Aug 23, 2017

Aug 22, 2017

Aug 21, 2017

Aug 20, 2017

Aug 18, 2017

Aug 17, 2017

Aug 16, 2017

Aug 15, 2017

Aug 13, 2017

Aug 7, 2017

Aug 4, 2017

Aug 3, 2017

Aug 2, 2017

Jul 23, 2017

Jul 20, 2017

Jul 16, 2017

Jul 7, 2017

Jun 19, 2017

Jun 17, 2017

Jun 11, 2017

Jun 10, 2017

May 28, 2017

May 23, 2017

May 22, 2017

May 19, 2017

May 17, 2017

May 9, 2017

Apr 26, 2017

Apr 23, 2017

Apr 22, 2017

Apr 21, 2017

Mar 20, 2017

Dec 20, 2016

Oct 31, 2016

Oct 5, 2016

Sep 26, 2016

Sep 23, 2016

Sep 22, 2016

Jul 13, 2016

Jul 7, 2016

May 27, 2016

May 24, 2016

May 1, 2016

Apr 30, 2016

Apr 29, 2016

Mar 6, 2016

Feb 25, 2016

Feb 24, 2016

Feb 23, 2016

Feb 18, 2016

Feb 17, 2016

Jan 10, 2016

Jan 9, 2016

Dec 22, 2015

Dec 21, 2015

Dec 16, 2015

Dec 15, 2015

Dec 8, 2015

Nov 24, 2015

Nov 1, 2015

Oct 29, 2015

Oct 18, 2015

Sep 27, 2015

Sep 23, 2015

Sep 16, 2015

Aug 31, 2015

Aug 26, 2015

Aug 22, 2015

Aug 19, 2015

Aug 4, 2015

Aug 3, 2015

Jul 27, 2015

Jul 24, 2015

Jul 21, 2015

Jul 20, 2015

Jul 18, 2015

Jul 17, 2015

Jul 16, 2015

Jul 10, 2015

Jun 29, 2015

Jun 8, 2015

May 21, 2015

May 14, 2015

May 5, 2015

May 4, 2015

Apr 22, 2015

Apr 15, 2015

Apr 14, 2015

Apr 7, 2015

Apr 6, 2015

Mar 27, 2015

Mar 26, 2015

Mar 25, 2015

Mar 23, 2015

Mar 22, 2015

Mar 21, 2015

Mar 20, 2015

Mar 14, 2015

Feb 10, 2015

Feb 5, 2015

Feb 4, 2015

Feb 3, 2015

Jan 21, 2015

Jan 16, 2015

Jan 13, 2015

Jan 12, 2015

Jan 8, 2015

Jan 7, 2015

Dec 14, 2013

May 3, 2013

Mar 26, 2013

Mar 24, 2013

Mar 23, 2013

Mar 13, 2013

Mar 12, 2013

Mar 10, 2013

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages .

Source Distribution

Uploaded Sep 5, 2024 Source

Built Distribution

Uploaded Sep 5, 2024 Python 3

Hashes for hypothesis-6.112.0.tar.gz

Hashes for hypothesis-6.112.0-py3-none-any.whl

• português (Brasil)

Supported by

Table of Contents

Testing your python code with hypothesis, installing & using hypothesis, a quick example, understanding hypothesis, using hypothesis strategies, filtering and mapping strategies, composing strategies, constraints & satisfiability, writing reusable strategies with functions.

• @composite: Declarative Strategies
• @example: Explicitly Testing Certain Values

Hypothesis Example: Roman Numeral Converter

I can think of a several Python packages that greatly improved the quality of the software I write. Two of them are pytest and hypothesis . The former adds an ergonomic framework for writing tests and fixtures and a feature-rich test runner. The latter adds property-based testing that can ferret out all but the most stubborn bugs using clever algorithms, and that’s the package we’ll explore in this course.

In an ordinary test you interface with the code you want to test by generating one or more inputs to test against, and then you validate that it returns the right answer. But that, then, raises a tantalizing question: what about all the inputs you didn’t test? Your code coverage tool may well report 100% test coverage, but that does not, ipso facto , mean the code is bug-free.

One of the defining features of Hypothesis is its ability to generate test cases automatically in a manner that is:

Repeated invocations of your tests result in reproducible outcomes, even though Hypothesis does use randomness to generate the data.

You are given a detailed answer that explains how your test failed and why it failed. Hypothesis makes it clear how you, the human, can reproduce the invariant that caused your test to fail.

You can refine its strategies and tell it where or what it should or should not search for. At no point are you compelled to modify your code to suit the whims of Hypothesis if it generates nonsensical data.

So let’s look at how Hypothesis can help you discover errors in your code.

You can install hypothesis by typing pip install hypothesis . It has few dependencies of its own, and should install and run everywhere.

Hypothesis plugs into pytest and unittest by default, so you don’t have to do anything to make it work with it. In addition, Hypothesis comes with a CLI tool you can invoke with hypothesis . But more on that in a bit.

I will use pytest throughout to demonstrate Hypothesis, but it works equally well with the builtin unittest module.

Before I delve into the details of Hypothesis, let’s start with a simple example: a naive CSV writer and reader. A topic that seems simple enough: how hard is it to separate fields of data with a comma and then read it back in later?

But of course CSV is frighteningly hard to get right. The US and UK use '.' as a decimal separator, but in large parts of the world they use ',' which of course results in immediate failure. So then you start quoting things, and now you need a state machine that can distinguish quoted from unquoted; and what about nested quotes, etc.

The naive CSV reader and writer is an excellent stand-in for any number of complex projects where the requirements outwardly seem simple but there lurks a large number of edge cases that you must take into account.

Here the writer simply string quotes each field before joining them together with ',' . The reader does the opposite: it assumes each field is quoted after it is split by the comma.

A naive roundtrip pytest proves the code “works”:

And evidently so:

And for a lot of code that’s where the testing would begin and end. A couple of lines of code to test a couple of functions that outwardly behave in a manner that anybody can read and understand. Now let’s look at what a Hypothesis test would look like, and what happens when we run it:

At first blush there’s nothing here that you couldn’t divine the intent of, even if you don’t know Hypothesis. I’m asking for the argument fields to have a list ranging from one element of generated text up to ten. Aside from that, the test operates in exactly the same manner as before.

Now watch what happens when I run the test:

Hypothesis quickly found an example that broke our code. As it turns out, a list of [','] breaks our code. We get two fields back after round-tripping the code through our CSV writer and reader — uncovering our first bug.

In a nutshell, this is what Hypothesis does. But let’s look at it in detail.

Simply put, Hypothesis generates data using a number of configurable strategies . Strategies range from simple to complex. A simple strategy may generate bools; another integers. You can combine strategies to make larger ones, such as lists or dicts that match certain patterns or structures you want to test. You can clamp their outputs based on certain constraints, like only positive integers or strings of a certain length. You can also write your own strategies if you have particularly complex requirements.

Strategies are the gateway to property-based testing and are a fundamental part of how Hypothesis works. You can find a detailed list of all the strategies in the Strategies reference of their documentation or in the hypothesis.strategies module.

The best way to get a feel for what each strategy does in practice is to import them from the hypothesis.strategies module and call the example() method on an instance:

You may have noticed that the floats example included inf in the list. By default, all strategies will – where feasible – attempt to test all legal (but possibly obscure) forms of values you can generate of that type. That is particularly important as corner cases like inf or NaN are legal floating-point values but, I imagine, not something you’d ordinarily test against yourself.

And that’s one pillar of how Hypothesis tries to find bugs in your code: by testing edge cases that you would likely miss yourself. If you ask it for a text() strategy you’re as likely to be given Western characters as you are a mishmash of unicode and escape-encoded garbage. Understanding why Hypothesis generates the examples it does is a useful way to think about how your code may interact data it has no control over.

Now if it were simply generating text or numbers from an inexhaustible source of numbers or strings, it wouldn’t catch as many errors as it actually does . The reason for that is that each test you write is subjected to a battery of examples drawn from the strategies you’ve designed. If a test case fails, it’s put aside and tested again but with a reduced subset of inputs, if possible. In Hypothesis it’s known as shrinking the search space to try and find the smallest possible result that will cause your code to fail. So instead of a 10,000-length string, if it can find one that’s only 3 or 4, it will try to show that to you instead.

You can tell Hypothesis to filter or map the examples it draws to further reduce them if the strategy does not meet your requirements:

Here I ask for integers where the number is greater than 0 and is evenly divisible by 8. Hypothesis will then attempt to generate examples that meets the constraints you have imposed on it.

You can also map , which works in much the same way as filter. Here I’m asking for lowercase ASCII and then uppercasing them:

Having said that, using either when you don’t have to (I could have asked for uppercase ASCII characters to begin with) is likely to result in slower strategies.

A third option, flatmap , lets you build strategies from strategies; but that deserves closer scrutiny, so I’ll talk about it later.

You can tell Hypothesis to pick one of a number of strategies by composing strategies with | or st.one_of() :

An essential feature when you have to draw from multiple sources of examples for a single data point.

When you ask Hypothesis to draw an example it takes into account the constraints you may have imposed on it: only positive integers; only lists of numbers that add up to exactly 100; any filter() calls you may have applied; and so on. Those are constraints. You’re taking something that was once unbounded (with respect to the strategy you’re drawing an example from, that is) and introducing additional limitations that constrain the possible range of values it can give you.

But consider what happens if I pass filters that will yield nothing at all:

At some point Hypothesis will give up and declare it cannot find anything that satisfies that strategy and its constraints.

Hypothesis gives up after a while if it’s not able to draw an example. Usually that indicates an invariant in the constraints you’ve placed that makes it hard or impossible to draw examples from. In the example above, I asked for numbers that are simultaneously below zero and greater than zero, which is an impossible request.

As you can see, the strategies are simple functions, and they behave as such. You can therefore refactor each strategy into reusable patterns:

The benefit of this approach is that if you discover edge cases that Hypothesis does not account for, you can update the pattern in one place and observe its effects on your code. It’s functional and composable.

One caveat of this approach is that you cannot draw examples and expect Hypothesis to behave correctly. So I don’t recommend you call example() on a strategy only to pass it into another strategy.

For that, you want the @composite decorator.

@composite : Declarative Strategies

If the previous approach is unabashedly functional in nature, this approach is imperative.

The @composite decorator lets you write imperative Python code instead. If you cannot easily structure your strategy with the built-in ones, or if you require more granular control over the values it emits, you should consider the @composite strategy.

Instead of returning a compound strategy object like you would above, you instead draw examples using a special function you’re given access to in the decorated function.

This example draws two randomized names and returns them as a tuple:

Note that the @composite decorator passes in a special draw callable that you must use to draw samples from. You cannot – well, you can , but you shouldn’t – use the example() method on the strategy object you get back. Doing so will break Hypothesis’s ability to synthesize test cases properly.

Because the code is imperative you’re free to modify the drawn examples to your liking. But what if you’re given an example you don’t like or one that breaks a known invariant you don’t wish to test for? For that you can use the assume() function to state the assumptions that Hypothesis must meet if you try to draw an example from generate_full_name .

Let’s say that first_name and last_name must not be equal:

Like the assert statement in Python, the assume() function teaches Hypothesis what is, and is not, a valid example. You use this to great effect to generate complex compound strategies.

I recommend you observe the following rules of thumb if you write imperative strategies with @composite :

If you want to draw a succession of examples to initialize, say, a list or a custom object with values that meet certain criteria you should use filter , where possible, and assume to teach Hypothesis why the value(s) you drew and subsequently discarded weren’t any good.

The example above uses assume() to teach Hypothesis that first_name and last_name must not be equal.

If you can put your functional strategies in separate functions, you should. It encourages code re-use and if your strategies are failing (or not generating the sort of examples you’d expect) you can inspect each strategy in turn. Large nested strategies are harder to untangle and harder still to reason about.

If you can express your requirements with filter and map or the builtin constraints (like min_size or max_size ), you should. Imperative strategies that use assume may take more time to converge on a valid example.

@example : Explicitly Testing Certain Values

Occasionally you’ll come across a handful of cases that either fails or used to fail, and you want to ensure that Hypothesis does not forget to test them, or to indicate to yourself or your fellow developers that certain values are known to cause issues and should be tested explicitly.

The @example decorator does just that:

You can add as many as you like.

Let’s say I wanted to write a simple converter to and from Roman numerals.

Here I’m collecting Roman numerals into numerals , one at a time, by looping over SYMBOLS of valid numerals, subtracting the value of the symbol from number , until the while loop’s condition ( number >= 1 ) is False .

The test is also simple and serves as a smoke test. I generate a random integer and convert it to Roman numerals with to_roman . When it’s all said and done I turn the string of numerals into a set and check that all members of the set are legal Roman numerals.

Now if I run pytest on it seems to hang . But thanks to Hypothesis’s debug mode I can inspect why:

Ah. Instead of testing with tiny numbers like a human would ordinarily do, it used a fantastically large one… which is altogether slow.

OK, so there’s at least one gotcha; it’s not really a bug , but it’s something to think about: limiting the maximum value. I’m only going to limit the test, but it would be reasonable to limit it in the code also.

Changing the max_value to something sensible, like st.integers(max_value=5000) and the test now fails with another error:

It seems our code’s not able to handle the number 0! Which… is correct. The Romans didn’t really use the number zero as we would today; that invention came later, so they had a bunch of workarounds to deal with the absence of something. But that’s neither here nor there in our example. Let’s instead set min_value=1 also, as there is no support for negative numbers either:

OK… not bad. We’ve proven that given a random assortment of numbers between our defined range of values that, indeed, we get something resembling Roman numerals.

One of the hardest things about Hypothesis is framing questions to your testable code in a way that tests its properties but without you, the developer, knowing the answer (necessarily) beforehand. So one simple way to test that there’s at least something semi-coherent coming out of our to_roman function is to check that it can generate the very numerals we defined in SYMBOLS from before:

Here I’m sampling from a tuple of the SYMBOLS from earlier. The sampling algorithm’ll decide what values it wants to give us, all we care about is that we are given examples like ("I", 1) or ("V", 5) to compare against.

So let’s run pytest again:

Oops. The Roman numeral V is equal to 5 and yet we get five IIIII ? A closer examination reveals that, indeed, the code only yields sequences of I equal to the number we pass it. There’s a logic error in our code.

In the example above I loop over the elements in the SYMBOLS dictionary but as it’s ordered the first element is always I . And as the smallest representable value is 1, we end up with just that answer. It’s technically correct as you can count with just I but it’s not very useful.

Fixing it is easy though:

Rerunning the test yields a pass. Now we know that, at the very least, our to_roman function is capable of mapping numbers that are equal to any symbol in SYMBOLS .

Now the litmus test is taking the numeral we’re given and making sense of it. So let’s write a function that converts a Roman numeral back into decimal:

Like to_roman we walk through each character, get the numeral’s numeric value, and add it to the running total. The test is a simple roundtrip test as to_roman has an inverse function from_roman (and vice versa) such that :

Invertible functions are easier to test because you can compare the output of one against the input of another and check if it yields the original value. But not every function has an inverse, though.

Running the test yields a pass:

So now we’re in a pretty good place. But there’s a slight oversight in our Roman numeral converters, though: they don’t respect the subtraction rule for some of the numerals. For instance VI is 6; but IV is 4. The value XI is 11; and IX is 9. Only some (sigh) numerals exhibit this property.

So let’s write another test. This time it’ll fail as we’ve yet to write the modified code. Luckily we know the subtractive numerals we must accommodate:

Pretty simple test. Check that certain numerals yield the value, and that the values yield the right numeral.

With an extensive test suite we should feel fairly confident making changes to the code. If we break something, one of our preexisting tests will fail.

The rules around which numerals are subtractive is rather subjective. The SUBTRACTIVE_SYMBOLS dictionary holds the most common ones. So all we need to do is read ahead of the numerals list to see if there exists a two-digit numeral that has a prescribed value and then we use that instead of the usual value.

The to_roman change is simple. A union of the two numeral symbol dictionaries is all it takes . The code already understands how to turn numbers into numerals — we just added a few more.

This method requires Python 3.9 or later. Read how to merge dictionaries

If done right, running the tests should yield a pass:

And that’s it. We now have useful tests and a functional Roman numeral converter that converts to and from with ease. But one thing we didn’t do is create a strategy that generates Roman numerals using st.text() . A custom composite strategy to generate both valid and invalid Roman numerals to test the ruggedness of our converter is left as an exercise to you.

In the next part of this course we’ll look at more advanced testing strategies.

Unlike a tool like faker that generates realistic-looking test data for fixtures or demos, Hypothesis is a property-based tester . It uses heuristics and clever algorithms to find inputs that break your code.

Testing a function that does not have an inverse to compare the result against – like our Roman numeral converter that works both ways – you often have to approach your code as though it were a black box where you relinquish control of the inputs and outputs. That is harder, but makes for less brittle code.

It’s perfectly fine to mix and match tests. Hypothesis is useful for flushing out invariants you would never think of. Combine it with known inputs and outputs to jump start your testing for the first 80%, and augment it with Hypothesis to catch the remaining 20%.

Be Inspired Sign up and we’ll tell you about new articles and courses

Absolutely no spam. We promise!

Liked the Article?

Why not follow us …, be inspired get python tips sent to your inbox.

We'll tell you about the latest courses and articles.

Next-Gen App & Browser Testing Cloud

Trusted by 2 Mn+ QAs & Devs to accelerate their release cycles

Automation Playwright Testing Selenium Python Tutorial

• What Is Hypothesis Testing in Python: A Hands-On Tutorial

Jaydeep Karale

Posted On: June 5, 2024

In software testing, there is an approach known as property-based testing that leverages the concept of formal specification of code behavior and focuses on asserting properties that hold true for a wide range of inputs rather than individual test cases.

Python is an open-source programming language that provides a Hypothesis library for property-based testing. Hypothesis testing in Python provides a framework for generating diverse and random test data, allowing development and testing teams to thoroughly test their code against a broad spectrum of inputs.

In this blog, we will explore the fundamentals of Hypothesis testing in Python using Selenium and Playwright. We’ll learn various aspects of Hypothesis testing, from basic usage to advanced strategies, and demonstrate how it can improve the robustness and reliability of the codebase.

TABLE OF CONTENTS

What Is a Hypothesis Library?

Decorators in hypothesis, strategies in hypothesis, setting up python environment for hypothesis testing, how to perform hypothesis testing in python, hypothesis testing in python with selenium and playwright.

• How to Run Hypothesis Testing in Python With Date Strategy?
• How to Write Composite Strategies in Hypothesis Testing in Python?

Frequently Asked Questions (FAQs)

Hypothesis is a property-based testing library that automates test data generation based on properties or invariants defined by the developers and testers.

In property-based testing, instead of specifying individual test cases, developers define general properties that the code should satisfy. Hypothesis then generates a wide range of input data to test these properties automatically.

Property-based testing using Hypothesis allows developers and testers to focus on defining the behavior of their code rather than writing specific test cases, resulting in more comprehensive testing coverage and the discovery of edge cases and unexpected behavior.

Writing property-based tests usually consists of deciding on guarantees our code should make – properties that should always hold, regardless of what the world throws at the code.

Examples of such guarantees can be:

• Your code shouldn’t throw an exception or should only throw a particular type of exception (this works particularly well if you have a lot of internal assertions).
• If you delete an object, it is no longer visible.
• If you serialize and then deserialize a value, you get the same value back.

Before we proceed further, it’s worthwhile to understand decorators in Python a bit since the Hypothesis library exposes decorators that we need to use to write tests.

In Python, decorators are a powerful feature that allows you to modify or extend the behavior of functions or classes without changing their source code. Decorators are essentially functions themselves, which take another function (or class) as input and return a new function (or class) with added functionality.

Decorators are denoted by the @ symbol followed by the name of the decorator function placed directly before the definition of the function or class to be modified.

Let us understand this with the help of an example:

In the example above, only authenticated users are allowed to create_post() . The logic to check authentication is wrapped in its own function, authenticate() .

This function can now be called using @authenticate before beginning a function where it’s needed & Python would automatically know that it needs to execute the code of authenticate() before calling the function.

If we no longer need the authentication logic in the future, we can simply remove the @authenticate line without disturbing the core logic. Thus, decorators are a powerful construct in Python that allows plug-n-play of repetitive logic into any function/method.

Now that we know the concept of Python decorators, let us understand the given decorators that which Hypothesis provides.

Hypothesis @given Decorator

This decorator turns a test function that accepts arguments into a randomized test. It serves as the main entry point to the Hypothesis.

The @given decorator can be used to specify which arguments of a function should be parameterized over. We can use either positional or keyword arguments, but not a mixture of both.

Some valid declarations of the @given decorator are:

Some invalid declarations of @given are:

Hypothesis @example Decorator

When writing production-grade applications, the ability of a Hypothesis to generate a wide range of input test data plays a crucial role in ensuring robustness.

However, there are certain inputs/scenarios the testing team might deem mandatory to be tested as part of every test run. Hypothesis has the @example decorator in such cases where we can specify values we always want to be tested. The @example decorator works for all strategies.

Let’s understand by tweaking the factorial test example.

The above test will always run for the input value 41 along with other custom-generated test data by the Hypothesis st.integers() function.

By now, we understand that the crux of the Hypothesis is to test a function for a wide range of inputs. These inputs are generated automatically, and the Hypothesis lets us configure the range of inputs. Under the hood, the strategy method takes care of the process of generating this test data of the correct data type.

Hypothesis offers a wide range of strategies such as integers, text, boolean, datetime, etc. For more complex scenarios, which we will see a bit later in this blog, the hypothesis also lets us set up composite strategies.

While not exhaustive, here is a tabular summary of strategies available as part of the Hypothesis library.

Let’s see the steps to how to set up a test environment to perform Hypothesis testing in Python.

• Create a separate virtual environment for this project using the built-in venv module of Python using the command.

• Activate the newly created virtual environment using the activate script present within the environment.

• Install the Hypothesis library necessary for property-based testing using the pip install hypothesis command. The installed package can be viewed using the command pip list. When writing this blog, the latest version of Hypothesis is 6.102.4. For this article, we have used the Hypothesis version 6.99.6.

• Install python-dotenv , pytest, Playwright, and Selenium packages which we will need to run the tests on the cloud. We will talk about this in more detail later in the blog.

Our final project structure setup looks like below:

With the setup done, let us now understand Hypothesis testing in Python with various examples, starting with the introductory one and then working toward more complex ones.

Subscribe to the LambdaTest YouTube Channel for quick updates on the tutorials around Selenium Python and more.

Let’s now start writing tests to understand how we can leverage the Hypothesis library to perform Python automation .

For this, let’s look at one test scenario to understand Hypothesis testing in Python.

Test Scenario:

Implementation:

This is what the initial implementation of the function looks like:

It takes in an integer as an input. If the input is 0, it raises an error; if not, it uses the range() function to generate a list of numbers within, iterate over it, calculate the factorial, and return it.

Let’s now write a test using the Hypothesis library to test the above function:

Code Walkthrough:

Let’s now understand the step-by-step code walkthrough for Hypothesis testing in Python.

Step 1: From the Hypothesis library, we import the given decorator and strategies method.

Step 2: Using the imported given and strategies, we set our test strategy of passing integer inputs within the range of 1 to 30 to the function under test using the min_value and max_value arguments.

Step 3: We write the actual test_factorial where the integer generated by our strategy is passed automatically by Hypothesis into the value num.

Using this value we call the factorial function once for value num and num – 1.

Next, we divide the factorial of num by the factorial of num -1 and assert if the result of the operation is equal to the original num.

Test Execution:

Let’s now execute our hypothesis test using the pytest -v -k “test_factorial” command.

And Hypothesis confirms that our function works perfectly for the given set of inputs, i.e., for integers from 1 to 30.

We can also view detailed statistics of the Hypothesis run by passing the argument –hypothesis-show-statistics to pytest command as:

The difference between the reuse and generate phase in the output above is explained below:

• Reuse Phase: During the reuse phase, the Hypothesis attempts to reuse previously generated test data. If a test case fails or raises an exception, the Hypothesis will try to shrink the failing example to find a minimal failing case.

This phase typically has a very short runtime, as it involves reusing existing test data or shrinking failing examples. The output provides statistics about the typical runtimes and the number of passing, failing, and invalid examples encountered during this phase.

• Generate Phase: During the generate phase, the Hypothesis generates new test data based on the defined strategies. This phase involves generating a wide range of inputs to test the properties defined by the developer.

The output provides statistics about the typical runtimes and the number of passing, failing, and invalid examples generated during this phase. While this helped us understand what passing tests look like with a Hypothesis, it’s also worthwhile to understand how a Hypothesis can catch bugs in the code.

Let’s rewrite the factorial() function with an obvious bug, i.e., remove the check for when the input value is 0.

We also tweak the test to remove the min_value and max_value arguments.

Let us now rerun the test with the same command:

We can clearly see how Hypothesis has caught the bug immediately, which is shown in the above output. Hypothesis presents the input that resulted in the failing test under the Falsifying example section of the output.

So far, we’ve performed Hypothesis testing locally. This works nicely for unit tests , but when setting up automation for building more robust and resilient test suites, we can leverage a cloud grid like LambdaTest that supports automation testing tools like Selenium and Playwright.

LambdaTest is an AI-powered test orchestration and execution platform that enables developers and testers to perform automation testing with Selenium and Playwright at scale. It provides a remote test lab of 3000+ real environments.

How to Perform Hypothesis Testing in Python Using Cloud Selenium Grid?

Selenium is an open-source suite of tools and libraries for web automation . When combined with a cloud grid, it can help you perform Hypothesis testing in Python with Selenium at scale.

Let’s look at one test scenario to understand Hypothesis testing in Python with Selenium.

The code to set up a connection to LambdaTest Selenium Grid is stored in a crossbrowser_selenium.py file.

The test_selenium.py contains code to test the Hypothesis that tests will only run on the Firefox browser.

Let’s now understand the step-by-step code walkthrough for Hypothesis testing in Python using Selenium Grid.

Step 1: We import the necessary Selenium methods to initiate a connection to LambdaTest Selenium Grid.

The FirefoxOptions() method is used to configure the setup when connecting to LambdaTest Selenium Grid using Firefox.

Step 2: We use the load_dotenv package to access the LT_ACCESS_KEY required to access the LambdaTest Selenium Grid, which is stored in the form of environment variables.

The LT_ACCESS_KEY can be obtained from your LambdaTest Profile > Account Settings > Password & Security .

Step 3: We initialize the CrossBrowserSetup class, which prepares the remote connection URL using the username and access_key.

Step 4: The add() method is responsible for checking the browsertype and then setting the capabilities of the LambdaTest Selenium Grid.

LambdaTest offers a variety of capabilities, such as cross browser testing , which means we can test on various operating systems such as Windows, Linux, and macOS and multiple browsers such as Chrome, Firefox, Edge, and Safari.

For the purpose of this blog, we will be testing that connection to the LambdaTest Selenium Grid should only happen if the browsertype is Firefox.

Step 5: If the connection to LambdaTest happens, the add() returns True ; else, it returns False .

Let’s now understand a step-by-step walkthrough of the test_selenium.py file.

Step 1: We set up the imports of the given decorator and the Hypothesis strategy. We also import the CrossBrowserSetup class.

Step 2: @setting(deadline=None) ensures the test doesn’t timeout if the connection to the LambdaTest Grid takes more time.

We use the @given decorator to set the strategy to just use Firefox as an input to the test_add() argument broswertype_1. We then initialize an instance of the CrossBrowserSetup class & call the add() method using the broswertype_1 & assert if it returns True .

The commented strategy @given(strategy.just(‘Chrome’)) is to demonstrate that the add() method, when called with Chrome, returns False .

Let’s now run the test using pytest -k “test_hypothesis_selenium.py”.

We can see that the test has passed, and the Web Automation Dashboard reflects that the connection to the Selenium Grid has been successful.

On opening one of the execution runs, we can see a detailed step-by-step test execution.

How to Perform Hypothesis Testing in Python Using Cloud Playwright Grid?

Playwright is a popular open-source tool for end-to-end testing developed by Microsoft. When combined with a cloud grid, it can help you perform Hypothesis testing in Python at scale.

Let’s look at one test scenario to understand Hypothesis testing in Python with Playwright.

Let’s now understand the step-by-step code walkthrough for Hypothesis testing in Python using Playwright Grid.

Step 1: To connect to the LambdaTest Playwright Grid, we need a Username and Access Key, which can be obtained from the Profile page > Account Settings > Password & Security.

We use the python-dotenv module to load the Username and Access Key, which are stored as environment variables.

The capabilities dictionary is used to set up the Playwright Grid on LambdaTest.

We configure the Grid to use Windows 11 and the latest version of Chrome.

Step 3: The function interact_with_lambdatest interacts with the LambdaTest eCommerce Playground website to simulate adding a product to the cart and proceeding to checkout.

It starts a Playwright session and retrieves the version of the Playwright being used. The LambdaTest CDP URL is created with the appropriate capabilities. It connects to the Chromium browser instance on LambdaTest.

A new page instance is created, and the LambdaTest eCommerce Playground website is navigated. The specified product is added to the cart by clicking through the required buttons and links. The unit price of the product is extracted from the web page.

The browser page is then closed.

Step 4: We define a Hypothesis strategy quantity_strategy using st.integers to generate random integers representing product quantities. The generated integers range from 1 to 10

Using the @given decorator from the Hypothesis library, we define a property-based test function test_website_interaction that takes a quantity parameter generated by the quantity_strategy .

Inside the test function, we use the interact_with_lambdatest function to simulate interacting with the website and calculate the total price based on the generated quantity.

We assert that the total_price returned by interact_with_lambdatest matches the expected value calculated as quantity * 146.00.

Let’s now run the test on the Playwright Cloud Grid using pytest -v -k “test_hypothesis_playwright.py ”

The LambdaTest Web Automation Dashboard shows successfully passed tests.

Run Your Hypothesis Tests With Selenium & Playwright on Cloud. Try LambdaTest Today!

How to Perform Hypothesis Testing in Python With Date Strategy?

In the previous test scenario, we saw a simple example where we used the integer() strategy available as part of the Hypothesis. Let’s now understand another strategy, the date() strategy, which can be effectively used to test date-based functions.

Also, the output of the Hypothesis run can be customized to produce detailed results. Often, we may wish to see an even more verbose output when executing a Hypothesis test.

To do so, we have two options: either use the @settings decorator or use the –hypothesis-verbosity=<verbosity_level> when performing pytest testing .

Let’s now understand the code step-by-step.

Step 1: The function generate_expiry_alert() , which takes in an expiry_date as input and returns a boolean depending on whether the difference between the current date and expiry_date is less than or equal to 45 days.

Step 2: To ensure we test the generate_expiry_alert() for a wide range of date inputs, we use the date() strategy.

We also enable verbose logging and set the max_examples=1000 , which requests Hypothesis to generate 1000 date inputs at the max.

Step 3: On the inputs generated by Hypothesis in Step 3, we call the generate_expiry_alert() function and store the returned boolean in alert_generated.

We then compare the value returned by the function generate_expiry_alert() with a locally calculated copy and assert if the match.

We execute the test using the below command in the verbose mode, which allows us to see the test input dates generated by the Hypothesis.

As we can see, Hypothesis ran 1000 tests, 2 with reused data and 998 with unique newly generated data, and found no issues with the code.

Now, imagine the trouble we would have had to take to write 1000 tests manually using traditional example-based testing.

How to Perform Hypothesis Testing in Python With Composite Strategies?

So far, we’ve been using simple standalone examples to demo the power of Hypothesis. Let’s now move on to more complicated scenarios.

The implementation of the UserRewards class is stored in a user_rewards.py file for better readability.

The tests for the UserRewards class are stored in test_user_rewards.py .

Let’s now understand what is happening with both the class file and the test file step-by-step, starting first with the UserReward class.

Step 1: The class takes in a single argument initial_points to initialize the object.

Step 2: The get_reward_points() function returns the customers current reward points.

Step 3: The spend_reward_points() takes in the spent_points as input and returns True if spent_points are less than or equal to the customer current point balance and updates the customer reward_points by subtracting the spent_points , else it returns False .

That is it for our simple UserRewards class. Next, we understand what’s happening in the test_user_rewards.py step-by-step.

Step 1: We import the @given decorator and strategies from Hypothesis and the UserRewards class.

Step 2: Since reward points will always be integers, we use the integer() Hypothesis strategy to generate 1000 sample inputs starting with 0 and store them in a reward_points_strategy variable.

Step 3: Use the rewards_point_strategy as an input we run the test_get_reward_points() for 1000 samples starting with value 0.

For each input, we initialize the UserRewards class and assert that the method get_reward_points() returns the same value as the initial_points .

Step 4: To test the spend_reward_points() function, we generate two sets of sample inputs first, an initial reward_points using the reward_points_strategy we defined in Step 2 and a spend_amount which simulates spending of points.

Step 5: Write the test_spend_reward_points , which takes in the initial_points and spend_amount as arguments and initializes the UserRewards class with initial_point .

We also initialize a remaining_points variable to track the points remaining after the spend.

Step 6: If the spend_amount is less than the initial_points allocated to the customer, we assert if spend_reward_points returns True and update the remaining_points else, we assert spend_reward_points returns False .

Step 7: Lastly, we assert if the final remaining_points are correctly returned by get_rewards_points , which should be updated after spending the reward points.

Let’s now run the test and see if Hypothesis is able to find any bugs in the code.

To test if the Hypothesis indeed works, let’s make a small change to the UserRewards by commenting on the logic to deduct the spent_points in the spend_reward_points() function.

We run the test suite again using the command pytest -s –hypothesis-show-statistics -k “test_user_rewards “.

This time, the Hypothesis highlights the failures correctly.

Thus, we can catch any bugs and potential side effects of code changes early, making it perfect for unit testing and regression testing .

To understand composite strategies a bit more, let’s now test the shopping cart functionality and see how composite strategy can help write robust tests for even the most complicated of real-world scenarios.

Let’s view the implementation of the ShoppingCart class written in the shopping_cart.py file.

Let’s now view the tests written to verify the correct behavior of all aspects of the ShoppingCart class stored in a separate test_shopping_cart.py file.

Code Walkthrough of ShoppingCart class:

Let’s now understand what is happening in the ShoppingCart class step-by-step.

Step 1: We import the Python built-in Enum class and the auto() method.

The auto function within the Enum module automatically assigns sequential integer values to enumeration members, simplifying the process of defining enumerations with incremental values.

We define an Item enum corresponding to items available for sale on the LambdaTest eCommerce Playground website.

Step 2: We initialize the ShoppingCart class with an empty dictionary of items.

Step 3: The add_item() method takes in the item, price, and quantity as input and adds it to the shopping cart state held in the item dictionary.

Step 4: The remove_item() method takes in an item and quantity and removes it from the shopping cart state indicated by the item dictionary.

Step 5: The get_total_price() method iterates over the item dictionary, multiples the quantity by price, and returns the total_price of items in the cart.

Code Walkthrough of test_shopping_cart:

Let’s now understand step-by-step the tests written to ensure the correct working of the ShoppingCart class.

Step 1: First, we set up the imports, including the @given decorator, strategies, and the ShoppingCart class and Item enum.

The SearchStrategy is one of the various strategies on offer as part of the Hypothesis. It represents a set of rules for generating valid inputs to test a specific property or behavior of a function or program.

Step 2: We use the @st.composite decorator to define a custom Hypothesis strategy named items_strategy. This strategy takes a single argument, draw, which is a callable used to draw values from other strategies.

The st.sampled_from strategy randomly samples values from a given iterable. Within the strategy, we use draw(st.sampled_from(list(Item))) to draw a random Item instance from a list of all enum members.

Each time the items_strategy is used in a Hypothesis test, it will generate a random instance of the Item enum for testing purposes.

Step 3: The price_strategy runs on similar logic as the item_strategy but generates an integer value between 1 and 100.

Step 4: The qty_strategy runs on similar logic as the item_strategy but generates an integer value between 1 and 10.

Step 5: We use the @given decorator from the Hypothesis library to define a property-based test.

The items_strategy() , price_strategy() , and qty_strategy() functions are used to generate random values for the item, price, and quantity parameters, respectively.

Inside the test function, we create a new instance of a ShoppingCart .

We then add an item to the cart using the generated values for item, price, and quantity.

Finally, we assert that the item was successfully added to the cart and that the quantity matches the generated quantity.

Step 6: We use the @given decorator from the Hypothesis library to define a property-based test.

The items_strategy(), price_strategy() , and qty_strategy() functions are used to generate random values for the item, price, and quantity parameters, respectively.

Inside the test function, we create a new instance of a ShoppingCart . We then add the same item to the cart twice to simulate two quantity additions to the cart.

We remove one instance of the item from the cart. After that, we compare the item quantity before and after removal to ensure it decreases by 1.

The test verifies the behavior of the remove_item() method of the ShoppingCart class by testing it with randomly generated inputs for item, price , and quantity.

Step 7: We use the @given decorator from the Hypothesis library to define a property-based test.

The items_strategy(), price_strategy(), and qty_strategy() functions are used to generate random values for the item, price, and quantity parameters, respectively.

We add the same item to the cart twice to ensure it’s present, then remove one instance of the item from the cart. After that, we calculate the total price of items remaining in the cart.

Finally, we assert that the total price matches the price of one item times its remaining quantity.

The test verifies the correctness of the get_total_price() method of the ShoppingCart class by testing it with randomly generated inputs for item, price , and quantity .

Let’s now run the test using the command pytest –hypothesis-show-statistics -k “test_shopping_cart”.

We can verify that Hypothesis was able to find no issues with the ShoppingCart class.

Let’s now amend the price_strategy and qty_strategy to remove the min_value and max_value arguments.

And rerun the test pytest -k “test_shopping_cart” .

The tests run clearly reveal that we have bugs with respect to handling scenarios when quantity and price are passed as 0.

This also reveals the fact that setting the test inputs correctly to ensure we do comprehensive testing is key to writing robots and resilient tests.

Choosing min_val and max_val should only be done when we know beforehand the bounds of inputs the function under test will receive. If we are unsure what the inputs are, maybe it’s important to come up with the right strategies based on the behavior of the function under test.

In this blog we have seen in detail how Hypothesis testing in Python works using the popular Hypothesis library. Hypothesis testing falls under property-based testing and is much better than traditional testing in handling edge cases.

We also explored Hypothesis strategies and how we can use the @composite decorator to write custom strategies for testing complex functionalities.

We also saw how Hypothesis testing in Python can be performed with popular test automation frameworks like Selenium and Playwright. In addition, by performing Hypothesis testing in Python with LambdaTest on Cloud Grid, we can set up effective automation tests to enhance our confidence in the code we’ve written.

What are the three types of Hypothesis tests?

There are three main types of hypothesis tests based on the direction of the alternative hypothesis:

• Right-tailed test: This tests if a parameter is greater than a certain value.
• Left-tailed test: This tests if a parameter is less than a certain value.
• Two-tailed test: This tests for any non-directional difference, either greater or lesser than the hypothesized value.

What is Hypothesis testing in the ML model?

Hypothesis testing is a statistical approach used to evaluate the performance and validity of machine learning models. It helps us determine if a pattern observed in the training data likely holds true for unseen data (generalizability).

Jaydeep is a software engineer with 10 years of experience, most recently developing and supporting applications written in Python. He has extensive with shell scripting and is also an AI/ML enthusiast. He is also a tech educator, creating content on Twitter, YouTube, Instagram, and LinkedIn. Link to his YouTube channel- https://www.youtube.com/@jaydeepkarale

See author's profile

Author’s Profile

Got Questions? Drop them on LambdaTest Community. Visit now

Test Your Web Or Mobile Apps On 3000+ Browsers

Related Articles

How To Write Test Cases Effectively: Your Step-by-step Guide

September 5, 2024

Manual Testing | Automation |

How to Use Assert and Verify in Selenium

Faisal Khatri

September 2, 2024

Automation | Tutorial |

How to Handle File Upload in Selenium

August 29, 2024

Introducing KaneAI – World’s First End-to-End Testing Assistant

August 21, 2024

Automation | LambdaTest Updates |

ExpectedConditions In Selenium: Types And Examples

August 13, 2024

CSS Grid Best Practices: Guide With Examples

Tahera Alam

Web Development | LambdaTest Experiments | Tutorial |

Try LambdaTest Now !!

Get 100 minutes of automation test minutes FREE!!

Download Whitepaper

You'll get your download link by email.

Don't worry, we don't spam!

We use cookies to give you the best experience. Cookies help to provide a more personalized experience and relevant advertising for you, and web analytics for us. Learn More in our Cookies policy , Privacy & Terms of service .

Schedule Your Personal Demo ×

Test faster, fix more

How do I use pytest fixtures with Hypothesis?

pytest is a great test runner, and is the one Hypothesis itself uses for testing (though Hypothesis works fine with other test runners too).

It has a fairly elaborate fixture system , and people are often unsure how that interacts with Hypothesis. In this article we’ll go over the details of how to use the two together.

Mostly, Hypothesis and py.test fixtures don’t interact: Each just ignores the other’s presence.

When using a @given decorator, any arguments that are not provided in the @given will be left visible in the final function:

This then outputs the following:

We’ve hidden the arguments ‘a’ and ‘c’, but the unspecified arguments ‘b’ and ‘d’ are still left to be passed in. In particular, they can be provided as py.test fixtures:

This also works if we want to use @given with positional arguments:

The positional argument fills in from the right, replacing the ‘a’ argument and leaving us with ‘stuff’ to be provided by the fixture.

Personally I don’t usually do this because I find it gets a bit confusing - if I’m going to use fixtures then I always use the named variant of given. There’s no reason you can’t do it this way if you prefer though.

@given also works fine in combination with parametrized tests:

This will run 3 tests, one for each value for ‘stuff’.

There is one unfortunate feature of how this interaction works though: In pytest you can declare fixtures which do set up and tear down per function. These will “work” with Hypothesis, but they will run once for the entire test function rather than once for each time given calls your test function. So the following will fail:

The counter will not get reset at the beginning of each call to the test function, so it will be incremented each time and the test will start failing after the first call.

There currently aren’t any great ways around this unfortunately. The best you can really do is do manual setup and teardown yourself in your tests using Hypothesis (e.g. by implementing a version of your fixture as a context manager).

Long-term, I’d like to resolve this by providing a mechanism for allowing fixtures to be run for each example (it’s probably not correct to have every function scoped fixture run for each example), but for now it’s stalled because it requires changes on the py.test side as well as the Hypothesis side and we haven’t quite managed to find the time and place to collaborate on figuring out how to fix this yet.

• Quick start guide
• Edit on GitHub

Quick start guide ¶

This document should talk you through everything you need to get started with Hypothesis.

An example ¶

Suppose we’ve written a run length encoding system and we want to test it out.

We have the following code which I took straight from the Rosetta Code wiki (OK, I removed some commented out code and fixed the formatting, but there are no functional modifications):

We want to write a test for this that will check some invariant of these functions.

The invariant one tends to try when you’ve got this sort of encoding / decoding is that if you encode something and then decode it then you get the same value back.

Let’s see how you’d do that with Hypothesis:

(For this example we’ll just let pytest discover and run the test. We’ll cover other ways you could have run it later).

The text function returns what Hypothesis calls a search strategy. An object with methods that describe how to generate and simplify certain kinds of values. The @given decorator then takes our test function and turns it into a parametrized one which, when called, will run the test function over a wide range of matching data from that strategy.

Anyway, this test immediately finds a bug in the code:

Hypothesis correctly points out that this code is simply wrong if called on an empty string.

If we fix that by just adding the following code to the beginning of our encode function then Hypothesis tells us the code is correct (by doing nothing as you’d expect a passing test to).

If we wanted to make sure this example was always checked we could add it in explicitly by using the @example decorator:

This can be useful to show other developers (or your future self) what kinds of data are valid inputs, or to ensure that particular edge cases such as "" are tested every time. It’s also great for regression tests because although Hypothesis will remember failing examples , we don’t recommend distributing that database.

It’s also worth noting that both @example and @given support keyword arguments as well as positional. The following would have worked just as well:

Suppose we had a more interesting bug and forgot to reset the count each time. Say we missed a line in our encode method:

Hypothesis quickly informs us of the following example:

Note that the example provided is really quite simple. Hypothesis doesn’t just find any counter-example to your tests, it knows how to simplify the examples it finds to produce small easy to understand ones. In this case, two identical values are enough to set the count to a number different from one, followed by another distinct value which should have reset the count but in this case didn’t.

Installing ¶

Hypothesis is available on PyPI as “hypothesis” . You can install it with:

You can install the dependencies for optional extensions with e.g. pip install hypothesis[pandas,django] .

If you want to install directly from the source code (e.g. because you want to make changes and install the changed version), check out the instructions in CONTRIBUTING.rst .

Running tests ¶

In our example above we just let pytest discover and run our tests, but we could also have run it explicitly ourselves:

We could also have done this as a unittest.TestCase :

A detail: This works because Hypothesis ignores any arguments it hasn’t been told to provide (positional arguments start from the right), so the self argument to the test is simply ignored and works as normal. This also means that Hypothesis will play nicely with other ways of parameterizing tests. e.g it works fine if you use pytest fixtures for some arguments and Hypothesis for others.

Writing tests ¶

A test in Hypothesis consists of two parts: A function that looks like a normal test in your test framework of choice but with some additional arguments, and a @given decorator that specifies how to provide those arguments.

Here are some other examples of how you could use that:

Note that as we saw in the above example you can pass arguments to @given either as positional or as keywords.

Where to start ¶

You should now know enough of the basics to write some tests for your code using Hypothesis. The best way to learn is by doing, so go have a try.

If you’re stuck for ideas for how to use this sort of test for your code, here are some good starting points:

Try just calling functions with appropriate arbitrary data and see if they crash. You may be surprised how often this works. e.g. note that the first bug we found in the encoding example didn’t even get as far as our assertion: It crashed because it couldn’t handle the data we gave it, not because it did the wrong thing.

Look for duplication in your tests. Are there any cases where you’re testing the same thing with multiple different examples? Can you generalise that to a single test using Hypothesis?

This piece is designed for an F# implementation , but is still very good advice which you may find helps give you good ideas for using Hypothesis.

If you have any trouble getting started, don’t feel shy about asking for help .

Effective Python Testing With Pytest

Table of Contents

How to Install pytest

Less boilerplate, nicer output, less to learn, easier to manage state and dependencies, easy to filter tests, allows test parametrization, has a plugin-based architecture, when to create fixtures, when to avoid fixtures, how to use fixtures at scale, marks: categorizing tests, parametrization: combining tests, durations reports: fighting slow tests, pytest-randomly, pytest-django.

Watch Now This tutorial has a related video course created by the Real Python team. Watch it together with the written tutorial to deepen your understanding: Testing Your Code With pytest

Testing your code brings a wide variety of benefits. It increases your confidence that the code behaves as you expect and ensures that changes to your code won’t cause regressions. Writing and maintaining tests is hard work, so you should leverage all the tools at your disposal to make it as painless as possible. pytest is one of the best tools that you can use to boost your testing productivity.

In this tutorial, you’ll learn:

• What benefits pytest offers
• How to ensure your tests are stateless
• How to make repetitious tests more comprehensible
• How to run subsets of tests by name or custom groups
• How to create and maintain reusable testing utilities

Free Bonus: 5 Thoughts On Python Mastery , a free course for Python developers that shows you the roadmap and the mindset you’ll need to take your Python skills to the next level.

Take the Quiz: Test your knowledge with our interactive “Effective Testing with Pytest” quiz. You’ll receive a score upon completion to help you track your learning progress:

Interactive Quiz

In this quiz, you'll test your understanding of pytest, a Python testing tool. With this knowledge, you'll be able to write more efficient and effective tests, ensuring your code behaves as expected.

To follow along with some of the examples in this tutorial, you’ll need to install pytest . As most Python packages , pytest is available on PyPI . You can install it in a virtual environment using pip :

• Linux + macOS

The pytest command will now be available in your installation environment.

What Makes pytest So Useful?

If you’ve written unit tests for your Python code before, then you may have used Python’s built-in unittest module. unittest provides a solid base on which to build your test suite, but it has a few shortcomings.

A number of third-party testing frameworks attempt to address some of the issues with unittest , and pytest has proven to be one of the most popular. pytest is a feature-rich, plugin-based ecosystem for testing your Python code.

If you haven’t had the pleasure of using pytest yet, then you’re in for a treat! Its philosophy and features will make your testing experience more productive and enjoyable. With pytest , common tasks require less code and advanced tasks can be achieved through a variety of time-saving commands and plugins. It’ll even run your existing tests out of the box, including those written with unittest .

As with most frameworks, some development patterns that make sense when you first start using pytest can start causing pains as your test suite grows. This tutorial will help you understand some of the tools pytest provides to keep your testing efficient and effective even as it scales.

Most functional tests follow the Arrange-Act-Assert model:

• Arrange , or set up, the conditions for the test
• Act by calling some function or method
• Assert that some end condition is true

Testing frameworks typically hook into your test’s assertions so that they can provide information when an assertion fails. unittest , for example, provides a number of helpful assertion utilities out of the box. However, even a small set of tests requires a fair amount of boilerplate code .

Imagine you’d like to write a test suite just to make sure that unittest is working properly in your project. You might want to write one test that always passes and one that always fails:

You can then run those tests from the command line using the discover option of unittest :

As expected, one test passed and one failed. You’ve proven that unittest is working, but look at what you had to do:

• Import the TestCase class from unittest
• Create TryTesting , a subclass of TestCase
• Write a method in TryTesting for each test
• Use one of the self.assert* methods from unittest.TestCase to make assertions

That’s a significant amount of code to write, and because it’s the minimum you need for any test, you’d end up writing the same code over and over. pytest simplifies this workflow by allowing you to use normal functions and Python’s assert keyword directly:

That’s it. You don’t have to deal with any imports or classes. All you need to do is include a function with the test_ prefix. Because you can use the assert keyword, you don’t need to learn or remember all the different self.assert* methods in unittest , either. If you can write an expression that you expect to evaluate to True , and then pytest will test it for you.

Not only does pytest eliminate a lot of boilerplate, but it also provides you with a much more detailed and easy-to-read output.

You can run your test suite using the pytest command from the top-level folder of your project:

pytest presents the test results differently than unittest , and the test_with_unittest.py file was also automatically included. The report shows:

• The system state, including which versions of Python, pytest , and any plugins you have installed
• The rootdir , or the directory to search under for configuration and tests
• The number of tests the runner discovered

These items are presented in the first section of the output:

The output then indicates the status of each test using a syntax similar to unittest :

• A dot ( . ) means that the test passed.
• An F means that the test has failed.
• An E means that the test raised an unexpected exception.

The special characters are shown next to the name with the overall progress of the test suite shown on the right:

For tests that fail, the report gives a detailed breakdown of the failure. In the example, the tests failed because assert False always fails:

This extra output can come in extremely handy when debugging. Finally, the report gives an overall status report of the test suite:

When compared to unittest, the pytest output is much more informative and readable.

In the next section, you’ll take a closer look at how pytest takes advantage of the existing assert keyword.

Being able to use the assert keyword is also powerful. If you’ve used it before, then there’s nothing new to learn. Here are a few assertion examples so you can get an idea of the types of test you can make:

They look very much like normal Python functions. All of this makes the learning curve for pytest shallower than it is for unittest because you don’t need to learn new constructs to get started.

Note that each test is quite small and self-contained. This is common—you’ll see long function names and not a lot going on within a function. This serves mainly to keep your tests isolated from each other, so if something breaks, you know exactly where the problem is. A nice side effect is that the labeling is much better in the output.

To see an example of a project that creates a test suite along with the main project, check out the Build a Hash Table in Python With TDD tutorial. Additionally, you can work on Python practice problems to try test-driven development yourself while you get ready for your next interview or parse CSV files .

In the next section, you’re going to be examining fixtures, a great pytest feature to help you manage test input values.

Your tests will often depend on types of data or test doubles that mock objects your code is likely to encounter, such as dictionaries or JSON files.

With unittest , you might extract these dependencies into .setUp() and .tearDown() methods so that each test in the class can make use of them. Using these special methods is fine, but as your test classes get larger, you may inadvertently make the test’s dependence entirely implicit . In other words, by looking at one of the many tests in isolation, you may not immediately see that it depends on something else.

Over time, implicit dependencies can lead to a complex tangle of code that you have to unwind to make sense of your tests. Tests should help to make your code more understandable. If the tests themselves are difficult to understand, then you may be in trouble!

pytest takes a different approach. It leads you toward explicit dependency declarations that are still reusable thanks to the availability of fixtures . pytest fixtures are functions that can create data, test doubles, or initialize system state for the test suite. Any test that wants to use a fixture must explicitly use this fixture function as an argument to the test function, so dependencies are always stated up front:

Looking at the test function, you can immediately tell that it depends on a fixture, without needing to check the whole file for fixture definitions.

Note: You usually want to put your tests into their own folder called tests at the root level of your project.

For more information about structuring a Python application, check out the video course on that very topic.

Fixtures can also make use of other fixtures, again by declaring them explicitly as dependencies. That means that, over time, your fixtures can become bulky and modular. Although the ability to insert fixtures into other fixtures provides enormous flexibility, it can also make managing dependencies more challenging as your test suite grows.

Later in this tutorial, you’ll learn more about fixtures and try a few techniques for handling these challenges.

As your test suite grows, you may find that you want to run just a few tests on a feature and save the full suite for later. pytest provides a few ways of doing this:

• Name-based filtering : You can limit pytest to running only those tests whose fully qualified names match a particular expression. You can do this with the -k parameter.
• Directory scoping : By default, pytest will run only those tests that are in or under the current directory.
• Test categorization : pytest can include or exclude tests from particular categories that you define. You can do this with the -m parameter.

Test categorization in particular is a subtly powerful tool. pytest enables you to create marks , or custom labels, for any test you like. A test may have multiple labels, and you can use them for granular control over which tests to run. Later in this tutorial, you’ll see an example of how pytest marks work and learn how to make use of them in a large test suite.

When you’re testing functions that process data or perform generic transformations, you’ll find yourself writing many similar tests. They may differ only in the input or output of the code being tested. This requires duplicating test code, and doing so can sometimes obscure the behavior that you’re trying to test.

unittest offers a way of collecting several tests into one, but they don’t show up as individual tests in result reports. If one test fails and the rest pass, then the entire group will still return a single failing result. pytest offers its own solution in which each test can pass or fail independently. You’ll see how to parametrize tests with pytest later in this tutorial.

One of the most beautiful features of pytest is its openness to customization and new features. Almost every piece of the program can be cracked open and changed. As a result, pytest users have developed a rich ecosystem of helpful plugins.

Although some pytest plugins focus on specific frameworks like Django , others are applicable to most test suites. You’ll see details on some specific plugins later in this tutorial.

Fixtures: Managing State and Dependencies

pytest fixtures are a way of providing data, test doubles, or state setup to your tests. Fixtures are functions that can return a wide range of values. Each test that depends on a fixture must explicitly accept that fixture as an argument.

In this section, you’ll simulate a typical test-driven development (TDD) workflow.

Imagine you’re writing a function, format_data_for_display() , to process the data returned by an API endpoint. The data represents a list of people, each with a given name, family name, and job title. The function should output a list of strings that include each person’s full name (their given_name followed by their family_name ), a colon, and their title :

In good TDD fashion, you’ll want to first write a test for it. You might write the following code for that:

While writing this test, it occurs to you that you may need to write another function to transform the data into comma-separated values for use in Excel :

Your to-do list grows! That’s good! One of the advantages of TDD is that it helps you plan out the work ahead. The test for the format_data_for_excel() function would look awfully similar to the format_data_for_display() function:

Notably, both the tests have to repeat the definition of the people variable, which is quite a few lines of code.

If you find yourself writing several tests that all make use of the same underlying test data, then a fixture may be in your future. You can pull the repeated data into a single function decorated with @pytest.fixture to indicate that the function is a pytest fixture:

You can use the fixture by adding the function reference as an argument to your tests. Note that you don’t call the fixture function. pytest takes care of that. You’ll be able to use the return value of the fixture function as the name of the fixture function:

Each test is now notably shorter but still has a clear path back to the data it depends on. Be sure to name your fixture something specific. That way, you can quickly determine if you want to use it when writing new tests in the future!

When you first discover the power of fixtures, it can be tempting to use them all the time, but as with all things, there’s a balance to be maintained.

Fixtures are great for extracting data or objects that you use across multiple tests. However, they aren’t always as good for tests that require slight variations in the data. Littering your test suite with fixtures is no better than littering it with plain data or objects. It might even be worse because of the added layer of indirection.

As with most abstractions, it takes some practice and thought to find the right level of fixture use.

Nevertheless, fixtures will likely be an integral part of your test suite. As your project grows in scope, the challenge of scale starts to come into the picture. One of the challenges facing any kind of tool is how it handles being used at scale, and luckily, pytest has a bunch of useful features that can help you manage the complexity that comes with growth.

As you extract more fixtures from your tests, you might see that some fixtures could benefit from further abstraction. In pytest , fixtures are modular . Being modular means that fixtures can be imported , can import other modules, and they can depend on and import other fixtures. All this allows you to compose a suitable fixture abstraction for your use case.

For example, you may find that fixtures in two separate files, or modules , share a common dependency. In this case, you can move fixtures from test modules into more general fixture-related modules. That way, you can import them back into any test modules that need them. This is a good approach when you find yourself using a fixture repeatedly throughout your project.

If you want to make a fixture available for your whole project without having to import it, a special configuration module called conftest.py will allow you to do that.

pytest looks for a conftest.py module in each directory. If you add your general-purpose fixtures to the conftest.py module, then you’ll be able to use that fixture throughout the module’s parent directory and in any subdirectories without having to import it. This is a great place to put your most widely used fixtures.

Another interesting use case for fixtures and conftest.py is in guarding access to resources. Imagine that you’ve written a test suite for code that deals with API calls . You want to ensure that the test suite doesn’t make any real network calls even if someone accidentally writes a test that does so.

pytest provides a monkeypatch fixture to replace values and behaviors, which you can use to great effect:

By placing disable_network_calls() in conftest.py and adding the autouse=True option, you ensure that network calls will be disabled in every test across the suite. Any test that executes code calling requests.get() will raise a RuntimeError indicating that an unexpected network call would have occurred.

Your test suite is growing in numbers, which gives you a great feeling of confidence to make changes and not break things unexpectedly. That said, as your test suite grows, it might start taking a long time. Even if it doesn’t take that long, perhaps you’re focusing on some core behavior that trickles down and breaks most tests. In these cases, you might want to limit the test runner to only a certain category of tests.

In any large test suite, it would be nice to avoid running all the tests when you’re trying to iterate quickly on a new feature. Apart from the default behavior of pytest to run all tests in the current working directory, or the filtering functionality, you can take advantage of markers .

pytest enables you to define categories for your tests and provides options for including or excluding categories when you run your suite. You can mark a test with any number of categories.

Marking tests is useful for categorizing tests by subsystem or dependencies. If some of your tests require access to a database, for example, then you could create a @pytest.mark.database_access mark for them.

Pro tip : Because you can give your marks any name you want, it can be easy to mistype or misremember the name of a mark. pytest will warn you about marks that it doesn’t recognize in the test output.

You can use the --strict-markers flag to the pytest command to ensure that all marks in your tests are registered in your pytest configuration file, pytest.ini . It’ll prevent you from running your tests until you register any unknown marks.

For more information on registering marks, check out the pytest documentation .

When the time comes to run your tests, you can still run them all by default with the pytest command. If you’d like to run only those tests that require database access, then you can use pytest -m database_access . To run all tests except those that require database access, you can use pytest -m "not database_access" . You can even use an autouse fixture to limit database access to those tests marked with database_access .

Some plugins expand on the functionality of marks by adding their own guards. The pytest-django plugin, for instance, provides a django_db mark. Any tests without this mark that try to access the database will fail. The first test that tries to access the database will trigger the creation of Django’s test database.

The requirement that you add the django_db mark nudges you toward stating your dependencies explicitly. That’s the pytest philosophy, after all! It also means that you can much more quickly run tests that don’t rely on the database, because pytest -m "not django_db" will prevent the test from triggering database creation. The time savings really add up, especially if you’re diligent about running your tests frequently.

pytest provides a few marks out of the box:

• skip skips a test unconditionally.
• skipif skips a test if the expression passed to it evaluates to True .
• xfail indicates that a test is expected to fail, so if the test does fail, the overall suite can still result in a passing status.
• parametrize creates multiple variants of a test with different values as arguments. You’ll learn more about this mark shortly.

You can see a list of all the marks that pytest knows about by running pytest --markers .

On the topic of parametrization, that’s coming up next.

You saw earlier in this tutorial how pytest fixtures can be used to reduce code duplication by extracting common dependencies. Fixtures aren’t quite as useful when you have several tests with slightly different inputs and expected outputs. In these cases, you can parametrize a single test definition, and pytest will create variants of the test for you with the parameters you specify.

Imagine you’ve written a function to tell if a string is a palindrome . An initial set of tests could look like this:

All of these tests except the last two have the same shape:

This is starting to smell a lot like boilerplate. pytest so far has helped you get rid of boilerplate, and it’s not about to let you down now. You can use @pytest.mark.parametrize() to fill in this shape with different values, reducing your test code significantly:

The first argument to parametrize() is a comma-delimited string of parameter names. You don’t have to provide more than one name, as you can see in this example. The second argument is a list of either tuples or single values that represent the parameter value(s). You could take your parametrization a step further to combine all your tests into one:

Even though this shortened your code, it’s important to note that in this case you actually lost some of the more descriptive nature of the original functions. Make sure you’re not parametrizing your test suite into incomprehensibility. You can use parametrization to separate the test data from the test behavior so that it’s clear what the test is testing, and also to make the different test cases easier to read and maintain.

Each time you switch contexts from implementation code to test code, you incur some overhead . If your tests are slow to begin with, then overhead can cause friction and frustration.

You read earlier about using marks to filter out slow tests when you run your suite, but at some point you’re going to need to run them. If you want to improve the speed of your tests, then it’s useful to know which tests might offer the biggest improvements. pytest can automatically record test durations for you and report the top offenders.

Use the --durations option to the pytest command to include a duration report in your test results. --durations expects an integer value n and will report the slowest n number of tests. A new section will be included in your test report:

Each test that shows up in the durations report is a good candidate to speed up because it takes an above-average amount of the total testing time. Note that short durations are hidden by default. As spelled out in the report, you can increase the report verbosity and show these by passing -vv together with --durations .

Be aware that some tests may have an invisible setup overhead. You read earlier about how the first test marked with django_db will trigger the creation of the Django test database. The durations report reflects the time it takes to set up the database in the test that triggered the database creation, which can be misleading.

You’re well on your way to full test coverage. Next, you’ll be taking a look at some of the plugins that are part of the rich pytest plugin ecosystem.

Useful pytest Plugins

You learned about a few valuable pytest plugins earlier in this tutorial. In this section, you’ll be exploring those and a few others in more depth—everything from utility plugins like pytest-randomly to library-specific ones, like those for Django.

Often the order of your tests is unimportant, but as your codebase grows, you may inadvertently introduce some side effects that could cause some tests to fail if they were run out of order.

pytest-randomly forces your tests to run in a random order. pytest always collects all the tests it can find before running them. pytest-randomly just shuffles that list of tests before execution.

This is a great way to uncover tests that depend on running in a specific order, which means they have a stateful dependency on some other test. If you built your test suite from scratch in pytest , then this isn’t very likely. It’s more likely to happen in test suites that you migrate to pytest .

The plugin will print a seed value in the configuration description. You can use that value to run the tests in the same order as you try to fix the issue.

If you want to measure how well your tests cover your implementation code, then you can use the coverage package. pytest-cov integrates coverage, so you can run pytest --cov to see the test coverage report and boast about it on your project front page.

pytest-django provides a handful of useful fixtures and marks for dealing with Django tests. You saw the django_db mark earlier in this tutorial. The rf fixture provides direct access to an instance of Django’s RequestFactory . The settings fixture provides a quick way to set or override Django settings. These plugins are a great boost to your Django testing productivity!

If you’re interested in learning more about using pytest with Django, then check out How to Provide Test Fixtures for Django Models in Pytest .

pytest can be used to run tests that fall outside the traditional scope of unit testing. Behavior-driven development (BDD) encourages writing plain-language descriptions of likely user actions and expectations, which you can then use to determine whether to implement a given feature. pytest-bdd helps you use Gherkin to write feature tests for your code.

You can see which other plugins are available for pytest with this extensive list of third-party plugins .

pytest offers a core set of productivity features to filter and optimize your tests along with a flexible plugin system that extends its value even further. Whether you have a huge legacy unittest suite or you’re starting a new project from scratch, pytest has something to offer you.

In this tutorial, you learned how to use:

• Fixtures for handling test dependencies, state, and reusable functionality
• Marks for categorizing tests and limiting access to external resources
• Parametrization for reducing duplicated code between tests
• Durations to identify your slowest tests
• Plugins for integrating with other frameworks and testing tools

Install pytest and give it a try. You’ll be glad you did. Happy testing!

If you’re looking for an example project built with pytest , then check out the tutorial on building a hash table with TDD , which will not only get you up to speed with pytest , but also help you master hash tables!

🐍 Python Tricks 💌

Get a short & sweet Python Trick delivered to your inbox every couple of days. No spam ever. Unsubscribe any time. Curated by the Real Python team.

About Dane Hillard

Dane is a Technical Architect at ITHAKA and is currently writing Publishing Python Packages.

Each tutorial at Real Python is created by a team of developers so that it meets our high quality standards. The team members who worked on this tutorial are:

Master Real-World Python Skills With Unlimited Access to Real Python

Join us and get access to thousands of tutorials, hands-on video courses, and a community of expert Pythonistas:

Join us and get access to thousands of tutorials, hands-on video courses, and a community of expert Pythonistas:

What Do You Think?

What’s your #1 takeaway or favorite thing you learned? How are you going to put your newfound skills to use? Leave a comment below and let us know.

Commenting Tips: The most useful comments are those written with the goal of learning from or helping out other students. Get tips for asking good questions and get answers to common questions in our support portal . Looking for a real-time conversation? Visit the Real Python Community Chat or join the next “Office Hours” Live Q&A Session . Happy Pythoning!

Keep Learning

Related Topics: intermediate testing

Recommended Video Course: Testing Your Code With pytest

Keep reading Real Python by creating a free account or signing in:

Already have an account? Sign-In

Almost there! Complete this form and click the button below to gain instant access:

5 Thoughts On Python Mastery

🔒 No spam. We take your privacy seriously.

Testing in Python with Pytest

Ensuring the reliability and functionality of your code is crucial. This is where testing comes into play, serving as a safety net that helps catch bugs and verify that your application behaves as expected.

We just published a pytest course on the freeCodeCamp.org YouTube channel. Pytest is one of the most popular testing frameworks for Python and this course teaches the process of writing effective and efficient test cases.

Farhan Ali cerated this course. He is a freelance web developer who specializes in both design and fullstack web development.

What is pytest? Pytest is a robust testing framework for Python that makes it easier to write simple and scalable test cases. With its simple syntax, Pytest allows developers to get started quickly with minimal boilerplate code. It supports fixtures, parametrization, and numerous plugins, making it a versatile and powerful tool for writing and organizing test cases.

Here are the sections in this course:

• Our First Tests: The first section of the course introduces the basics of writing tests using pytest. Viewers will learn how to set up their testing environment, create simple test functions, and run tests to validate their code.
• Class-based Tests: Moving on, the course dives into class-based tests, which provide a structured way to organize test cases. This section will cover the use of test classes, setup and teardown methods, and the advantages of grouping related tests together.
• Fixtures: One of pytest's most powerful features is fixtures. This section explains how to use fixtures to set up preconditions for tests, manage resources, and ensure consistent test environments. By mastering fixtures, developers can write more maintainable and reusable test code.
• Mark & Parametrize: This part of the course explores the use of marks and parametrization in pytest. Marks allow for easy filtering and categorization of tests, while parametrization helps in running the same test with different inputs, ensuring comprehensive test coverage.
• Mocking: Mocking is an essential technique in testing, especially when dealing with external dependencies. This section will teach viewers how to use mocking to simulate external services, isolate test cases, and validate interactions between different parts of the codebase.
• Testing with ChatGPT: The final section of the course demonstrates the innovative integration of ChatGPT in the pytest testing framework. ChatGPT, developed by OpenAI, is a powerful language model that can generate human-like text based on given prompts. By leveraging ChatGPT, developers can create more dynamic and intelligent test cases, ultimately enhancing the quality and effectiveness of their test suites.

Pytest's flexibility, combined with the course's comprehensive content, will empower developers to write efficient and effective test cases, ensuring their code is robust and reliable.

Watch the full course on the freeCodeCamp.org YouTube channel (2-hour watch).

I'm a teacher and developer with freeCodeCamp.org. I run the freeCodeCamp.org YouTube channel.

If this article was helpful, share it .

Learn to code for free. freeCodeCamp's open source curriculum has helped more than 40,000 people get jobs as developers. Get started

The Python IDE for data science and web development

• Twitter Twitter
• Youtube Youtube

Pytest vs. Unittest: Which Is Better?

Python, being a versatile and widely used programming language, offers several testing frameworks to facilitate the testing process. Two prominent choices are pytest and unittest, both of which come with their own sets of features and advantages.

In this article, we’ll be covering the following sections:

• Introduction to each Python testing framework.
• Syntax and coding examples to give you a starting point.
• Advantages and disadvantages.
• A thorough comparison between the two testing frameworks.
• And finally, all the features that PyCharm provides for pytest and unittest.

This will help you determine which testing framework is the best for your needs.

Let’s get started.

Python Testing Frameworks

Python offers a variety of testing frameworks that cater to different needs and preferences. These frameworks facilitate the process of writing, organizing, and executing tests, ensuring the reliability and correctness of your code.

Two of the most popular Python testing frameworks are pytest and unittest. 

What is pytest?

Pytest is a popular and powerful testing framework for Python that simplifies the process of writing and executing tests. It is known for its simplicity, scalability, and ability to handle complex test scenarios while significantly reducing the boilerplate code required for testing.

In recent years, pytest has become one of the most popular choices for Python testing .

Features of pytest

• Fixture support: pytest provides a powerful fixture mechanism for setting up and tearing down resources needed for testing. This enhances test organization and readability.
• Parameterization: pytest allows easy parameterization of test functions, enabling the testing of multiple inputs without duplicating code. This enhances test coverage and maintains clean test code.
• Rich plugin architecture: pytest boasts a rich set of features and a vibrant plugin ecosystem. This extensibility allows customization and integration with other tools, making it adaptable to diverse testing needs.
• Concise syntax: The syntax for writing tests in pytest is concise and readable, making it easy for developers to adopt. This simplicity is conducive to rapid test development and maintenance.

How to write tests with pytest

Before writing tests with pytest, make sure you have it installed first. 

You can use Python Packages to install pytest in PyCharm:

Or you can also open your PyCharm terminal and write: 

Now, let’s take a simple example in which you’re creating an automated quiz.

In this program, we ask the user three questions and then return the number of answers they got correct. You can clone this exact project from this git repository.

We’ll be using this same example to create both pytest and unittest tests so you can see the difference in how they are structured.

Here’s how to write the test in pytest:

Overall, we have four test functions that test different cases. We are using mocking here to simulate different values a user can enter.

One important thing to note is that all test functions need to start with ‘test’ in pytest. If that’s not the case, then the pytest framework will fail to recognize it as a test function and won’t run it.

You can run each test separately or all of them together. 

Here’s the result:

Advantages and disadvantages of pytest

In this section, we will delve into the advantages of pytest, highlighting its strengths in areas apart from concise syntax, powerful fixtures, parameterization, and rich plugin support. Additionally, we’ll explore the potential drawbacks, acknowledging factors like a moderate learning curve for beginners, integration considerations, and nuances related to test discovery.

Advantages of pytest

1. Test discovery: pytest features automatic test discovery, eliminating the need for explicit configuration. This simplifies the testing process by identifying and executing test cases without manual intervention.

2. Skip/xfail tests: pytest provides the ability to skip or mark tests as expected failures using markers (@pytest.mark.skip and @pytest.mark.xfail). This flexibility is useful for excluding certain tests under specific conditions or marking tests that are known to fail.

3. Powerful assertion introspection: pytest provides detailed information on test failures, making it easier to diagnose and fix issues. The output includes clear and informative messages that help developers identify the exact cause of a failure.

4. Parallel test execution: pytest supports parallel test execution, allowing multiple tests to run concurrently. This feature can significantly reduce the overall test execution time, making it suitable for projects with large test suites.

5. Mocking and patching: pytest simplifies mocking and patching with built-in fixtures and libraries. This makes it easy to isolate components during testing and replace dependencies with controlled mocks.

6. Community support: pytest has a large and active community, leading to continuous development, regular updates, and a wealth of community-contributed plugins and resources. This vibrant community ensures that developers have access to extensive documentation and support.

7. Test isolation: pytest lets you run tests independently of each other, ensuring that the outcome of one test does not affect the outcome of another. This promotes cleaner, more robust, and maintainable test suites by ensuring that tests are reliable and easy to understand and debug.

These advantages collectively contribute to pytest’s popularity and effectiveness in various testing scenarios. Its flexibility, readability, and extensive feature set make it a valuable choice for both beginners and experienced developers alike.

While pytest is a powerful and widely adopted Python testing framework, like any tool, it has some potential disadvantages that users may encounter. It’s essential to consider these drawbacks when deciding whether pytest is the right choice for a particular project. 

Here are some potential disadvantages of pytest.

Disadvantages of pytest

1. Not in the Python Standard Library: Unlike unittest, which is part of the Python Standard Library, pytest is a third-party library. This means that for projects relying heavily on the Python Standard Library, there might be an additional step in installing and managing pytest.

2. Learning curve for beginners: For new developers, there might be a learning curve, especially if they are accustomed to other testing frameworks or if they are new to testing in general. Understanding and leveraging advanced features may take some time.

3. Integration with certain IDEs: While pytest integrates well with many IDEs, there can be occasional challenges in setting up the integration with some. Although this is not a widespread issue, it may require additional configuration in some cases.

4. Parallel test execution configuration: While pytest supports parallel test execution, configuring it for optimal performance might require additional setup and consideration. Developers need to understand and configure parallel execution settings correctly to avoid unexpected behaviors.

5. Slower test execution in some cases: In certain scenarios, pytest may exhibit slightly slower test execution compared to some other testing frameworks. This can be attributed to the additional features and flexibility provided by pytest, which may introduce some overhead.

6. Limited built-in fixtures for unittest-style setup/teardown: For developers accustomed to unittest-style setup and teardown using the setUp and tearDown methods, pytest might seem different. pytest encourages the use of fixtures, and while powerful, it might feel less intuitive for those transitioning from unittest.

7. Strictness in test discovery: pytest’s test discovery can be less strict compared to unittest in some cases. While this flexibility is an advantage for many, it might lead to unintended consequences if not carefully managed, especially in large and complex projects.

It’s important to note that many of these disadvantages are context-dependent, and what might be a drawback in one scenario could be an advantage in another.

Understanding the specific needs of a project and the preferences of the development team is crucial when evaluating whether pytest is the right testing framework for a given situation.

What Is unittest?

Unittest is a built-in testing framework in Python that follows the xUnit style and is inspired by Java’s JUnit. Since it’s part of the Python Standard Library, unittest provides a solid foundation for writing and organizing tests.

Features of unittest

• Test discovery: unittest automatically discovers and runs test cases without the need for explicit configuration. This simplifies test management and ensures all relevant tests are executed.
• Fixture support: unittest supports setup and teardown methods for test fixtures. While not as flexible as pytest’s fixtures, unittest’s fixtures provide the necessary functionality for resource management.
• Assertions: unittest comes with a variety of assertion methods for verifying expected outcomes. This includes methods like ‘assertEqual’, ‘assertTrue’, and more, ensuring comprehensive testing.
• Test suites: unittest allows grouping tests into test suites for better organization. This is beneficial for managing large codebases with numerous test cases.

How to write tests with unittest

We’ll be using the same quiz program example as before. This time, let’s test it with unittest.

Notice that in unittest, we don’t directly define our test functions. You need to define a test class that will contain all your test functions. In this case, it’s  class Test (unittest.TestCase). All unittest test classes need to be derived from unittest.TestCase, as this is the only way that the test functions will be invoked by the unittest framework.

Just like pytest, test functions in unittest also needs to start with ‘test’ in order for the framework to recognize and run it.

Here’s how to run the unittests individually or altogether in PyCharm:

Notice the difference between how unittest displays the test result vs how pytest does it. As you can see, unittest’s output is more summarized, whereas pytest displays test results for every individual test. 

Advantages and disadvantages of unittest

Unittest, the built-in testing framework in Python, offers a range of advantages that make it a solid choice for many developers and projects.

Here are some key advantages of unittest:

Advantages of unittest

• Community and industry standard: As a part of the xUnit family, unittest adheres to industry standards for unit testing frameworks. Developers familiar with xUnit-style frameworks in other languages will find unittest conventions familiar and consistent.
• Part of the Python Standard Library: unittest comes bundled with the Python Standard Library, ensuring its availability without the need for additional installations. This standardization promotes consistency across Python projects.
• Widely adopted and well-documented: As part of the Python Standard Library, unittest is widely adopted and well-documented. This makes it a familiar choice for developers transitioning from other languages with xUnit-style testing frameworks.
• Integration with IDEs: unittest integrates seamlessly with many integrated development environments (IDEs), providing features like test discovery, execution, and result visualization. This integration streamlines the testing workflow for developers using these tools.
• Standardized test output: unittest produces standardized test output, making it easy to integrate with other tools or continuous integration systems. The consistent output format enhances interoperability and ease of use.
• Consistent test isolation: unittest ensures consistent test isolation by creating a new instance of the test class for each test method. This prevents unintended side effects between different test cases.

While unittest may not have some of the advanced features found in third-party testing frameworks like pytest, its simplicity, standardization, and widespread adoption make it a reliable choice for many Python projects, particularly those where compatibility with the Python Standard Library is a priority.

However, there are some potential disadvantages to it, as well. It’s important to consider these limitations when deciding whether unittest is the most suitable choice for a particular project.

Disadvantages of unittest

1. Verbose syntax: One of the commonly mentioned criticisms of unittest is its verbose syntax. Compared to some other testing frameworks like pytest, writing test cases with unittest can require more lines of code, leading to a potential increase in boilerplate.

2. Limited parameterization support: unittest does not have native support for parameterized tests. 

3. Discovery requires rest prefix: By default, unittest requires test methods to be named with the prefix “test” for automatic discovery. While this convention is widely adopted, it can be restrictive if you want more flexibility in naming your test methods.

4. Learning curve for advanced features : While the basics of unittest are relatively easy to grasp, some of the more advanced features, such as creating custom test loaders or test runners, may have a more challenging learning curve.

5. Limited support for parallel test execution: unittest has limited built-in support for parallel test execution. Achieving parallelization may require additional effort and the use of external tools or libraries.

Despite these disadvantages, unittest remains a robust and reliable testing framework, particularly for projects that prioritize compatibility with the Python Standard Library and maintainability over advanced features provided by third-party frameworks.

Differences between pytest and unittest

The following table provides an overview of some key features and characteristics of both pytest and unittest.

Keep in mind that the suitability of a testing framework often depends on specific project requirements and team preferences.

Each framework has its strengths, and the choice between pytest and unittest should align with the testing needs and development philosophy of your project.

That said, for new projects with modern Python, pytest has become the default choice.

Pytest and unittest with PyCharm

Pytest in pycharm.

1. Intelligent code completion

PyCharm enhances the pytest experience with its intelligent code completion feature. As you write your test cases, PyCharm provides context-aware suggestions, making it easier to explore available fixtures, functions, and other elements related to pytest. This not only improves development speed but also helps you avoid typos and potential errors.

2. Test discovery and navigation

One of the key features of PyCharm is its robust support for test discovery. PyCharm automatically identifies and lists all your pytest tests, making it effortless to navigate through your test suite. You can quickly jump to the test definition, explore test fixtures, and view test outcomes directly from the editor. This streamlined workflow is particularly beneficial in large projects with extensive test suites. PyCharm also makes it very easy to run just one test, one file, or one directory, without knowing command line flags.

3. Integrated debugging

PyCharm provides seamless integration with pytest’s debugging capabilities. You can set breakpoints within your test code, run tests in debug mode, and step through the code to identify issues. The debugger interface in PyCharm offers a visual representation of the call stack, variable values, and breakpoints, facilitating efficient debugging and issue resolution.

4. Test result visualization

PyCharm presents test results in a clear and visually appealing manner. After running your pytest suite, you can easily navigate through the results, identify passed and failed tests, and view detailed information about each test case. This visual feedback helps developers quickly assess the health of their codebase and address any failures promptly.

5. Integration with version control systems

PyCharm seamlessly integrates with version control systems, ensuring that changes to both code and pytest tests are tracked consistently. This integration facilitates collaboration within development teams and helps maintain a unified versioning approach.

Unittest in PyCharm

1. Test discovery and execution

PyCharm provides robust support for discovering and executing unittest test cases. The IDE automatically identifies and lists all unittest tests in your project, simplifying test management. With just a few clicks, you can run individual test cases, entire test modules, or the entire test suite, ensuring a smooth and efficient testing process.

2. Code navigation and refactoring

PyCharm enhances the navigation and refactoring capabilities for unittest. Developers can easily navigate between test cases and corresponding code, making it straightforward to understand the relationship between tests and the tested code. Additionally, PyCharm supports various refactoring operations, allowing developers to modify their codebase confidently while maintaining the integrity of their test suites.

3. Code coverage analysis

PyCharm features a code coverage analysis tool that helps developers assess the extent to which their code is covered by unittests. This feature is valuable for identifying areas that may require additional testing, ensuring comprehensive test coverage.

4. Test configuration

PyCharm simplifies the configuration of unittest by providing an intuitive interface for adjusting testing parameters. Developers can customize test run configurations, specify test environments, and control other testing-related settings. This flexibility ensures that developers can adapt unittest to meet the specific requirements of their projects.

By using PyCharm, developers can seamlessly integrate both pytest and unittest into their workflow. PyCharm testing provides features like test discovery, execution, and result visualization for both frameworks.

How to choose between pytest vs. unittest?

Choosing between pytest and unittest depends on various factors, including project requirements, team preferences, and the complexity of testing scenarios. pytest is favored for its concise syntax, rich feature set, and flexibility, making it suitable for projects with diverse testing needs. 

On the other hand, unittest, being part of the standard library, provides simplicity and a consistent testing structure, making it a solid choice for projects that prioritize standardization and simplicity.

Ultimately, both pytest and unittest are capable testing frameworks, and the choice between them should align with your specific needs and development philosophy. Remember that the right testing framework can significantly contribute to the success of your project by increasing code quality and reliability.

Subscribe to PyCharm Blog updates

By submitting this form, I agree that JetBrains s.r.o. ("JetBrains") may use my name, email address, and location data to send me newsletters, including commercial communications, and to process my personal data for this purpose. I agree that JetBrains may process said data using third-party services for this purpose in accordance with the JetBrains Privacy Policy . I understand that I can revoke this consent at any time in my profile . In addition, an unsubscribe link is included in each email.

Thanks, we've got you!

Discover more

Learning Resources for pytest

Discover a trove of resources to help you master pytest, including guides from JetBrains and the community. Learn how PyCharm helps you when you’re working with pytest.

Three pytest Features You Will Love

Exploring better ways to test in Python? Find out how pytest can simplify your testing strategy in our comprehensive blog post.

• Stack Overflow for Teams Where developers & technologists share private knowledge with coworkers
• Advertising & Talent Reach devs & technologists worldwide about your product, service or employer brand
• OverflowAI GenAI features for Teams
• OverflowAPI Train & fine-tune LLMs
• Labs The future of collective knowledge sharing
• About the company Visit the blog

Collectives™ on Stack Overflow

Find centralized, trusted content and collaborate around the technologies you use most.

Q&A for work

Connect and share knowledge within a single location that is structured and easy to search.

Get early access and see previews of new features.

Multiple strategies for same function parameter in python hypothesis

I am writing a simple test code in Python using the Hypothesis package. It there a way to use multiple strategies for the same function parameter? As an example, use integers() and floats() to test my values parameter without writing two separate test functions?

The above code doesn't work, but it represents what i would like to achieve.

NB: I have already tried the trivial solution of creating two different tests with two different strategies:

However I was wondering if there was a nicer way to do this: when the nummber of strategies becomes larger, it is quite redundant as code.

• python-hypothesis

• 2 so your values can be either integers() or floats()? if so you can simply do @given(integers() | floats()) –  Azat Ibrakov Commented Nov 3, 2021 at 23:02
• Is it possible?? I didn’t know it. if it works, it solved my problem –  Mattia Surricchio Commented Nov 4, 2021 at 8:03
• 1 Yes, that works! You can also use st.one_of(integers(), floats()) which may be easier for programattic use. –  Zac Hatfield-Dodds Commented Nov 4, 2021 at 8:27
• Well you can add an answer and i will accept it! –  Mattia Surricchio Commented Nov 4, 2021 at 8:28

In general if you need your values to be one of several things (like int s or float s in your example) then we can combine strategies for separate things into one with | operator (which operates similar to one for set s -- a union operator and works by invoking __or__ magic method ):

or you can use strategies.one_of as @Zac Hatfield-Dodds mentioned which does pretty much the same thing:

Your Answer

Reminder: Answers generated by artificial intelligence tools are not allowed on Stack Overflow. Learn more

Sign up or log in

Post as a guest.

Required, but never shown

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy .

Not the answer you're looking for? Browse other questions tagged python pytest python-hypothesis or ask your own question .

• The Overflow Blog
• The hidden cost of speed
• The creator of Jenkins discusses CI/CD and balancing business with open source
• Featured on Meta
• Announcing a change to the data-dump process
• Bringing clarity to status tag usage on meta sites
• What does a new user need in a homepage experience on Stack Overflow?
• Feedback requested: How do you use tag hover descriptions for curating and do...
• Staging Ground Reviewer Motivation

Hot Network Questions

• Applying to faculty jobs in universities without a research group in your area
• Transform a list of rules into a list of function definitions
• Swapping touch digitizer from one Android phone to another
• How do I prove the amount of a flight delay in UK court?
• Where is this railroad track as seen in Rocky II during the training montage?
• What would be a good weapon to use with size changing spell
• help to grep a string from a site
• SOT 23-6 SMD marking code GC1MGR
• In which town of Europe (Germany ?) were this 2 photos taken during WWII?
• How to truncate text in latex?
• The question about the existence of an infinite non-trivial controversy
• Does a party have to wait 1d4 hours to start a Short Rest if no healing is available and an ally is only stabilized?
• Could a lawyer agree not to take any further cases against a company?
• Sum[] function not computing the sum
• Why does Jeff think that having a story at all seems gross?
• Is it a good idea to perform I2C Communication in the ISR?
• How to connect 20 plus external hard drives to a computer?
• Is the 2024 Ukrainian invasion of the Kursk region the first time since WW2 Russia was invaded?
• Is my magic enough to keep a person without skin alive for a month?
• Current in a circuit is 50% lower than predicted by Kirchhoff's law
• What prevents random software installation popups from mis-interpreting our consents
• Do Ethernet Switches have MAC Addresses?
• Is "She played good" a grammatically correct sentence?
• Is it helpful to use a thicker gauge wire for only part of a long circuit run that could have higher loads?