Property-based testing is probably the thing I missed the most from my time working with Haskel. It is such an elegant way of testing functionalities that it is hard to not use it. As you can imagine, I look for a Property-based testing framework in any language I have under my hands. Usually, unsuccessfully.
However, in recent years I am working a lot with Typescript and, luckily, Typescript has a good property-based testing library: fast-check.
Wait. You don’t know what is property-based testing? Let’s start from the beginning.
What is Property-Based Testing
Property-Based Testing (PBT) is a testing methodology mainly introduced by Haskell’s Quick Test library. As the name says, it is based on properties. A property, in short, is a condition that is verified independently of the specific input.
For instance, “reversing a list twice return the original list” is a property. “Sorting a sorted list does not change the list” is another one. “Adding N elements to a queue and removing N elements gives me the empty queue” is another one. As you can see, these are facts that are always true for every possible (legal) input. As a consequence, we can leave to the PBT framework the hard work of generating inputs and edge cases that will make our test fail.
With this simple intuition, we get two great benefits:
- We do not need to generate the input data: we just need to define what is a legal input for this function (e.g., it is a string, it is a number, it is a list of strings, it is a tuple, it is a pair
<string, number>, and so on).
- Writing a test is reduced to just specifying the properties we need to check (e.g., “sorting a sorted list does not change the list”).
The PBT framework will take care of everything else. Not only it will be able to test your properties using a wide range of randomly generated inputs, but it will also return the smallest problematic input in case of failure. A really useful information for debugging.
Let’s see the example code.
As you can see, we are testing two properties of our
- A string should always contain itself.
- A string should always contain its substrings.
In this snippet, we can identify three main elements.
A “runner” is the element that executes and tests a property. In the example code
fc.assert is the runner.
fc.assert is also the most basic runner: it executes the tests on the given property and throws if it fails.
There are more runners in fast-check (
fs.sample and others). However,
fc.assert is a good solution 99% of the times. If you want to know more about runners, you can check the documentation.
The second element is
fc.property. This is a function that creates the core element of PBT: a property.
In short, a property is defined by a list of input generators (called arbitraries) and a function taking such inputs.
For instance, let’s look at
With this line, we are creating a property (defined with a function returning a boolean, e.g.,
text => contains(text, text)) with one string parameter generated by
There is also
fc.asyncProperty for properties that need asynchronous properties.
Arbitraries are object used to generate the random inputs for the properties. Fast-check includes any kind of arbitraries in addition to the possibility to create custom ones.
You can have
fc.json() and many, many more! You can find a complete list here.
Running the tests
Running the tests is straightforward. If you already have a mocha runner configured, you just need to do
npm test as usual. It is important to know that PBT is slower that ad-hoc tests because each property is tested hundreds of times.
In case of failure you may get a message like this.
1) should always contain its substrings Error: Property failed after 1 tests (seed: 1527422598337, path: 0:0): ["","",""] Shrunk 1 time(s) Got error: Property failed by returning false
Not only you get an error notification but you also get a counterexample of your property (that is, an input for which your function is not correct). In this case
Note that, this is not just “a counterexample”: it is the smallest. The process for which a PBT goes from a failure to the smallest counterexample is called shrinking and it is THE beste feature of PBT. Maybe it could be the topic of an article by itself.
So, what do you think? I am personally pretty happy with fast-check so far. I hope you will be too.