Skip to content

Property-based Testing

Kotest is split into several subprojects which can be used independently. One of these subprojects is the property test framework.

Quick Start

To use Kotest's property-based testing you need to add the module io.kotest:kotest-property:<version> to your build.

Info

Upgrading from 3.x? The kotest-property module is only available in version 4.0+. It replaces the previous property test classes which are now deprecated.

Introduction To Property Testing

Developers typically write example-based tests. These are your garden variety unit tests you know and love. You provide the inputs and the expected values, and a test framework like Kotest checks that the two align, failing the build if they don't match up.

One problem with this approach is that it is very easy to miss errors due to edge cases or lack of coverage in the chosen inputs. With property testing, hundreds or thousands of values are fed into the same test, and the values are randomly generated by your property test framework.

A good property test framework will include things like negative infinity, empty lists, strings with non-ascii characters, and so on. Things we often forget about when writing example based tests.

Property tests were originally conceived in frameworks like Quickcheck with the notion of testing a property on some object, something that should hold true for all inputs. An example is the length of string A plus the length of string B should always be equal to the length of A + B.

This is where the term property testing originates.

Kotest supports this through the io.kotest.property.forAll function which accepts an n-arity function (a, ..., n) -> Boolean that tests the property.

For example, here is the property test that we mentioned just a few paragraphs ago. It checks that for any two Strings, the length of a + b is the same as the length of a plus the length of b. In this example Kotest would execute the test 1000 times for random String combinations.

class PropertyExample: StringSpec({
   "String size" {
      forAll<String, String> { a, b ->
         (a + b).length == a.length + b.length
      }
   }
})

Notice that the function must evaluate to a boolean value. We provide the type parameters to forAll so the framework knows which type of values to generate (in this case strings).

If we don't want to provide a property that returns a boolean, Kotest also provides for io.kotest.property.checkAll which accepts an n-arity function (a, ..., n) -> Unit in which you can simply execute assertions against the inputs. For example:

class PropertyExample: StringSpec({
   "integers under addition should have an identity value" {
      checkAll<Int, Int, Int> { a, b, c ->
         a + 0 shouldbe a
         0 + a shouldBe a
      }
   }
})

The checkAll approach will consider a test valid if no exceptions were thrown.

Iterations

By default, Kotest will run the property test 1000 times. We can easily customize this by specifying the iteration count when invoking the test method.

Let's say we want to run a test 10,000 times.

class PropertyExample: StringSpec({
   "some test" {
      checkAll<Double, Double>(10000) { a, b ->
         // test here
      }
   }
})

Configuration

Kotest provides for the ability to specify some configuration options when running a property test. We do this by passing in an instance of PropTestConfig to the test methods.

For example:

class PropertyExample: StringSpec({
   "String size" {
      forAll<String, String>(PropTestConfig(options here...)) { a,b ->
         (a + b).length == a.length + b.length
      }
   }
})

Seed

The most common configuration option is specifying the seed for the random instance. This is used when you want to reliably create the same values each time the test is run. You might want to do this if you find a test failure, and you want to ensure that that particular set of values continues to be executed in the future as a kind of regression test.

Tip

Whenever a property test fails, Kotest will output the seed that was used, so you can copy it into another test to "fix" that seed value.

For example:

class PropertyExample: StringSpec({
   "String size" {
      forAll<String, String>(PropTestConfig(seed = 127305235)) { a,b ->
         (a + b).length == a.length + b.length
      }
   }
})

Min Failure

By default, Kotest tolerates no failure. Perhaps you want to run some non-deterministic test a bunch of times, and you're happy to accept some small number of failures. You can specify that in config.

class PropertyExample: StringSpec({
   "some flakey test" {
      forAll<String, String>(PropTestConfig(maxFailure = 3)) { a,b ->
         // max of 3 inputs can fail
      }
   }
})

PropTestListener

Sometimes in property test it is required to perform some setup and tear down in each iteration of test. For this purpose you can register a PropTestListener with PropTestConfig.

class PropertyExample: StringSpec({
   "some property test which require setup and tear down in each iteration" {
      forAll<String, String>(PropTestConfig(listeners = listOf(MyPropTestListener))) { a,b ->
         // some assertion
      }
   }
})

Generators

Generated values are provided by instances of the sealed class Gen. You can think of a Gen as kind of like an input stream but for property testing. Each Gen will provide a (usually) infinite stream of these values.

Kotest has two types of generators - Arb for arbitrary (random) values and Exhaustive for a finite set of values in a closed space.

Both types of gens can be mixed and matched in property tests. For example, you could test a function with 100 random positive integers (arbitrary) alongside every even number from 0 to 200 (exhaustive).

Some generators are only available on the JVM. See the full list here.

Arb

Arbs generate random values across a given space. The values may be repeated, and some values may never be generated at all. For example generating 1000 random integers between 0 and Int.MAX will clearly not return all possible values, and some values may happen to be generated more than once.

An arb will generate an infinite stream of values.

Typical arbs include numbers across a wide number line, strings in the unicode set, random lists, random data classes, emails, codepoint and chars.

Exhaustive

Exhaustives generate all values from a given space. This is useful when you want to ensure every value in that space is used. For instance for enum values, it is usually more helpful to ensure each enum is used, rather than picking randomly from the enums values and potentially missing some and duplicating others.

Typical exhaustives include small collections, enums, boolean values, powerset of a list or set, pre-defined small integer ranges, and predefined string ranges.

Specifying Generators

You saw earlier when using forAll or checkAll that if we specify the type parameters, Kotest will provide an appropriate gen. This is fine for basic tests but often we want more control over the sample space.

To do this, we can instantiate the generators ourselves by using extension functions on Arb and/or Exhaustive and passing those into the assert/check methods.

For example, we may want to test a function for numbers in a certain range only.

class PropertyExample: StringSpec({
    "is allowed to drink in Chicago" {
        forAll(Arb.int(21..150)) { a ->
            isDrinkingAge(a) // assuming some function that calculates if we're old enough to drink
        }
    }
    "is allowed to drink in London" {
        forAll(Arb.int(18..150)) { a ->
            isDrinkingAge(a) // assuming some function that calculates if we're old enough to drink
        }
    }
})

Actually, ages are a small space, it would probably be better not to leave the values to chance.

class PropertyExample: StringSpec({
    "is allowed to drink in Chicago" {
        forAll(Exhaustive.int(21..150)) { a ->
            isDrinkingAge(a) // assuming some function that calculates if we're old enough to drink
        }
    }
    "is allowed to drink in London" {
        forAll(Exhaustive.int(18..150)) { a ->
            isDrinkingAge(a) // assuming some function that calculates if we're old enough to drink
        }
    }
})

You can mix and match arbs and exhaustives in the same test of course, since they are both generators.

class PropertyExample: StringSpec({
    "some dummy test" {
        checkAll(Arb.emails(), Exhaustive.enum<Foo>) { email, foo ->
           // test here
        }
    }
})

See here for a list of the built in generators.