Skip to main content
Version: 5.8

Statistics

Sometimes you may like to know the types of values being generated by Kotest, to ensure that your generators are configured in the way you expect. Property test statistics are designed to fulfil this need.

The function collect is the entry point to statistcs and is used to count categories of values. We use this by invoking it inside a property test with the category we want to increment.

For example, lets say we wanted to gather statistics on the values of RoundingMode used by a BigDecimal. We would invoke checkAll as normal, passing the rounding mode to the collect function.

checkAll(Arb.enum<RoundingMode>(), Arb.bigDecimal()) { mode, decimal ->
collect(mode)
// test here
}

Now after the test is completed, Kotest will output the test name, and the counts/percentages for each category:

Statistics: [collecting stats] (1000 iterations, 1 args)

HALF_DOWN 142 (14%)
HALF_UP 141 (14%)
CEILING 132 (13%)
FLOOR 122 (12%)
UP 119 (12%)
UNNECESSARY 119 (12%)
HALF_EVEN 118 (12%)
DOWN 107 (11%)

The category we use does not have to be an enum. It can be any object, and you can wrap in conditionals if you want more control. For example:

checkAll(Arb.int()) { k ->
when {
k % 2 == 0 -> collect("EVEN")
else -> collect("ODD")
}
// test here
}

Labels​

Sometimes you may wish to have orthogonal sets of statistics. For example, for a simple number test, we might want to confirm that a certain percentage are even numbers, and a certain percentage are negative. One way would be to have EVEN_POS, EVEN_NEG, ODD_POS, ODD_NEG:

checkAll(Arb.int()) { k ->
when {
k > 0 && k % 2 == 0 -> collect("EVEN_POS")
k % 2 == 0 -> collect("EVEN_NEG")
k > 0 -> collect("ODD_POS")
else -> collect("ODD_NEG")
}
// test here
}

This gives us one set of outputs:

EVEN_POS                                                       142 (27%)
EVEN_NEG 141 (23%)
ODD_POS 132 (24%)
ODD_NEG 122 (26%)

However, as the combinations grow this will become unwieldy, so Kotest supports labelled statistics. You can think of this as distinct sets of statistics. To use labels, just pass the label name as the first arg to the collect method.

checkAll(Arb.int()) { k ->
when {
k % 2 == 0 -> collect("even_odd", "EVEN")
else -> collect("even_odd", "ODD")
}
when {
k > 0 -> collect("pos_neg", "POS")
else -> collect("pos_neg", "NEG")
}
// test here
}

Now, Kotest will output multiple sets of statistics, with the label name in the title:

Statistics: [collecting labelled stats] (1000 iterations, 1 args) [even_odd]

ODD 520 (52%)
EVEN 480 (48%)


Statistics: [collecting labelled stats] (1000 iterations, 1 args) [pos_neg]

NEG 527 (53%)
POS 473 (47%)

Report Mode​

By default, statistics are printed for every property test. There are four modes which can be configured using the global configuration object PropertyTesting.

The possible options are:

ModeFunction
PropertyTesting.statisticsReportMode = StatisticsReportMode.OFFdisable all statistics reporting
PropertyTesting.statisticsReportMode = StatisticsReportMode.OFFenables all statistics reporting
PropertyTesting.statisticsReportMode = StatisticsReportMode.SUCCESSoutput statistics only on successful tests
PropertyTesting.statisticsReportMode = StatisticsReportMode.FAILEDoutput statistics only on failed tests

Checking Coverage of Statistics​

If you wish to programmatically assert that certain values are being generated, then you can use the specify constraints that must be met.

For example, in our previous rounding example, we can check that at least 10% of inputs are covering HALF_DOWN, and 10% are covering FLOOR using withCoveragePercentages:

withCoveragePercentages(mapOf(RoundingMode.HALF_DOWN to 10.0, RoundingMode.FLOOR to 10.0)) {
checkAll(Arb.enum<RoundingMode>(), Arb.bigDecimal()) { mode, decimal ->
collect(mode)
// use the mode / decimal
}
}

If we want to check by absolute numbers rather than percentages, we can use withCoverageCounts:

withCoverageCounts(mapOf(RoundingMode.HALF_DOWN to 75, RoundingMode.FLOOR to 75)) {
checkAll(Arb.enum<RoundingMode>(), Arb.bigDecimal()) { mode, decimal ->
collect(mode)
// use the mode / decimal
}
}

Custom Reports​

You can customize the report format, or generate reports from the raw data, by using your own instance of StatisticsReporter. This is configured via the global configuration object PropertyTesting.

For example:

object MyStatisticsReporter : object : StatisticsReporter { ... }
PropertyTesting.statisticsReporter = MyStatisticsReporter