Statistics

Sometimes you may like to know the types of values being generated by Kotest, to ensure that your generators are configured in the way you expect. Property test statistics are designed to fulfil this need.

The function collect is the entry point to statistcs and is used to count categories of values. We use this by invoking it inside a property test with the category we want to increment.

For example, lets say we wanted to gather statistics on the values of RoundingMode used by a BigDecimal. We would invoke checkAll as normal, passing the rounding mode to the collect function.

checkAll(Arb.enum<RoundingMode>(), Arb.bigDecimal()) { mode, decimal ->
  collect(mode)
  // test here
}

Now after the test is completed, Kotest will output the test name, and the counts/percentages for each category:

Statistics: [collecting stats] (1000 iterations, 1 args)

HALF_DOWN                                                     142 (14%)
HALF_UP                                                       141 (14%)
CEILING                                                       132 (13%)
FLOOR                                                         122 (12%)
UP                                                            119 (12%)
UNNECESSARY                                                   119 (12%)
HALF_EVEN                                                     118 (12%)
DOWN                                                          107 (11%)

The category we use does not have to be an enum. It can be any object, and you can wrap in conditionals if you want more control. For example:

checkAll(Arb.int()) { k ->
  when {
    k % 2 == 0 -> collect("EVEN")
    else -> collect("ODD")
  }
  // test here
}

Labels

Sometimes you may wish to have orthogonal sets of statistics. For example, for a simple number test, we might want to confirm that a certain percentage are even numbers, and a certain percentage are negative. One way would be to have EVEN_POS, EVEN_NEG, ODD_POS, ODD_NEG:

checkAll(Arb.int()) { k ->
  when {
    k > 0 && k % 2 == 0 -> collect("EVEN_POS")
    k % 2 == 0 -> collect("EVEN_NEG")
    k > 0 -> collect("ODD_POS")
    else -> collect("ODD_NEG")
  }
  // test here
}

This gives us one set of outputs:

EVEN_POS                                                       142 (27%)
EVEN_NEG                                                       141 (23%)
ODD_POS                                                        132 (24%)
ODD_NEG                                                        122 (26%)

However, as the combinations grow this will become unwieldy, so Kotest supports labelled statistics. You can think of this as distinct sets of statistics. To use labels, just pass the label name as the first arg to the collect method.

checkAll(Arb.int()) { k ->
  when {
    k % 2 == 0 -> collect("even_odd", "EVEN")
    else -> collect("even_odd", "ODD")
  }
  when {
    k > 0 -> collect("pos_neg", "POS")
    else -> collect("pos_neg", "NEG")
  }
  // test here
}

Now, Kotest will output multiple sets of statistics, with the label name in the title:

Statistics: [collecting labelled stats] (1000 iterations, 1 args) [even_odd]

ODD                                                           520 (52%)
EVEN                                                          480 (48%)


Statistics: [collecting labelled stats] (1000 iterations, 1 args) [pos_neg]

NEG                                                           527 (53%)
POS                                                           473 (47%)

Report Mode

By default, statistics are printed for every property test. There are four modes which can be configured using the global configuration object PropertyTesting.

The possible options are:

Mode	Function
PropertyTesting.statisticsReportMode = StatisticsReportMode.OFF	disable all statistics reporting
PropertyTesting.statisticsReportMode = StatisticsReportMode.OFF	enables all statistics reporting
PropertyTesting.statisticsReportMode = StatisticsReportMode.SUCCESS	output statistics only on successful tests
PropertyTesting.statisticsReportMode = StatisticsReportMode.FAILED	output statistics only on failed tests

Checking Coverage of Statistics

If you wish to programmatically assert that certain values are being generated, then you can use the specify constraints that must be met.

For example, in our previous rounding example, we can check that at least 10% of inputs are covering HALF_DOWN, and 10% are covering FLOOR using withCoveragePercentages:

withCoveragePercentages(mapOf(RoundingMode.HALF_DOWN to 10.0, RoundingMode.FLOOR to 10.0)) {
  checkAll(Arb.enum<RoundingMode>(), Arb.bigDecimal()) { mode, decimal ->
    collect(mode)
    // use the mode / decimal
  }
}

If we want to check by absolute numbers rather than percentages, we can use withCoverageCounts:

withCoverageCounts(mapOf(RoundingMode.HALF_DOWN to 75, RoundingMode.FLOOR to 75)) {
  checkAll(Arb.enum<RoundingMode>(), Arb.bigDecimal()) { mode, decimal ->
    collect(mode)
    // use the mode / decimal
  }
}

Custom Reports

You can customize the report format, or generate reports from the raw data, by using your own instance of StatisticsReporter. This is configured via the global configuration object PropertyTesting.

For example:

object MyStatisticsReporter : object : StatisticsReporter { ... }
PropertyTesting.statisticsReporter = MyStatisticsReporter