Customizing miteʼs stats

There are two components of the mite pipeline that are concerned with the reporting of test data: the stats component and the prometheus exporter. The prometheus exporter works in a fully automatic way, based on the behavior of the stats component. Therefore, when customizing the statistics that mite exports, it is sufficient to work on the stats component; the prometheus exporter will do the right thing.

Mite stats are registered as entry points in python. Therefore, any modules that you install that extend miteʼs stats will automatically be picked up when the stats process starts. It logs a message for each moduleʼs stats that it finds, so you will know exactly what is loaded.

Writing custom stats

Prometheus, miteʼs chosen data backend, supports three different kinds of metrics: a Gauge, a Counter, and a Histogram.

Note

Prometheus also supports a Summary metric type, but this is just (for our purposes) a less performant version of a Histogram, so it will not be further discussed.

Each of these is exposed as a class in the mite.stats module. When instantiating one of these classes to create a custom metric, you will need to supply a name, a matcher, and an extractor. The name is directly translated into the prometheus name for the metric, so it is useful for it to start with mite_ so that related metrics will be grouped together (especially if the prometheus instance that collects your mite metrics also collects other data, like CPU and memory usage of application processes).

The stats component acts by processing a stream of messages generated by the runner and controller. The matcher is a function that filters these messages, determining which are of interest to a particular stat and which it can ignore. Because each message has a type field, it is often useful to match on the value of that field, which is the purpose of the included mite.stats.matcher_by_type function. (However, the matcher receives the entire message dict as an argument, and can perform arbitrary computation on its contents if it wishes.)

The extractor function operates on the messages that have been selected by the matcher. The extractor pulls out some labels from the message, which will be added as labels to the time series in prometheus. It is also, in most cases, responsible for extracting a numerical value from the message. (The exception is stats of the Counter type, which count the occurrences of a particular message type. There, no value dependent on the message is needed, as the value of the counter is always incremented by one). The built-in mite.stats.extractor function covers most use-cases. It takes two arguments. The first is a sequence of strings, which will be extracted as keys from the message dictionary. The values of these keys will be used as the labels on the prometheus metric. The second argument is a single string, which names the dictionary key containing the numeric value to be accumulated. For more advanced usages, it will be necessary to construct a mite.stats.Extractor class directly. See mite.stats.controller_report_extractor for an example of this.

Here is an example of a custom stat from our work on AMQP testing:

def _amqp_extract(msg):
    for key, value in msg["total_received"].items():
        yield (key,), value


_MITE_STATS = [
    Gauge(
        "mite_amqp_tx_stats",
        matcher_by_type("amqp_tx_stats"),
        extractor(["message_name"], "total_sent")
    ),
    Gauge(
        "mite_amqp_rx_stats",
        matcher_by_type("amqp_rx_stats"),
        Extractor(labels=["message_name"], extract=_amqp_extract)
    )
]

In our AMQP injection functions, we send two messages. The first, amqp_tx_stats, names a message_name and contains a total_sent value. That is extracted into a Gauge by the first stat. Because each message_name is represented by a separate message, this is a simple stat. The second is the mite_rx_stats. This message is send by the worker coroutine that drains the AMQP queues. Once a second, it reports on the total number of messages it has received – of all types. Therefore, we need to write a custom _amqp_extract function, which will yield a sequence of (key, value) tuples.

We also need to inform mite about our stats, using pythonʼs entry points mechanism. To do so, we add a section to the pyproject.toml file for our package:

[project.entry-points.mite_stats]
mite_amqp = "your_package.amqp:_MITE_STATS"

Configuring custom stats

By default, all stats processors that are registered as entrypoints will be used by the mite stats component. However, because each processor has to execute code for each message (even if only to determine that it is not interested in that message), there can be a large cumulative performance impact if you have many stats processors registered. The --stats-include-processors and --stats-exclude-processors can be used to fine-tune this behavior. If --stats-include-processors is passed, then only processors whose name is on the list will be included. If --stats-exclude-processors is passed, then all processors will be included except those on the list. (You can pass both arguments, which will result in the processors from the include list but not on the exclude list being used – but it doesnʼt make much sense to do this, just pass the difference of the two sets as the include argument, and omit the exclude one).

The format of both argumentsʼ value is a comma-separated list without spaces: --stats-include-processors=processor1,processor2

Available processors

The following processors are provided by mite:

  • mite – statistics about the controller and runner workloads

  • mite_finagle – statistics about miteʼs Finagle RPC calls

  • mite_http – statistics about miteʼs http calls

  • mite_kafka – statistics about miteʼs Kafka producer and consumer activity

  • mite_playwright – statistics about miteʼs Playwright browser testing, including page load, network resource and paint timings

  • mite_selenium – statistics about miteʼs selenium-integrated browser testing