Post run assertions

Asserts allow checking statistic values for the individual participant after selenium script execution has finished. Asserts are automatically calculated for each participant to check if given values are within allowed thresholds.

Available assert paths can be divided in two categories:

These categories contain parts of full assert path as in most cases additional path parameters are present, which indicate what aggregation function will be used to retrieve the final value of assert.

Machine statistics asserts:

Name (pathValue)

Description

CPU (cpu)

CPU percentage

RAM (ram)

RAM usage (in bytes)

Network bytes (network/bytes)

Network bytes (total). Incoming and outgoing data

Network bitrate (network/bitrate)

Network bitrate per second. Incoming and outgoing data

Network packets (network/packets)

Network packets (total). Incoming and outgoing data

Network packet loss (network/packetsLost)

Network packets lost (percentage). Incoming and outgoing data

Network errors (network/errors)

Network errors (total). Incoming and outgoing data

webRTC statistics asserts:

Name (pathValue)

Description

Bitrate (bitrate)

Actual bitrate of media in kilobits

Packets lost (packetsLost)

Number of packets lost during test time

Packets (packets)

Number of packets overall (per second)

Jitter (jitter)

Media jitter in milliseconds

Jitter buffer (jitterBuffer)

Incoming media jitter selected size in milliseconds

Audio volume (level)

Audio volume in absolute values

Round trip time (rtt)

Data round-trip time in milliseconds

Bytes (bytes)

Total number of bytes transmitted

Codec (codec)

Codec of the stream as a string value

Packet metrics measure “packets per second” instead of “total packets”. To assert total packets use ../packets/../total

webRTC assertions currently work only on Chrome browser, but support for FireFox is coming soon.

All media tracks are merged together for each participant - to assert the total in/out stats. WebRTC dump is available for download to perform customized assertions.

Aggregator functions

Metrics are collected once per second and need aggregation to be compared to a single value. We have several aggregation functions defined:

  • Minimum

  • Maximum

  • Average

  • Standard deviation

  • Relative standard deviation*

  • Percentile

*Relative standard deviation asserts offset relative to average value

Relative standard deviation gives more context by reflecting the offset keeping the scale of value. For example, if standard deviation is only 1 byte it does not give any indications about the significance of fluctuations. But by using relative standard deviation it would return fluctuation in percent, immediately giving perspective on the fluctuations without the need to check average value.