Post run assertions

Asserts allow checking statistic values for the individual participant after selenium script execution has finished. Asserts are automatically calculated for each participant to check if given values are within allowed thresholds.

Available assert paths can be divided in two categories:

These categories contain parts of full assert path as in most cases additional path parameters are present, which indicate what aggregation function will be used to retrieve the final value of assert.

Machine statistics asserts:

Name (pathValue)

Description

CPU (cpu)

CPU percentage

RAM (ram)

RAM usage (in bytes)

Network bytes (network/bytes)

Network bytes (total). Incoming and outgoing data

Network bitrate (network/bitrate)

Network bitrate per second. Incoming and outgoing data

Network packets (network/packets)

Network packets (total). Incoming and outgoing data

Network packet loss (network/packetsLost)

Network packets lost (percentage). Incoming and outgoing data

Network errors (network/errors)

Network errors (total). Incoming and outgoing data

WebRTC statistics asserts:

Name (pathValue)

Description

Bitrate (bitrate)

Actual bitrate of media in kilobits

Packets lost (packetsLost)

Number of packets lost during test time

Packets (packets)

Number of packets overall (per second)

Jitter (jitter)

Media jitter in seconds

Jitter buffer (jitterBuffer)

Incoming media jitter selected size in milliseconds

Audio volume (level)

Audio volume in absolute values

Round trip time (rtt)

Data round-trip time in seconds

Bytes (bytes)

Total number of bytes transmitted

Codec (codec)

Codec of the stream as a string value

FPS (fps)

Video frames per second

Packet metrics measure “packets per second” instead of “total packets”. To assert total packets use ../packets/../total

WebRTC assertions currently work only on Chrome browser, but support for FireFox is coming soon.

All media tracks are merged together for each participant - to assert the total in/out stats. WebRTC dump is available for download to perform customised assertions.

Aggregator functions

Metrics are collected once per second and need aggregation to be compared to a single value. We have several aggregation functions defined:

  • Minimum

  • Maximum

  • Average

  • Standard deviation

  • Relative standard deviation*

  • Percentile

*Relative standard deviation asserts offset relative to average value

Relative standard deviation gives more context by reflecting the offset keeping the scale of value. For example, if standard deviation is only 1 byte it does not give any indications about the significance of fluctuations. But by using relative standard deviation it would return fluctuation in percent, immediately giving perspective on the fluctuations without the need to check average value.

When using percent asserts key, the values are converted to percents relative to the maximum value. RAM is an exception where the used compute unit for participant will determine the maximum value.