Post Run Assertions

Post Run Assertions #

Asserts allow checking statistic values for the individual participant after selenium script execution has finished. Asserts are automatically calculated for each participant to check if given values are within allowed thresholds.

Available assert paths can be divided in two categories:

These categories contain parts of full assert path as in most cases additional path parameters are present, which indicate what aggregation function will be used to retrieve the final value of assert.

Machine statistics asserts: #

Name (pathValue)Description
CPU (cpu)CPU percentage
RAM (ram)RAM usage (in bytes)
Network bytes (network/bytes)Network bytes (total). Incoming and outgoing data
Network bitrate (network/bitrate)Network bitrate per second. Incoming and outgoing data
Network packets (network/packets)Network packets (total). Incoming and outgoing data
Network packet loss (network/packetsLost)Network packets lost (percentage). Incoming and outgoing data
Network errors (network/errors)Network errors (total). Incoming and outgoing data

WebRTC statistics asserts: #

Name (pathValue)Description
Bitrate (bitrate)Actual bitrate of media in kilobits
Packets lost (packetsLost)Number of packets lost during test time
Packets (packets)Number of packets overall (per second)
Jitter (jitter)Media jitter in milliseconds
Jitter buffer (jitterBuffer)Incoming media jitter selected size in seconds
Audio volume (level)Audio volume in absolute values
Round trip time (rtt)Data round-trip time in seconds
Bytes (bytes)Total number of bytes transmitted
Codec (codec)Codec of the stream as a string value
FPS (fps)Video frames per second
Packet metrics measure “packets per second” instead of “total packets”.
To assert total packets use ../packets/../total
WebRTC assertions currently work only on Chrome browser, but support for FireFox is coming soon.
All media tracks are merged together for each participant - to assert the total in/out stats. WebRTC dump is available for download to perform customized assertions.

Aggregator functions #

Metrics are collected once per second and need aggregation to be compared to a single value. We have several aggregation functions defined:

  • Minimum
  • Maximum
  • Average
  • Standard deviation
  • Relative standard deviation*
  • Percentile

*Relative standard deviation asserts offset relative to average value

Relative standard deviation gives more context by reflecting the offset keeping the scale of value. For example, if standard deviation is only 1 byte it does not give any indications about the significance of fluctuations. But by using relative standard deviation it would return fluctuation in percent, immediately giving perspective on the fluctuations without the need to check average value.