Post Run Assertions

Post Run Assertions #

Post run asserts allow checking statistic values for the individual participant after Selenium script execution has finished. Asserts are automatically calculated for each participant to check if the given values are within allowed thresholds. If the assertion failed for a participant, this participant’s status in the results report will be “Fail” too. All assert path values and operators are predefined and can be selected from the dropdown menus.

Available assert paths can be divided in two categories:

These categories contain parts of full assert path as in most cases additional path parameters are present, which indicate what aggregation function will be used to retrieve the final value of assert.

Machine statistics asserts: #

Name (pathValue)Description
CPU (cpu)CPU use percentage.
RAM (ram)RAM usage in bytes.
Network bytes (network/bytes)Total amount of network bytes (total). Incoming and outgoing data.
Network bitrate (network/bitrate)Network bitrate per second. Incoming and outgoing data.
Network packets (network/packets)Total amount of network packets. Incoming and outgoing data.
Network packet loss (network/packetsLost)Lost network packets percentage. Incoming and outgoing data.
Network errors (network/errors)Total amount of network errors. Incoming and outgoing data.

WebRTC statistics asserts: #

Name (pathValue)Description
Bitrate (bitrate)Actual bitrate of media in kilobits. Incoming and outgoing data.
Packets lost (packetsLost)Number of packets lost during test time. Incoming and outgoing data.
Packets (packets)Total amount of network packets (per second). Incoming and outgoing data.
Jitter (jitter)Media jitter in seconds.
Jitter buffer (jitterBuffer)Incoming media jitter selected size in seconds.
Audio volume (level)Audio volume in absolute values. Incoming and outgoing data.
Round trip time (rtt)Data round-trip time in seconds.
Bytes (bytes)Total number of bytes transmitted. Incoming and outgoing data.
Codec (codec)Codec of the stream as a string value. Incoming and outgoing data.
FPS (fps)Video frames per second. Incoming and outgoing data.
Packet metrics measure “packets per second” instead of “total packets”. To assert total packets use ../packets/../total
WebRTC assertions currently work only on Chrome browser, but support for FireFox is coming soon.
All media tracks are merged together for each participant - to assert the total in/out stats. WebRTC dump is available for download to perform customized assertions.

Aggregator functions #

Metrics are collected once per second and need aggregation to be compared to a single value. We have several aggregation functions defined:

  • Minimum
  • Maximum
  • Average
  • Standard deviation
  • Relative standard deviation*
  • Percentile (25th, 50th, 75th, 95th, 99th)

*Relative standard deviation asserts offset relative to average value

Relative standard deviation gives more context by reflecting the offset keeping the scale of value. For example, if standard deviation is only 1 byte it does not give any indications about the significance of fluctuations. But by using relative standard deviation it would return fluctuation in percent, immediately giving perspective on the fluctuations without the need to check average value.