Post Run Assertions

Post Run Assertions #

Post run asserts allow checking statistic values for the individual participant after Selenium script execution has finished. Asserts are automatically calculated for each participant to check if the given values are within allowed thresholds. If the assertion failed for a participant, this participant’s status in the results report will be “Fail” too. All assert path values and operators are predefined and can be selected from the dropdown menus.

Available assert paths can be divided into two categories:

These categories contain parts of the full assert path. Additional path parameters such as:

Machine statistics asserts: #

All machine statistics assert paths must start with path value machine. The following table contains path values for machine statistics

Path valueUnitDescription
cpu/available% percentageTotal CPU power available
cpu/used% percentageTotal CPU power used
cpu/percent% percentageRelative CPU usage
ram/availableMB megabytesTotal RAM available
ram/usedMB megabytesTotal RAM used
ram/percent% percentageRelative RAM usage
network/bitratekbit/s kilobits per secondNetwork bitrate
network/bytesKB kilobytesTotal amount of network traffic
network/errorscountNumber of network errors
network/packetscount/s packets per secondNumber of network packets
network/packetsLostcountNumber of lost network packets

WebRTC statistics asserts: #

All WebRTC statistics assert paths must start with path value webrtc. WebRTC statistics are further divided into audio and video paths. The following table contains path values shared by webrtc/audio and webrtc/video statistics.

Path valueUnitDescription
bitratekbit/s kilobits per secondActual bitrate of media
bytesKB kilobytesAmount of data conveyed
codecstring valueCodec of the stream
jitterms millisecondsDuration of media jitter
jitterBufferms millisecondsDuration of jitter buffer
packetscount/s packets per secondNumber of network packets
packetsLostcountNumber of lost network packets
rttms milisecondsData round-trip time

webrtc/audio has one unique statistic with path value level. It describes audio volume in range from 0 to 1.

webrtc/video has one unique statistic with path value fps. It describes video frames per second.

All media tracks are merged together for each participant - to assert the total in/out stats. WebRTC dump is available for download to perform customized assertions.

Direction #

Some metrics are further divided by the direction of the data stream into incoming and outgoing data. Defined by path values in and out.

Aggregator functions #

Metrics are collected once per second and need aggregation to be compared to a single value. We have several aggregation functions defined:

  • Total
  • Minimum
  • Maximum
  • Average
  • Standard deviation
  • Relative standard deviation
  • Percentile (25th, 50th, 75th, 95th, 99th)
Relative standard deviation gives more context by reflecting the offset keeping the scale of value. For example, if standard deviation is only 1 byte it does not give any indications about the significance of fluctuations. But by using relative standard deviation it would return fluctuation in percent, immediately giving perspective on the fluctuations without the need to check average value.

Examples #

Full Metric pathDescription of metric
machine/network/packets/out/avgAverage value of outgoing (sent) packets per second.
webrtc/audio/level/in/maxMaximum value of incoming audio volume.