Post Run Assertions
Post run asserts allow checking statistic values for the individual participant after Selenium script execution has finished. Asserts are automatically calculated for each participant to check if the given values are within allowed thresholds. If the assertion failed for a participant, this participant’s status in the results report will be “Fail” too. All assert path values and operators are predefined and can be selected from the dropdown menus.
Available assert paths can be divided into two categories:
These categories contain parts of the full assert path. Additional path parameters such as:
- direction of data stream
- aggregation function of statistic will be used to retrieve the final value of assert.
Machine statistics asserts:
All machine statistics assert paths must start with path value machine. The
following table contains path values for machine statistics
| Path value | Unit | Description |
|---|---|---|
cpu/available | % percentage | Total CPU power available |
cpu/used | % percentage | Total CPU power used |
cpu/percent | % percentage | Relative CPU usage |
ram/available | MB megabytes | Total RAM available |
ram/used | MB megabytes | Total RAM used |
ram/percent | % percentage | Relative RAM usage |
network/bitrate | kbit/s kilobits per second | Network bitrate |
network/bytes | KB kilobytes | Total amount of network traffic |
network/errors | count | Number of network errors |
network/packets | count/s packets per second | Number of network packets |
network/packetsLost | count | Number of lost network packets |
WebRTC statistics asserts:
All WebRTC statistics assert paths must start with path value webrtc. WebRTC
statistics are further divided into audio and video paths. The following
table contains path values shared by webrtc/audio and webrtc/video statistics.
| Path value | Unit | Description |
|---|---|---|
bitrate | kbit/s kilobits per second | Actual bitrate of media |
bytes | KB kilobytes | Amount of data conveyed |
codec | string value | Codec of the stream |
jitter | ms milliseconds | Duration of media jitter |
jitterBuffer | ms milliseconds | Duration of jitter buffer |
packets | count/s packets per second | Number of network packets |
packetsLost | count | Number of lost network packets |
rtt | ms milliseconds | Data round-trip time |
connections | count | Number of WebRTC connections established |
Audio-specific metric paths:
| Path value | Unit | Description |
|---|---|---|
level | relative floating point value | Audio volume in range from 0 to 1 |
Video-specific metric paths:
| Path value | Unit | Description |
|---|---|---|
fps | count per second | Number of frames encoded/decoded per second |
frameWidth | pixels | Frame width |
frameHeight | pixels | Frame height |
All media tracks are merged together for each participant - to assert the total in/out stats. WebRTC dump is available for download to perform customized assertions.
Direction
Some metrics are further divided by the direction of the data stream into incoming and
outgoing data. Defined by path values in and out.
Aggregator functions
Metrics are collected once per second and need aggregation to be compared to a single value. We have several aggregation functions defined:
- Total
- Minimum
- Maximum
- Average
- Standard deviation
- Relative standard deviation
- Percentile (25th, 50th, 75th, 95th, 99th)
- Percent
Relative standard deviation gives more context by reflecting the offset keeping the scale of value. For example, if standard deviation is only 1 byte it does not give any indications about the significance of fluctuations. But by using relative standard deviation it would return fluctuation in percent, immediately giving perspective on the fluctuations without the need to check average value.
Percent aggregator function will output value in the range of 0-100.
Examples
| Full Metric path | Description of metric |
|---|---|
machine/network/packets/out/avg | Average value of outgoing (sent) packets per second. |
webrtc/audio/level/in/max | Maximum value of incoming audio volume. |