Post Run Assertions #
Asserts allow checking statistic values for the individual participant after selenium script execution has finished. Asserts are automatically calculated for each participant to check if given values are within allowed thresholds.
Available assert paths can be divided in two categories:
These categories contain parts of full assert path as in most cases additional path parameters are present, which indicate what aggregation function will be used to retrieve the final value of assert.
Machine statistics asserts: #
|CPU (||CPU percentage|
|RAM (||RAM usage (in bytes)|
|Network bytes (||Network bytes (total). Incoming and outgoing data|
|Network bitrate (||Network bitrate per second. Incoming and outgoing data|
|Network packets (||Network packets (total). Incoming and outgoing data|
|Network packet loss (||Network packets lost (percentage). Incoming and outgoing data|
|Network errors (||Network errors (total). Incoming and outgoing data|
WebRTC statistics asserts: #
|Bitrate (||Actual bitrate of media in kilobits|
|Packets lost (||Number of packets lost during test time|
|Packets (||Number of packets overall (per second)|
|Jitter (||Media jitter in milliseconds|
|Jitter buffer (||Incoming media jitter selected size in seconds|
|Audio volume (||Audio volume in absolute values|
|Round trip time (||Data round-trip time in seconds|
|Bytes (||Total number of bytes transmitted|
|Codec (||Codec of the stream as a string value|
|FPS (||Video frames per second|
Packet metrics measure “packets per second” instead of “total packets”.
To assert total packets use ../packets/../total
WebRTC assertions currently work only on Chrome browser, but support for FireFox is coming soon.
All media tracks are merged together for each participant - to assert the total in/out stats. WebRTC dump is available for download to perform customized assertions.
Aggregator functions #
Metrics are collected once per second and need aggregation to be compared to a single value. We have several aggregation functions defined:
- Standard deviation
- Relative standard deviation*
*Relative standard deviation asserts offset relative to average value
Relative standard deviation gives more context by reflecting the offset keeping the scale of value. For example, if standard deviation is only 1 byte it does not give any indications about the significance of fluctuations. But by using relative standard deviation it would return fluctuation in percent, immediately giving perspective on the fluctuations without the need to check average value.