Test run queue

Test execution order and statuses

Parallel participant limit

Loadero allows running test in parallel independently from other runs. This gives you the ability to run your tests faster and get results quicker. All your tests will be run on Loadero's infrastructure (except when you are using your own custom AWS). This means that you share the infrastructure with other users who use the same AWS account. Every test mode has its own particular participant limit that allows parallel test execution (except custom AWS queues which have the same queue for all test modes).

All projects that are linked to the same AWS, share the same queue which is not shared with other AWS accounts, e.g., if you use custom AWS to which 2 projects are linked, those projects will share one queue for all test modes.

Assume queue has a limit of 150 parallel participants (real run queue may or may not have the same parameters) and you run 160 participant test and 5 participant test. It happened that both of your tests were assigned to the same queue and there are no other runs on it besides yours. Since the 160 participant test takes up all the concurrent participant limit, all other queued tests will have status pending. Once the test finishes running, the next queued test will be run (tests are queued in First-in First-out a.k.a. FIFO method).

Permanent node

There is often a need to run a small test to quickly check certain functionality, e.g., sanity test if a website is even accessible or short 2 attendee WebRTC video call. For such cases Loadero offers permanent node which is an always-on server for certain types of tests, hence it allows to run tests immediately, skipping node initialization phase.

The test must fulfill all of the following conditions to have a chance to be run on a permanent node:

  • Total participant count must not exceed 2 participants.

  • All participants must be located in Oregon, US region.

  • Test mode must be performance.

At the moment permanent node is available only using Loadero infrastructure and it is shared between all Loadero users who qualify for it. That means that if someone is already running a test on a permanent node, all following tests, while the permanent node is occupied, will be run on regular test nodes (node will need initialization phase).

The availability of permanent node depends on several factors, including the number of other tests running in Loadero and whether they qualify to be run on the permanent node.

Test Run statuses

During test execution your test run will go through several stages, which will be represented by the test run status. These are currently possible test run status with descriptions explaining current test run state.

Status

Description

Pending

Test run is still waiting in queue until it can be picked up by a test executor

Initializing

Test run has started test environment setup

Running

Test run is currently in progress

Waiting for results

Test has finished its execution and execution results are being processed

Stopping

Test run is currently being stopped by user invocation

Done

Test has finished its execution successfully

Aborted

Test was aborted before it finished its execution

Server error

Test triggered some server-side error that should not have happened

Database error

There was an error in database queries or connection during the test

AWS error

Some error from AWS was encountered which prevented successful test execution

Insufficient resources

Test execution did not start because not enough resource were available

Timeout exceeded

Test execution has exceeded a timeout in some test phase