Skip to main content

Overview

Welcome to the user documentation for Loadero! This page offers a general overview of what Loadero is, along with covering its main concepts. If you have arrived here with a specific question in mind, you can navigate to an appropriate section in the menu or use the search functionality in the top right.

First and foremost, Loadero is a performance testing tool. As such, Loadero ensures scalability up to tens of thousands of users without much effort required from the person preparing the test. Once a test has been set up and confirmed to be working for just a handful of users, then scaling the test up to a larger number of users can be as simple as changing a participant's count from 1 to 10'000 in one quick action.

Loadero is designed specifically for performing end-to-end testing for web applications and the actions performed during a test are defined in the form of a UI automation script. While performance testing via the API is widely practiced, it does not quite accurately imitate the behavior of users and is limited in its ability to gauge the user experience. Loadero aims to allow for more realistic scenario simulation, by having test participants utilize actual browsers to execute the actions, rather than directly triggering the endpoints of an API. Loadero supports Google Chrome and Mozilla Firefox as browser options.

Additionally, Loadero has also been designed to support WebRTC testing. Loadero is able to emulate microphone and webcam feeds for its test participants, allowing the testing of video and audio quality.

Prerequisites​

There are some prerequisites in order for Loadero to be able to access the application and successfully perform automation.

  1. Since Loadero is tailored towards web applications in particular, the application you are testing must be accessible via a public URL on the web.
  2. Loadero starts the servers that participants operate from dynamically. As a result the IP addresses are not consistent, however, there are two locations that have static IPs. If your web application has a firewall set up, you must whitelist one or both of these IP addresses, and use only the whitelisted locations in your tests.
  3. There must not be any CAPTCHA present on the application. Its very purpose is to prevent automation and Loadero will not be able to access a page guarded by CAPTCHA.

Participants​

Participants are virtual users. These users are hosted on AWS servers, where they execute the script that has been defined by the user.

One of the core concepts for participants are compute units (CUs). CUs are what determines how many resources are at the participant's disposal. One CU corresponds to 0.5 CPU cores and 1 GB of RAM. You are able to configure how many CUs are allocated to your participants. Compute units can be a particularly useful parameter, when you would like to find out how client-side behavior changes depending on how many resources are available to the user.

info

Loadero billing is dependent on how many CUs are consumed by tests.

Aside from compute units, you are also able to define:

  1. The location from which the participant will be executing the script.
  2. The browser type and version that the participant will be utilizing.
  3. Network conditions which allow you to simulate a diverse set of scenarios when it comes to network speed, e.g., whether the application is still operable for a user with subpar network conditions.
  4. Media and video feed which will be used to emulate microphone and webcam input for applications where these devices are in use.
note

Creating a participant is more akin to creating a configuration profile. The count field for the participant defines how many individual instances of this configuration will actually be generated. E.g., if you create a participant with count=5, there will be 5 participants with the exact same settings.

Test modes​

Loadero allows you to run your tests in 3 different test modes. Though all test modes provide Selenium logs, browser logs, and a WebRTC dump, as well as support all general script functionality, each of these test modes has benefits that are absent in other test modes:

  • Performance test mode is the only test mode that provides machine statistics (CPU/RAM/network usage), but only allows running up to 200 participants in the test. This is also the only test mode where the run report will allow you to access files that had been downloaded during the test.
  • Load test mode has no cap on maximum participants allowed in the test, however, Selenium logs will have reduced verbosity, making it less fit for debugging. This test mode is ideal for scaling up a script to simulate high loads on the application you are testing, once you have already confirmed the script works fine at a smaller scale.
  • Session recording mode provides a video recording (with audio included) that shows everything the participant went through from start to finish, however, this test mode only allows up to 10 participants in one test. This test mode is recommended for script debugging purposes. Keep in mind that the process of recording the session requires additional resources from the participant, so their script execution may be slower than in other modes.
info

We generally recommend using performance test mode if:

  • the test does not require more than 200 participants;
  • you are not interested in obtaining a session recording.

Running tests​

Running tests in Loadero is just a matter of clicking "Run test" once the test has been set up. A walkthrough for the whole process from test creation to running the test is provided in our quick-start guide. Loadero takes care of everything else beyond that point - setting up the servers, creating the virtual users, gathering the results, etc. However, keep in mind that script execution does not begin immediately as you run the test. Test runs will go through multiple phases, each of these phases is covered further below.

Pending​

This is the phase any test run will begin in once you choose to launch a test. This status signifies that Loadero is currently gauging whether there are enough instances available in your chosen locations or waiting for these instances to free up. Typically this phase lasts just a few seconds, but can last an extended period of time if:

  • several test runs had recently been started in rapid succession (not necessarily by you, but by other users as well) before your own run, so your run is still in the queue to be processed;
  • there are tests currently occupying so many servers in the provided location, that there isn't any room for your own run (e.g., a 20k user test based in Oregon may be preventing you from running your own load test in Oregon) - once these tests finish, the space will free up and the queue will continue.
tip

If your test run is stuck in Pending state, it may be a good idea to just cancel the test, set up your participants to use a different location, and then try running the test again - other locations may not be as congested. You will not be billed for compute units used in a test if you abort it while it is still in the Pending state.

Initializing​

This is the phase that takes place after Loadero has confirmed that there are instances available and then starts actually setting those instances up for the test. This phase typically lasts about 3-5 minutes.

info

Test script execution does not yet begin in this phase! The participants are only being prepared to be able to run the script.

danger

Once a test enters this phase, all compute units that were configured to be used by this test will be consumed. This means that if you decide to abort the test before it even enters the "Running" phase - your compute units will be consumed and you will have no results for that test. Evaluate your decision to abort tests carefully!

Running​

This is where the participants actually begin executing the script. More accurately - once the test enters the "Running" phase, it means that the first participant of the test has begun executing the script. The timing for when the rest of the participants start execution is dependent on the start interval and increment strategy chosen for the test.

You will also be able to see how long the test has been running, as Loadero provides a test duration timer.

Stopping​

This phase means that script execution has finished and Loadero is currently extracting all metrics, logs, and other information before providing you with the test report.

Done​

Once your test has finished execution and all results are retrieved by Loadero, your run will enter the Done phase. At this point, you are able to access the run report and peruse all the information available there.