Changelog

Changelog #

February 2023 #

Fixes and improvements #

  • Added support for new browser versions: Google Chrome 110 and Mozilla Firefox 110.
  • We have updated how WebRTC metrics are collected during test runs. This can impact RAM usage measurements since calculations are not done on the participant’s instance anymore during the test.

New features #

  • WebRTC metrics are now available for session recording tests as well, meaning that WebRTC metrics are now available in all three test modes. Note: this change is not retroactive, WebRTC metrics will not be available for test runs from before this change was made.
  • WebRTC metrics are now available even if the test run has been aborted, as long as any WebRTC metrics have been collected at all.

January 2023 #

Fixes and improvements #

  • Upgraded to NightWatch version 2.5.2.
  • Added support for new browser versions: Google Chrome 109 and Mozilla Firefox 108, 109.
  • Changed the in-project navigation layout - the navigation bar that used to reside on the left side of the page is now at the top of it instead.
  • Added a statsType field to the WebRTC dump, this field corresponds to the RTCStatsType enum.
  • Deprecation warning! To move towards compliance regarding that the id field for RTC stats should not be predictable, we will be changing them from the current format (e.g., RTCOutBoundRTPAudioStream_4444) to an unpredictable name. The member field which follows the last hyphen will remain as it currently is.

New features #

  • Until now users were able to upload avatars only for their accounts. The project details tab under project settings has been updated, introducing the option for users to upload an avatar for projects as well. Only members with the Administrator role in the project have the ability to do this.
  • Users will now be re-directed to the last project they had open upon logging in. You may still access the projects list by clicking on the Loadero logo in the top left corner of the page.
  • Warning! We will soon change our WebRTC stats collection engine, which may cause some changes in the formatting of the WebRTC dump. Consumed RAM during the test may also change because of this due to stat calculations no longer being done on the participant’s instance.

November 2022 #

Fixes and improvements #

  • Improved Loadero’s homepage about performance testing. You can check it out at Loadero Performance Testing.
  • Added support for new browser versions: Google Chrome 107, 108 and Mozilla Firefox 106, 107.

New features #

September 2022 #

Fixes and improvements #

  • Upgraded to NightWatch version 2.3.0
  • Added support for six new browser versions: Google Chrome 104, 105, 106 and Mozilla Firefox 103, 104, 105.
  • Warning! API endpoints /projects/{projectID}/tests/{testID}/runs/ and /projects/{projectID}/runs/ to read all runs will have its default result order changed to ascending sorted by id.
  • Reduced test start times - on average the tests will now spend about one minute less in the “Initializing Environment” status.
  • Fixed issue with Navigator.deviceMemory browser property not beeing initialized properly.

New features #

June 2022 #

Fixes and improvements #

  • Added support for five new browser versions: Google Chrome versions 102 and 103 and Mozilla Firefox versions 100, 101, 102.

New features #

  • TimeExecution custom commands results (called timecards) are now accessible through the API. Available via
    • GET /v2/projects/{projectID}/runs/{runID}/results/{resultID}/
    • GET /v2/projects/{projectID}/tests/{testID}/runs/{runID}/results/{resultID}/
  • Added 1st and 5th percentile aggregator functions to post run assets and run and participant result metrics.
  • Added a new page about our test creation offer.

April 2022 #

Fixes and improvements #

  • Added support for Google Chrome 101 and Mozilla Firefox 99 browsers.
  • Enabled /dev/shm usage for tests. If your application utilizes /dev/shm an increase in RAM usage can be expected.
  • Improved API performance by upgrading to Go 1.18
  • Deprecation warning! We will be fully deprecating profile_parameters field from run participant resource. The information has been moved to the participant_details field. Deprecation will affect these 4 routes:
    1. GET /v2/projects/{projectID}/runs/{runID}/results/
    2. GET /v2/projects/{projectID}/runs/{runID}/results/{resultID}/
    3. GET /v2/projects/{projectID}/tests/{testID}/runs/{runID}/results/
    4. GET /v2/projects/{projectID}/tests/{testID}/runs/{runID}/results/{resultID}/
  • Published a new blog post: How We Create Load Tests For Our Customers In Loadero.

March 2022 #

New features #

  • New script variables are available. Visit our wiki to learn about the available variables in Javascript, Java and Python:
    • compute_unit (String): compute unit of given participant
    • test.participant_timeout (Number): participant timeout in seconds of test
    • test.start_interval (Number): start interval in seconds of test
    • test.total_participants (Number): total participant count in the test
  • JSON web token libraries added. Visit out wiki to learn about the all available libraries available in Javascript, Java and Python.
  • Custom SSL certificate support in your tests. Contact support to find out how you can use this feature while it’s not yet available in UI!

Fixes and improvements #

  • Added support for Google Chrome 99, Google Chrome 100 and Mozilla Firefox 98 browsers.
  • Blog post about testing screen sharing got published.

February 2022 #

New features #

  • Added a custom command to measure the execution time of a function. Visit our wiki to learn about the command in Javascript, Java and Python.

Fixes and improvements #

  • Updated Selenium to version 4.0.0.
  • Added support for Google Chrome 98 and Mozilla Firefox 97 browsers.
  • Migrated Loadero application to a new domain app.loadero.com.

January 2022 / December 2021 #

New features #

  • Added graph granularity for WebRTC and machine statistics graphs. This allows increasing the graph’s delta time value to enable viewing the graph at a larger scale.
  • Session recordings filenames now contain a UNIX timestamp - recording start time.
  • Machine statistics can now be downloaded as JSON data. The data can be found in the participant’s results report under the “Logs” tab.

Fixes and improvements #

November 2021 #

New features #

  • Added new video feeds with reduced framerates and video feeds that are made to resemble video content from an online meeting.
  • Test results are now made available even if participants didn’t launch in case a test was aborted.

Fixes and improvements #

  • Improved performance making the results views load faster and optimizing the way AWS instances are run.
  • Created a new blog post entry in our blog series - A Beginner’s Guide To Test Automation With Javascript Part 4.
  • Fixed an issue that packetLoss/in/percent and packetLoss/out/percent metrics were calculated incorrectly. All existing packet loss percent metrics are re-calculated to be correct.
  • Added support for Mozilla Firefox 94 and Google Chrome 96 browsers.

October 2021 #

New features #

  • Added a metrics table for WebRTC statistics to the results reports page of performance and load tests.
  • Added new asserts for video frame width and height and WebRTC connection count.
  • Added static IP for Ohio region. All participants configured to run in US East - Ohio region will have IP: 3.20.19.36.

Fixes and improvements #

  • Added support for Mozilla Firefox 93, Google Chrome 95 browsers.
  • Bug fixes and improvements to Loadero web application.

September 2021 #

New features #

  • Added VAT status validation to prevent automatic subscription creation with an invalid VAT ID, which would result in our users paying extra when they should not.

Fixes and improvements #

  • Deprecation warning! We will be making result start and end times nullable. By the end of October we will be removing machine_stats from GET /projects/{projectID}/tests/{testID}/runs/{runID}/results/statistics/. Instead the data will be available in the same request under the metrics key.
  • Added support for Mozilla Firefox 92, Google Chrome 93 and 94 browsers.
  • Bugfix for empty classification filters returning no results.
  • More information can be found in our blog post.

August 2021 #

New features #

  • Added support for choosing audio and video feeds separately. This will allow using different combinations of audio and video inputs.

Fixes and improvements #

  • Added support for Mozilla Firefox 91 browsers.
  • Updated post-run assert value units. Updated assert value units can be found in our wiki page about post run assertions. Existing asserts do not require any changes, their expected values have been updated to the new units accordingly.

July / June 2021 #

New features #

  • Added a custom command for closing alerts to all languages supported in Loadero.
  • Added WebRTC metrics graphs for test participants with Mozilla Firefox.
  • Added new assert paths for WebRTC inbound jitter values and for machine network packets values.
  • Created a new blog post entry in our blog series - A Beginner’s Guide To Test Automation With Javascript Part 3.
  • Added expected VAT amount in project billing tab.

Fixes and improvements #

  • Updated Nightwatch.js to v1.7.6.
  • Added support for Google Chrome 92 and Mozilla Firefox 90 browsers.
  • Updated our application design to unify button styles.

May 2021 #

New features #

  • FireFox WebRTC dump key mapping to use the same format as Google Chrome WebRTC dumps have. This allows to use the same code to parse both dumps, simplifying any processing.
  • Aborted test run results - we’re now collecting the data from aborted test runs and displaying it. This allows to gain more visibility in the test run even if it was stopped during execution.

Fixes and improvements #

  • Major performance improvements in time to display results for large test runs.
  • Added support for Google Chrome 91 browser.

April 2021 #

New features #

  • Python language support for test script creation - we’ve added this language to the 2 existing languages in Loadero. Now you can choose among JavaScript, Java or Python to write your tests in.
  • Added new metric paths that will allow creating new post-run asserts.
  • FAQ section for pricing related questions in our landing page.
  • New media files have been added to run variety of scenarios in your WebRTC tests.

Fixes and improvements #

  • Improvements in result polling for large load tests.
  • Added support for Google Chrome 90 and Mozilla Firefox 88 browser.
  • Automatic request retry in our web app.
  • Performance improvements in web app result views.
  • Bugfixes and improvements in the checkout flow in web app.

March 2021 #

New features #

  • In-app payments - We have finally added a long awaited feature of in-app payments. This allows users to upgrade their plan in Loadero app, instead of contacting support to do that.
  • We have created a WebRTC testing guide where we have gathered tips and tricks for testing WebRTC applications. Download yours here.

Fixes and improvements #

  • Added support for Google Chrome 89 and Mozilla Firefox 87 browser.
  • We added a safeguard that forbids deleting a test while a run is active for that test.
  • We added sort functionality in test results - participants list can now be sorted by a specific property to locate interesting participants faster.
  • Deprecation warning! We will be changing the default value of participant compute unit from G2 to G1. This change only affects API users, and only in case compute units are not otherwise specified.

February 2021 #

Fixes and improvements #

  • Added support for Mozilla Firefox 86 browser.
  • We addressed issue with NightWatch Socket hangup. This is a well-known issue for NightWatch framework. While this is not a complete fix (the issue can be reproduced with too low compute unit assignment), it should only appear in rare cases.
  • Performance improvements on test initialization phase. This means that tests will spend less time in “Initializing Environment” status.
  • We have limited the maximum number of projects that can be created in a single account. We have seen some users abuse system by creating many trial projects, which will not be possible from now on. Each account can only create one trial project. If that project is paid, a new project can be created.
  • We have improved loading times for our frontend app, by adopting CDN network.

January 2021 #

New features #

  • New script editor - Script editing experience got way better with the new Monaco editor many of you are already familiar from Visual Studio Code.

Fixes and improvements #

  • Added support for Google Chrome 88 and Mozilla Firefox 85 browser.
  • Now previous position in the list when navigation around the web app is stored, for example, when exiting test edit view for a test that is located on the 2nd page, now will redirect to the 2nd page instead of the 1st.
  • Updated the way how error messages are presented in various forms to improve user experience of our web app.
  • Fixed a bug when duplicating test, groupless participants are not duplicated.
  • Improvements in web app performance and issue tracking to resolve incidents even quicker.
  • Renamed assert_paths to metric_paths and assert_key to metric_key (added redirect to the new routes during the transition period).
  • Deprecation warning! We will be deprecating machine_stats field in single participant results. The data will be moved to another endpoint, the link to the data will be available through log_paths.rru field in GET /v2/projects/{id}/tests/{id}/runs/{id}/results/{id}/ request response.

December 2020 #

New features #

  • Max available CPU - Now you can see max available CPU depending on the selected compute unit. Of course, this allows viewing CPU data in machine statistics graphs as well.
  • Filter result participants in web app - A long awaited and finally delivered feature, you can filter result participants based on different criteria, for example, status, browser or region.
  • Even more filtering options using API - Loadero API now allows to filter different resources in even finer detail.

Fixes and improvements #

  • Added support for Mozilla Firefox 84 browser.
  • Updated Nightwatch.js to v1.5.1.
  • Fixed an issue where setRequestHeader() custom command took into account header value capitalization even though it should be case-insensitive as per RFC 7230.
  • Fixed several minor frontend web app issues.

November 2020 #

New features #

  • Compute unit setting for participants - We added compute units setting in test participant creation, it allows to optimize cost of running the tests for the applications that don’t use much compute power, and test very resource demanding applications as well. We also updated our pricing, now it is based on the compute units usage instead of participants usage.
  • Video autoplay - We enabled video autoplay in both Mozilla Firefox and Google Chrome browsers for the Loadero test participants. If that is something that can impact your existing Loadero tests, make sure to review them and update accordingly.
  • Accepting untrusted SSL certificates - Sometimes when testing a website, accepting untrusted SSL certificates is necessary. Running a test in the Session recording mode or adding commands for taking screenshots previously helped to identify the need to accept. Now we enabled acceptSslCerts in our test runner for this.

Fixes and improvements #

  • Added support for Mozilla Firefox 83 and Google Chrome 87 browsers.
  • Fixed a server-error triggered due to syntax errors in JavaScript code.
  • Some fixes that improved the test runner stability.
  • Some smaller updates in the Loadero application and website pages.

October 2020 #

New features #

  • New media files for WebRTC testing - We have updated our built-in media files for simulating microphone and webcam inputs. Now all of them are running at 30 FPS with specific common audio files and frame markers that aid quality assessment.
  • Access to location for your tests - We added access to location for tests created in Loadero, now you can test websites that require location access.

Fixes and improvements #

  • Added support for Mozilla Firefox 81/82 and Google Chrome 85/86 browsers.
  • We have refactored project API access - it is simpler to add authorization for the requests you make to Loadero now. We also added an API endpoint for reading all classificator values at once, which will simplify the development of your tests if you are using Loadero API.

August 2020 #

New features #

  • Machine graphs - you can now inspect results for each individual test participant to see their machine resource usage over time.
  • Optional API parameters - When creating or modifying participant information, it is no longer necessary to provide location, network and media type values in the request, they will always be initialized to defaults (for location Oregon region will be used as the default).
  • Adjustable script modal - now you can see more of the script in a single view.

Fixes and improvements #

  • Added support for Mozilla Firefox 80 and Google Chrome 85 browsers.
  • Performance and stability improvements for Loadero API. This should result in improved app loading times as well as API response times.
  • We completed a major refactoring of Loadero backend infrastructure meaning that we will be able to deliver new awesome features to you even faster!
  • Updated setRequestHeader() and setUserAgent() custom commands so that you can modify User-Agent header value using any of the two commands.
  • Minor fixes as well as performance improvements for the Loadero web application.

July 2020 #

New features #

  • Export WebRTC charts images - now you can export each WebRTC chart as a PNG image so you can integrate them in your reports more easily.
  • A new custom command to change any HTTP request header - override any HTTP header value for the requests your service makes using setRequestHeader() function! This enables testing of different functionality that could only be tested when a custom header value is present. Read more about this command here for JavaScript and here for Java.
  • New API endpoints - allow you to read test run results without specifying the test ID in the request path- only project ID is mandatory. Check out our API documentation to find out technical details about the new endpoints.
  • New script variables - a unique identifier of the participant across all tests ever launched in Loadero (run participant ID) as well as a sequential number of the participant within the scope of the test (participant global ID). Read more about these two variables in our wiki for JavaScript and Java.
  • WebRTC internals dump in load tests - more data from your load tests including automatic post-run assertions for WebRTC metrics.

Fixes and improvements #

  • Added support for Mozilla Firefox 78 and 79 as well as Google Chrome 84 browsers.
  • Various bug fixes and performance improvements.

June 2020 #

New features #

  • Audio recording - allows recording full audio from a participant session.
  • Fully customized network conditions - we have added configuration options and functions that allow you to update only a specific parameter of the current network conditions, such as bitrate. Read more in our wiki about how to do that using JavaScript + Nightwatch and Java + TestUI.
  • Additional filtering options - added more filtering options for Machine Statistics overview in test results. While charts for these statistics are coming in the future, additional filtering options allow you to get a better overview of the resources used during the test.

Fixes and improvements #

  • We added support for Mozilla Firefox 77 browser
  • We have been working on multiple internal systems to reduce the time needed to notice and resolve different issues that can come up. With all these systems in place, we are able to monitor service status with much higher precision and are able to react to events faster than ever before.

May 2020 #

New features #

  • Additional aggregators for machine statistics - machine statistics now can be analyzed in more detail filtering by new categories (StdDev, Relative StdDev, 25%, 50%, 75%, 95%, 99%).

Fixes and improvements #

  • Added support for Mozilla Firefox 76 and Google Chrome 83 browser.
  • Fixed timing issue for linear increment strategy where some participants would start the test execution at the same time.
  • Improved media stream quality at the start of the test – media streams are at their top quality from the very beginning of test execution.
  • Fixed typo in one of our constant names: “asymmetric network”. We’ve stopped supporting the version with the typo and made sure the existing tests keep running even if you had been using it.
  • Fixed an issue when sometimes WebRTC statistics would not get recorded.
  • General performance updates for Loadero web application.

April 2020 #

New features #

  • waitForDownloadFinished() - a new custom command for “Javascript + Nightwatch” and “Java + TestUI” languages. This command will be helpful when downloading files in tests to ensure that the download has finished before the test exits.
  • Multiple downloads in Google Chrome - we’ve updated the options for the Google Chrome browser so that it now allows multiple downloads from the same page.
  • New media files - There are 3 new media types. These media types all have 1080p video feed with markers for visual verification of media quality. Each of the media types has an audio feed with different sound levels: -20dB, -30dB and -50dB.

Fixes and improvements #

  • We added support for Mozilla Firefox 75 and Google Chrome 81 browsers.
  • We’ve done improvements to Loadero’s test queue by improving the processing of aborted tests that were still in the queue. This means that the waiting time for the test execution to start has gotten even shorter!
  • Fixed an issue where WebRTC dump sometimes would not be available.
  • Fixed a problem that media type assert preconditions would not be evaluated.
  • Fixed an issue where double quotes in test name would cause the test to crash.

March 2020 #

New features #

  • New wiki - we’ve updated our documentation site! Check it out as you will definitely find something that is helpful to you. If you notice anything missing or unusual, don’t hesitate to let us know!
  • Multiple parallel tests - we have completed a major upgrade of the test run scheduling mechanism, now you can run tens or even hundreds of tests in parallel. This update applies to all test modes and, of course, to tests launched on custom AWS accounts.
  • Assert preconditions - we have added assert preconditions to our web application! Head to loadero.com to add preconditions to your existing asserts!
  • Resource ID - added resource ID subheader in our web app. This will help you to start using our API – just copy and paste ID where resource ID is needed.

Fixes and improvements #

  • Improved reliability of session record test mode for short tests – sometimes the recording was not stored.
  • Fixes and improvements for app loading screens, now when fetching data for logs and WebRTC statistics you will see a progress bar of how much of your data has been downloaded so far.
  • We added support for Mozilla Firefox 74 browser. Due to the current global situation, Google has decided to postpone the release of Google Chrome 81 browser hence new Google Chrome releases in Loadero will be postponed as well.