Added support for Google Chrome 124 and Mozilla Firefox 125.
Deprecation warning! We are planning to remove the media_type field from
participant and run_participant responses. If something you use is dependent
on reading this field, then you should stop doing so. This change will take
effect after 01.07.2024.
The success rate chart for each test in the tests list no longer represents a
trend across the past 7 days, but instead across the last 20 results, whenever
those may have been. Additionally, this graph will no longer take aborted runs
and runs that resulted in a server error into account.
When a user begins creating or editing an assert in the test builder, they
will now be forced to finish/cancel what they are doing before being allowed
to do anything else.
Location and filters are now preserved when opening the test builder from the
tests list or opening a run report from the recent runs list. If the user
chooses to navigate back, they will be directed to the same page with the same
applied set of filters as before.
All information on the Wiki pertaining to Nightwatch.js, TestUI, and Py-TestUI
has been completely re-structured.
Take a look!
Fixed an issue where certain Selenium errors in Py-TestUI runs would result
in Selenium logs that failed to actually describe anything that happened
during the test.
When upgrading a project, the UI will dynamically update when the
subscription is activated, instead of remaining in a static and often
misleading state of insisting that the subscription is not active yet.
Fixed a rare issue where sometimes the run metric aggregator selector (avg,
min, max, etc.) would not show up in the run report.
Dropdown elements (such as the sort strategy selectors in tests/runs lists)
now support keyboard inputs to aid option selection.
All informational banners have undergone a visual re-design.
Charts in the run report's summary have undergone a visual update.
machine/cpu/available/total and machine/ram/available/total asserts are
now calculated as intended again.
Various other small bug fixes and UX improvements.
You can now manage your own list of favorite tests. This feature can be
used to keep all the tests you frequently run in one place so that finding
them in your project is easier. Favorite tests are on a per-user basis,
multiple members of a project can have a different set of favorites - one
member favoriting a test will not add that favorite to your own list of
favorites. You can favorite a test by hovering over the test in the tests list
and pressing the star icon that appears.
Participants and asserts in the test builder can now be sorted by "Oldest".
Added support for Google Chrome 123 and Mozilla Firefox 124.
If a test is run with a large quantity of participants (multiple hundreds),
the physical locations of their servers will now be more evenly distributed
across the selected region. Beforehand the distributon may have been lop-sided
and overly concentrated in a specific section of the region. This should
improve the realism aspect of load tests.
Small fixes to make test runs less likely to result in a server error.
Added support for Google Chrome 122 and Mozilla Firefox 123.
Reduced likelihood of test runs under certain test configurations resulting in
a server error.
When in the midst of creating or editing a participant you accidentally click
outside the form, then you will no longer lose the changes you have made.
Instead you will be prompted to close either by saving changes or discarding
them.
Deprecation warning for Loadero API users: we plan to deprecate path_id
as an order_by option for asserts, having renamed it to path instead.
Similarly, we plan to remove the following filters for the run resource:
filter_started_from, filter_started_to, filter_finished_from,
filter_finished_to. These filters have been renamed to
filter_processing_started_from, filter_processing_finished_from, etc. We
are still maintaining backwards compatibility for the old names, but we
suggest migrating to the new names at your earliest convenience.
Warning: Loadero has now implemented result expiration, which will cause
logs, artifacts, and charts to eventually become unavailable. All runs from
before March 4th have a universal 90-day expiration period applied, with the
countdown starting on March 4th. All runs on March 4th or later have a
plan-specific artifact retention period. If you would like to hold onto this
information, we suggest you begin preparations for downloading this content
onto your local machine. Note that information such as success rate,
metrics, and assertion results will still remain even after expiration. You
will be able to see the expiration date if you navigate to any participant
result report's logs, artifacts, machine statistics, or WebRTC statistics tab.
Update to Loadero invoicing is coming soon which means that invoices will be
issued based on usage and may be sent more frequently than before.
Tests now show a success rate chart that displays the general trend of how
successful a test's runs have been within the past 7 days.
Added support for Google Chrome 121, Mozilla Firefox 122.
Performance tests are now able to support up to 200 participants (used to be
50). If you have any load tests with 50-200 participants, we suggest
switching these tests to performance test mode instead to make use of more
detailed metrics and logs.
The tests list has been totally revamped, not only visually but functionally
as well. Read about the changes in detail in our
monthly updates blog post.
Improved test run stability, reducing the likelihood of users encountering AWS
errors.
In December we had shortly switched over to using Chrome for Testing, however,
these browser versions resulted in degraded performance, which resulted in
false negatives in users' tests. We have reverted back to using regular Google
Chrome.
Various small fixes to the app's UI to ensure a more consistent and intuitive
user experience.
Projects using the Ultimate or Enterprise plan now have access to G8 and
G12 compute units, doubling the maximum amount of compute power you can
allocate to a participant.
Back when we implemented a new header in the run reports, you may have noticed
an Audio MOS field that was always blank. E-model MOS calculation has now
been implemented for all participants using Google Chrome, and the result of
this calculation will be visible in the Audio MOS field of your
participants' reports. Additionally, you are able to assert this value in your
tests via mos/audio/e-model/avg. Note: Mozilla Firefox does not provide
the required WebRTC metrics to support E-model MOS calculation.
You can now
receive and process email messages
within a Loadero test. This functionality allows to process time sensitive
information by emails such as verification links, authorization codes and
other similar actions.
Switched to using Chrome for Testing for participants that are using Google
Chrome.
Added support for Google Chrome 119, Mozilla Firefox 120.
Improved app responsiveness when rendering WebRTC charts and the WebRTC dump
preview for large amounts of data.
Various small improvements and fixes to the new test builder.
The process of changing a project's plan to a different one is now more
intuitive.
Improved Swagger documentation for assert, file, and group resources by adding
more informative descriptions on endpoints, their parameters and permitted
values.
Project members with the Administrator role can now set explicit limits for
the project in Project Settings -> Plan & Usage - this includes setting how
many compute units can be spent in one month, as well as how many compute
units are permitted within one test. The maximum permitted compute unit for a
participant and a maximum participant timeout value may also be set.
Added support for Google Chrome 118, Mozilla Firefox 119.
Small fixes to make errors/warnings in the app more informative on what the
problem was.
Various improvements to the UI, such as case-insensitive search functionality
for participants/asserts in the test builder, better formatted Time Execution
tables in the run report, as well as overall responsiveness improvements and
QoL changes.
Security improvements for authentication processes, including logging into the
app.
Improved the speed at which result statistics are retrieved for large runs,
making large-scale run reports load times faster.
Improved stability of instance start-up, reducing the likelihood of failing to
spin up the required instances for a run.
Added support for Mozilla Firefox 118, Google Chrome 117.
Stability improvements have been made across the whole Loadero platform,
improving test launch times and fixing even more potential issues during test
launch.
Updated the UI of recent runs page to include more relevant information as
well as generally update the UI.
Various small improvements to the user interface, including fixes for test
script modal and new test builder.
Deprecation warning! To move towards compliance regarding that the id
field for RTC stats
should not be predictable,
we will be changing them from the current format (e.g.,
RTCOutBoundRTPAudioStream_4444) to an unpredictable name. The member field
which follows the last hyphen will remain as it currently is. Use the internal
statsType to determine what values are represented.
Step time execution is now fully supported and you can get run metrics for the
execution times as well as assert that the timings for each participant are
within threshold.
Added support for Mozilla Firefox 116, Google Chrome 116, Mozilla Firefox 117.
Fixed a major issue with large-scale WebRTC tests having instances run out of
space to store WebRTC metrics. Long-duration, high-load WebRTC tests are now
significantly more stable.
Additional small fixes to increase the stability of high-load test runs.
Several improvements to the new test builder, most notably support for members
with Visitor role to be able to view the test if they have the new test
builder enabled.
Various small improvements to the user interface, including a re-design of the
Billing & Invoice tab of project settings.
Added additional filters to API endpoints: projectrunparticipant,
runparticipant, and result endpoints now support filter_compute_unit
parameter, to retrieve only participants with a specific compute unit value
(G0.5, G1, G2, G4, or G6). You can find more information about our
API here.
You are now able to set your cookie preferences in the Account Settings view
of the web app. You can also do this for the
landing page by clicking on Cookie Preferences in
the footer of the page.
The clipboard is now available for participants using Google Chrome, enabling
the usage of elements like Copy to Clipboard buttons.
Added support for Mozilla Firefox 115, Google Chrome 115.
Made even further improvements in result processing speed.
WebRTC statistics charts and machine statistics charts in run reports now
utilize a new library, with the intent to make charts more visually appealing.
Various bug fixes to the new test builder.
Fixed an issue where all of a user's projects, where they are the only member,
would get deleted if they attempted to delete their account but provided an
incorrect password.
When using Loadero-Python, you no longer need to provide an access token when
downloading logs or artifacts, as the access token used to initialize
APIClient is now recognized automatically.
Added the ability to focus elements in the landing page with TAB that were not
focusable before in this manner.
Added support for new browser versions: Google Chrome 113, Mozilla Firefox
113, Google Chrome 114.
Removed deprecated video files as video feed options for participants. This
change will not affect your tests, as the options were already getting changed
to non-deprecated equivalents server-side. This change simply makes it so that
these deprecated options are not selectable in the first place.
Increased the amount of space available for test runs on instances, decreasing
the risk of instance crashes for longer tests with a lot of metric collection.
Added more limitations to trials. Trial projects are now no longer able to set
G0.5 compute units for participants, tests are only allowed to contain up to
10 CUs, and only one test can be running at a time.
Re-designed the run result header in run reports and individual participant
reports.
Some minor UI changes in the project settings and account settings views.
The new test builder has been released to the public! You may enable it by
opening the account settings dropdown in the top right of the app and
selecting Enable New Test Builder. This will change the UI and overall
flow of test creation and test editing. Try it out and see what has changed!
You may switch back to the old test builder if the new test builder is not to
your liking. However, the old test builder will be discontinued later down the
line and all users will be migrated to the new test builder. New users will
start with the new test builder enabled by default.
Users with Administrator or Developer permissions are now able to create API
access tokens for the project. This functionality can be found in the API
access tab on the left side of the project settings. Note: This tab is
only available for projects with a paid plan enabled.
For tests where the Time Execution custom command is used, the individual
participant report presents the results of these calls in the form of a Gantt
chart. You no longer need to look at the Selenium log to find these values.
Added support for new browser versions: Google Chrome 111, Mozilla Firefox
111, Google Chrome 112, Mozilla Firefox 112.
We have re-designed the project creation process, introducing a brand new
project creation form. Note: along with this change we have also removed
the projects list view. As such, if you would like to create a new project or
navigate between existing projects, you must do so via the project selector in
the app's header.
Improved parallelization of test runs, you should find your tests stuck in
Pending status less often. Tip: if your test is stuck in Pending status
and participant locations do not matter to you, you may consider changing your
participant configurations to a different location - there may be room for new
tests in other locations.
When downloading a log or an artifact, you can now see the download progress.
Recently you may have noticed that tests take longer to initialize than they
used to. We have made improvements to bring this initialization time back down
to where it used to be.
We have also made improvements in the processing speed of results, which is
particularly relevant for load tests with hundreds or thousands of
participants.
You may now postpone the use of your account's trial. If you have a trial
available and decide not to use it when creating a new project, you can choose
to start a trial for this project later, as long as you have not begun
upgrading it to a paid plan. Note, that each account is only permitted one
trial.
Since WebRTC metrics have been made available for session recording tests, we
have also enabled WebRTC asserts in session recording tests as well.
Added support for new browser versions: Google Chrome 110 and Mozilla
Firefox 110.
We have updated how WebRTC metrics are collected during test runs. This can
impact RAM usage measurements since calculations are not done on the
participant's instance anymore during the test.
WebRTC metrics are now available for session recording tests as well, meaning
that WebRTC metrics are now available in all three test modes. Note: this
change is not retroactive, WebRTC metrics will not be available for test runs
from before this change was made.
WebRTC metrics are now available even if the test run has been aborted, as
long as any WebRTC metrics have been collected at all.
Added support for new browser versions: Google Chrome 109 and Mozilla Firefox
108, 109.
Changed the in-project navigation layout - the navigation bar that used to
reside on the left side of the page is now at the top of it instead.
Added a statsType field to the WebRTC dump, this field corresponds to the
RTCStatsType enum.
Deprecation warning! To move towards compliance regarding that the id
field for RTC stats
should not be predictable,
we will be changing them from the current format (e.g.,
RTCOutBoundRTPAudioStream_4444) to an unpredictable name. The member field
which follows the last hyphen will remain as it currently is.
Until now users were able to upload avatars only for their accounts. The
project details tab under project settings has been updated, introducing the
option for users to upload an avatar for projects as well. Only members with
the Administrator role in the project have the ability to do this.
Users will now be re-directed to the last project they had open upon logging
in. You may still access the projects list by clicking on the Loadero logo
in the top left corner of the page.
Warning! We will soon change our WebRTC stats collection engine, which may
cause some changes in the formatting of the WebRTC dump. Consumed RAM during
the test may also change because of this due to stat calculations no longer
being done on the participant's instance.
Added support for six new browser versions: Google Chrome 104, 105, 106 and
Mozilla Firefox 103, 104, 105.
Warning! API endpoints /projects/{projectID}/tests/{testID}/runs/ and
/projects/{projectID}/runs/ to read all runs will have its default result
order changed to ascending sorted by id.
Reduced test start times - on average the tests will now spend about one
minute less in the "Initializing Environment" status.
Fixed issue with Navigator.deviceMemory browser property not beeing
initialized properly.
Added support for Google Chrome 101 and Mozilla Firefox 99 browsers.
Enabled /dev/shm usage for tests. If your application utilizes /dev/shm an
increase in RAM usage can be expected.
Improved API performance by upgrading to Go 1.18
Deprecation warning! We will be fully deprecating profile_parameters
field from run participant resource. The information has been moved to the
participant_details field. Deprecation will affect these 4 routes:
GET /v2/projects/{projectID}/runs/{runID}/results/
GET /v2/projects/{projectID}/runs/{runID}/results/{resultID}/
GET /v2/projects/{projectID}/tests/{testID}/runs/{runID}/results/
GET /v2/projects/{projectID}/tests/{testID}/runs/{runID}/results/{resultID}/
Added graph granularity for WebRTC and machine statistics graphs. This allows
increasing the graph's delta time value to enable viewing the graph at a
larger scale.
Session recordings filenames now contain a UNIX timestamp - recording start
time.
Machine statistics can now be downloaded as JSON data. The data can be found
in the participant's results report under the "Logs" tab.
Fixed an issue that packetLoss/in/percent and packetLoss/out/percent
metrics were calculated incorrectly. All existing packet loss percent metrics
are re-calculated to be correct.
Added support for Mozilla Firefox 94 and Google Chrome 96 browsers.
Added VAT status validation to prevent automatic subscription creation with an
invalid VAT ID, which would result in our users paying extra when they should
not.
Deprecation warning! We will be making result start and end times
nullable. By the end of October we will be removing machine_stats from GET/projects/{projectID}/tests/{testID}/runs/{runID}/results/statistics/.
Instead the data will be available in the same request under the metrics
key.
Added support for Mozilla Firefox 92, Google Chrome 93 and 94 browsers.
Bugfix for empty classification filters returning no results.
Updated post-run assert value units. Updated assert value units can be found
in our wiki page about
post run assertions.
Existing asserts do not require any changes, their expected values have been
updated to the new units accordingly.
FireFox WebRTC dump key mapping to use the same format as Google Chrome
WebRTC dumps have. This allows to use the same code to parse both dumps,
simplifying any processing.
Aborted test run results - we're now collecting the data from aborted test
runs and displaying it. This allows to gain more visibility in the test run
even if it was stopped during execution.
Python language support for test script creation - we've added this
language to the 2 existing languages in Loadero. Now you can choose among
JavaScript, Java or Python to write your tests in.
Added new metric paths that will allow creating new post-run asserts.
FAQ section for pricing related questions in our landing page.
New media files have been added to run variety of scenarios in your WebRTC
tests.
In-app payments - We have finally added a long awaited feature of in-app
payments. This allows users to upgrade their plan in Loadero app, instead of
contacting support to do that.
We have created a WebRTC testing guide where we have gathered tips and tricks
for testing WebRTC applications. Download yours
here.
Added support for Google Chrome 89 and Mozilla Firefox 87 browser.
We added a safeguard that forbids deleting a test while a run is active for
that test.
We added sort functionality in test results - participants list can now be
sorted by a specific property to locate interesting participants faster.
Deprecation warning! We will be changing the default value of participant
compute unit from G2 to G1. This change only affects API users, and only in
case compute units are not otherwise specified.
We addressed issue with NightWatch Socket hangup. This is a well-known issue
for NightWatch framework. While this is not a complete fix (the issue can be
reproduced with too low compute unit assignment), it should only appear in
rare cases.
Performance improvements on test initialization phase. This means that tests
will spend less time in "Initializing Environment" status.
We have limited the maximum number of projects that can be created in a single
account. We have seen some users abuse system by creating many trial projects,
which will not be possible from now on. Each account can only create one trial
project. If that project is paid, a new project can be created.
We have improved loading times for our frontend app, by adopting CDN network.
Added support for Google Chrome 88 and Mozilla Firefox 85 browser.
Now previous position in the list when navigation around the web app is
stored, for example, when exiting test edit view for a test that is located on
the 2nd page, now will redirect to the 2nd page instead of the 1st.
Updated the way how error messages are presented in various forms to improve
user experience of our web app.
Fixed a bug when duplicating test, groupless participants are not duplicated.
Improvements in web app performance and issue tracking to resolve incidents
even quicker.
Renamed assert_paths to metric_paths and assert_key to metric_key
(added redirect to the new routes during the transition period).
Deprecation warning! We will be deprecating machine_stats field in
single participant results. The data will be moved to another endpoint, the
link to the data will be available through log_paths.rru field in
GET /v2/projects/{id}/tests/{id}/runs/{id}/results/{id}/ request response.
Max available CPU - Now you can see max available CPU depending on the
selected compute unit. Of course, this allows viewing CPU data in machine
statistics graphs as well.
Filter result participants in web app - A long awaited and finally
delivered feature, you can filter result participants based on different
criteria, for example, status, browser or region.
Even more filtering options using API - Loadero API now allows to filter
different resources in even finer detail.
Fixed an issue where setRequestHeader() custom command took into account
header value capitalization even though it should be case-insensitive as per
RFC 7230.
Compute unit setting for participants - We added compute units setting in
test participant creation, it allows to optimize cost of running the tests for
the applications that don’t use much compute power, and test very resource
demanding applications as well. We also updated our pricing, now it is based
on the compute units usage instead of participants usage.
Video autoplay - We enabled video autoplay in both Mozilla Firefox and
Google Chrome browsers for the Loadero test participants. If that is something
that can impact your existing Loadero tests, make sure to review them and
update accordingly.
Accepting untrusted SSL certificates - Sometimes when testing a website,
accepting untrusted SSL certificates is necessary. Running a test in the
Session recording mode or adding commands for taking screenshots previously
helped to identify the need to accept. Now we enabled acceptSslCerts in our
test runner for this.
New media files for WebRTC testing - We have updated our built-in media
files for simulating microphone and webcam inputs. Now all of them are running
at 30 FPS with specific common audio files and frame markers that aid quality
assessment.
Access to location for your tests - We added access to location for tests
created in Loadero, now you can test websites that require location access.
Added support for Mozilla Firefox 81/82 and Google Chrome 85/86 browsers.
We have refactored project API access - it is simpler to add authorization for
the requests you make to Loadero now. We also added an API endpoint for
reading all classificator values at once, which will simplify the development
of your tests if you are using Loadero API.
Machine graphs - you can now inspect results for each individual test
participant to see their machine resource usage over time.
Optional API parameters - When creating or modifying participant
information, it is no longer necessary to provide location, network and media
type values in the request, they will always be initialized to defaults (for
location Oregon region will be used as the default).
Adjustable script modal - now you can see more of the script in a single
view.
Export WebRTC charts images - now you can export each WebRTC chart as a
PNG image so you can integrate them in your reports more easily.
A new custom command to change any HTTP request header - override any HTTP
header value for the requests your service makes using setRequestHeader()
function! This enables testing of different functionality that could only be
tested when a custom header value is present. Read more about this command
here.
New API endpoints - allow you to read test run results without specifying
the test ID in the request path- only project ID is mandatory. Check out our
API documentation to find out technical
details about the new endpoints.
New script variables - a unique identifier of the participant across all
tests ever launched in Loadero (run participant ID) as well as a sequential
number of the participant within the scope of the test (participant global
ID). Read more about these two variables in
our wiki.
WebRTC internals dump in load tests - more data from your load tests
including automatic post-run assertions for WebRTC metrics.
Audio recording - allows recording full audio from a participant session.
Fully customized network conditions - we have added configuration options
and functions that allow you to update only a specific parameter of the
current network conditions, such as bitrate. Read more about how to do that
here.
Additional filtering options - added more filtering options for Machine
Statistics overview in test results. While charts for these statistics are
coming in the future, additional filtering options allow you to get a better
overview of the resources used during the test.
We have been working on multiple internal systems to reduce the time needed to
notice and resolve different issues that can come up. With all these systems
in place, we are able to monitor service status with much higher precision and
are able to react to events faster than ever before.
Additional aggregators for machine statistics - machine statistics now can
be analyzed in more detail filtering by new categories (StdDev,
Relative StdDev, 25%,50%, 75%, 95%, 99%).
Added support for Mozilla Firefox 76 and Google Chrome 83 browser.
Fixed timing issue for linear increment strategy where some participants would
start the test execution at the same time.
Improved media stream quality at the start of the test – media streams are at
their top quality from the very beginning of test execution.
Fixed typo in one of our constant names: “asymmetric network”. We’ve stopped
supporting the version with the typo and made sure the existing tests keep
running even if you had been using it.
Fixed an issue when sometimes WebRTC statistics would not get recorded.
General performance updates for Loadero web application.
waitForDownloadFinished() - a new custom command for "Javascript +
Nightwatch" and "Java + TestUI" languages. This command will be helpful when
downloading files in tests to ensure that the download has finished before the
test exits.
Multiple downloads in Google Chrome - we’ve updated the options for the
Google Chrome browser so that it now allows multiple downloads from the same
page.
New media files - There are 3 new media types. These media types all have
1080p video feed with markers for visual verification of media quality. Each
of the media types has an audio feed with different sound levels: -20dB, -30dB
and -50dB.
We added support for Mozilla Firefox 75 and Google Chrome 81 browsers.
We’ve done improvements to Loadero’s test queue by improving the processing of
aborted tests that were still in the queue. This means that the waiting time
for the test execution to start has gotten even shorter!
Fixed an issue where WebRTC dump sometimes would not be available.
Fixed a problem that media type assert preconditions would not be evaluated.
Fixed an issue where double quotes in test name would cause the test to crash.
New wiki - we’ve updated our documentation site! Check it out as you will
definitely find something that is helpful to you. If you notice anything
missing or unusual, don’t hesitate to let us know!
Multiple parallel tests - we have completed a major upgrade of the test
run scheduling mechanism, now you can run tens or even hundreds of tests in
parallel. This update applies to all test modes and, of course, to tests
launched on custom AWS accounts.
Assert preconditions - we have added assert preconditions to our web
application! Head to loadero.com to add preconditions
to your existing asserts!
Resource ID - added resource ID subheader in our web app. This will help
you to start using our API – just copy and paste ID where resource ID is
needed.
Improved reliability of session record test mode for short tests – sometimes
the recording was not stored.
Fixes and improvements for app loading screens, now when fetching data for
logs and WebRTC statistics you will see a progress bar of how much of your
data has been downloaded so far.
We added support for Mozilla Firefox 74 browser. Due to the current global
situation, Google has decided to postpone the release of Google Chrome 81
browser hence new Google Chrome releases in Loadero will be postponed as well.