Random Notes

Load Testing Service Comparison

This post is sponsored by Dotcom-Monitor.


Load testing is an integral part of deploying any web service. It should be done already in the development phase to find bottlenecks and after deployment when users’ usage patterns are better known. That’s not where it stops, though, as load testing can also be used as a regular part of the web service’s maintenance. Deploying new features without checking their effect on the performance of the service can be a fatal mistake, which is why load testing could be very important when integrated with a continuous integration or continuous deployment system.

In this article, I will review four different load testing services (in alphabetic order): Spirent’s Blitz.io, Load Impact, SendGrid’s Loader.io, and Dotcom-Monitor’s LoadView. Disclaimer: I have used the free trial of all the services for the testing and compared all the features I can use. This means I cannot describe the features not available for free trials, except in very light detail.

Running Tests

All of the services offer the most basic load testing features, namely sending a ton of requests to given endpoints with a specified amount of clients. Each one also allows setting request metadata, such as custom request headers, HTTP Basic authentication, timeouts, etc. Load Impact and LoadView analyse the URL you provide and add all linked resources such as CSS, JS, and images as separate requests by default. With Blitz.io and Loader.io you have to do this manually.

Each of the load testing services support the usual GET, POST, PUT and DELETE requests with custom data to send in the request. LoadView also supports any other request methods like OPTIONS and PATCH via the EveryStep Scripting Tool that is described in more detail later. They’re not used that often, but it’s good to have them should they be needed.

Blitz.io control panel.

The normal interface for creating a test is by a web form where the URL(s) to hit and other settings can be configured, such as the one shown above. Blitz.io, Load Impact, and LoadView also offer alternative methods to edit the test execution. Blitz.io has a cURL-like command line where you can set the request URL and other parameters. Load Impact, on the other hand, has a Lua script interface where you can write your own test scripts that execute the actual test operations.

Load Impact scripting interface.

LoadView offers a Windows program called EveryStep Scripting Tool that can be used to generate scripts for the load testing service. It works by recording interactions you have with the tested service in your browser. That is to say, pages you visit, things you click or hover on, etc. This allows for recording a very realistic scenario of user interaction that can be replayed in a load test without having to write code yourself. It seems like a great idea, but currently it is Windows only, so as a macOS user I could not try it out. It’s also Internet Explorer only, so the tested service must have IE11+ support.

In addition to plain HTTP requests, LoadView also supports testing WebSocket connections. Since the EveryStep Scripting Tool records all interactions using an actual browser, it can also track WebSocket connections and messages. With the advent of more interactive websites with live updating sections, the testing of WebSocket connections is a great feature that no other reviewed load testing service has – in fact it was difficult for me to find even a command line load testing tool that has such support when I was testing Code::Stats’s live updates’ performance (eventually I used Tsung).

Some services may incorporate internal load handling to deal with situations with bigger load. For example, the service may return HTTP 200, but skip rendering some content to use less resources. This would be normally counted as a successful request, but should you wish to mark it as a failure, Load Impact and LoadView support checking that the response contents contain certain things. LoadView’s scripting tool even allows taking screenshots of areas in the page to check that they are rendered properly.

As an interesting note, Loader.io is the only tested service that does not allow the user to select which geographical region the requests are sent from – they are always sent from their US East Amazon datacenter. This can be a problem if the request is sent to the other side of the world, as it adds unneeded latency that you need to account for in the test.

Scheduling / Monitoring

As I mentioned before, load testing can be used as a regular part of a service’s development and maintenance lifecycle. To help with this, each of the tested services support recurring schedules for load tests (this feature is not live in LoadView yet, but will be added later this year). This is a good way to stay on top of your service’s performance over a long time period.

Blitz.io and Loader.io are just for load testing, but Load Impact and LoadView’s company Dotcom-Monitor also offer server monitoring features. Load Impact’s monitoring is limited to running a server agent that reports metrics such as CPU and memory usage during the load test, but Dotcom-Monitor offers a comprehensive set of different monitoring tools and alerting that can be used to ensure the availability of your service. Those tools are not the focus of this article though, so I won’t elaborate on them further.

Integrations

Each of the services has some way to integrate with e.g. clients, CI tools, and communication systems. All but LoadView have an API for custom integrations, though for Blitz.io I was unable to find an actual documentation of it, just documentations of API client libraries. LoadView has an API for all the other services Dotcom-Monitor provide, but not one for load tests.

As for ready made integrations, Blitz.io supports New Relic, Scout, and CopperEgg for reading monitoring data and including it in the test reports. Loader.io has two-way webhook support and integrates with Travis CI, CircleCI, Jenkins, TeamCity, and GitHub to launch tests on every new software version. It also supports pushing results to New Relic and a HipChat chatroom. Load Impact has builtin integrations to the CI systems CircleCI, Jenkins, and TeamCity, and can push results to Slack or HipChat, or via webhooks. It can also read data from New Relic. LoadView has Zapier support for triggering automated tests and can write test and monitoring data to New Relic.

Results and Reports

Tests are worthless if you cannot get actionable data out of them. That’s why it matters how data is presented and what kind of reports you can get from it. All of the services offer the usual graphs of the amount of live clients, response times, and errors. Blitz.io and LoadView also allow downloading these reports as PDFs. Load Impact’s results view is the most customizable, as you can add custom graphs.

LoadView results graph.

Often the builtin reports are not enough, though. Maybe you want to graph the data in a certain way or run deeper analysis over the datasets of multiple test results. For this reason, you can export the data from all the services except Loader.io. Blitz.io’s export feature is close to useless, though, as it only exports the data points of the graphs you can see in the result page anyway. Load Impact and LoadView on the other hand export information about every request which can be highly useful.

Pricing

Estimating pricing of the services is not trivial, because all of them have slightly differing views of how the test load is calculated. For example, with Blitz.io reporting 1,000 concurrent users, my site was receiving around 800 requests per second. This means that each user was requesting about 1 request per second. With LoadView, each user is actually one thread of execution on the load generating server that sends requests as fast as it can. So for example 1 user might send 10 or 20 requests per second depending on the structure of the test and the size of the requests. However, what follows is a valiant attempt at a rough estimate for a 10 minute load test where the load rises linearly, starting from 1 request per second. Each user requests one URL.

Service5k req/s50k req/sNotes
Blitz.io$29.99$269.99Monthly pricing. Blitz.io used to have a free tier but I could not find it any longer.
Loader.ioFree*/$99.95$99.95Monthly pricing. There is a free tier of 10k users, but it only allows 1 minute tests to 2 URLs.
Load Impact$159$699Monthly pricing.
LoadView$20$150Price per test, plus monthly service cost of $9.99.

The values above are rough estimates and should be checked by the user before signing up for any service.

Conclusion

The tested services all have their strengths and weaknesses. None of them stand out as a silver bullet, so a choice must be made based on the features required.

Loader.io is the “lite” version of load testers. It has the basic features, is not too costly, and the free tier is especially tempting for small services and free time projects. But there is no reporting or data export, which is a problem if you plan to do serious data analysis. Not being able to choose where to send the requests from geographically is an annoying issue.

Blitz.io is somewhat of an outlier here. It’s very similar to Loader.io in its feature set, adding the cURL-style interface, but even though it offers PDF reports and at least some type of data export, I don’t know if they are worth the cost difference when doing bigger tests. It is kind of stuck between offering a very basic no-frills service or a comprehensive testing platform.

Both Load Impact and LoadView are on the more featureful end of the services. They fill all the basic requirements, but offer additions such as response content validation and automatically analysing and downloading all of a page’s resources. Data export of every request’s details allows deeper custom analysis.

Load Impact’s defining features are the comprehensive results page, plentiful integrations, and the powerful Lua scripting interface that allows you to customize your tests and write test scenarios that are as complex as you require.

The EveryStep Scripting Tool really sets LoadView apart, though, by allowing recording complex user scenarios with no programming required. It can even be used to test WebSocket connections, which is not supported by any of the other services. Dotcom-Monitor’s other services are also attractive, as both load testing and monitoring services can be acquired from the same provider.