Background
We were recently asked by a client to recommend a load testing tool for a web-based application that Finite Wisdom had developed for them. We narrowed our search among the many alternatives as follows:
We didn't consider open source offerings. We wanted more than code and “community support.”
We didn't consider products that tout any form of “open source inside” or “front-end to another tool” approach. We wanted end-to-end, single-source responsibility.
We considered only cloud-based tools. Their infrastructure, outside our firewall.
We considered only tools that explicitly offer a BLU (a.k.a. “Browser-Layer Users,” a.k.a. “Real Browser”) testing option. We saw many claims on Stack Overflow and elsewhere that product X or Y “can do that too,” but we were skeptical and cautious of needless complexity.
We avoided “one among our many tools” pitches. The feature cross-talk between those tools that inevitably creeps into their pitches quickly loses the reader.
This served to narrow our search down to three alternatives:
LoadView
LoadNinja
Loader
Let’s take a closer look at each of these three.
LoadView
Here are the pros and cons of LoadView:
Pros
Cloud-based
BLU option; over 40 specific browsers
Point-and-click scripting with “EveryStep Web Recorder”
“Quickly and Easily Build Test Scripts Without Touching a Line of Code”
We choose where load originates
Very helpful website
Extensive load testing links at the bottom of every website page
Specific discussions of AJAX-intensive load testing
Specific discussion of load testing for AngularJS, HTML5, Javascript, and JSON
Their Knowledge Base link wanders off into “another of our products” territory, but we think that load testing is in there
Extensive comparisons to competitor products
On-demand pricing available
Household name users mentioned on their site: Volvo, Comcast, Dell
Cons
Price is $200 to $1,500 per month
LoadNinja
Here are the pros and cons of LoadNinja:
Pros
Cloud-based
BLU option; limited screen resolutions
Point-and-click scripting with AI-enhanced “InstaPlay Recorder”
“Can create even the most complex test scripts and play them back in minutes without any additional effort”
Often mentioned in “best of” lists
Cons
Price is $1200 to $2400 per year; monthly price is not available
They choose where load originates; we saw no mention of user choices
Household name users mentioned on their site: none
Loader
Here are the pros and cons of Loader:
Pros
Cloud-based
We have another client that has used it, and offered this summary:
We ultimately selected loader.io because it was very cost effective and they have a free tier and it's completely cloud based and does not require using your own hardware and does not require coding test scripts. That being said, it does have limitations; it's definitely more of a load testing solution than an automated testing solution as you can't really make it click on certain things or post data/parameters. Essentially it offers 3 modes of load testing: users per second, users per test which increments (over a time period) and static load over a time period. It will report back in real-time the server response time and amount of users being sent to the application.I don't have a good comparison because other cloud based solutions like this were incredibly expensive so we immediately omitted them from our search, and we didn't want to maintain our own hardware.
Price is $0 to $100 per month
Cons
No explicit BLU option
If BLU is a must-have, then perhaps a different 3rd suggestion is warranted
Rather basic; in their words, “simple”
Recommendation
Our initial recommendation to the client was to use the the LoadView product, as It is very robust, allows per-month licensing, and the price per month appeared reasonable. We went down that route and found that the product was indeed robust, but their pricing was deceiving, hard to determine with certainty and appeared quite likely to become untenable.
We therefore stopped work with LoadView and started over with LoadNinja. It proved to be easy to use, fully capable of all that we wanted to test, and came at a reasonable price: $250/month with the ability to start/stop billing as necessary. So, we enable billing when a new test cycle begins and disable it once the test cycle is complete.
LoadNinja has worked well for us.