INTRODUCTION
There’s a lot of confusion and uncertainty when it comes to choosing performance testing tools. Most key performance indicators that you should be testing are already quantifiable, but most tools work only with HTTP traffic. Others are built for SOAP. Some sell extensions for specific platforms, and others give you the data in Python but not in Java…and those were just three of the major problems I ran into when I was writing a benchmarking tool for my own purposes. The purpose of this list is to provide a quick side-by-side comparison of tools written entirely in open source. Anything not applicable is left out and discussed why below. And when we run across two or more solutions for a particular problem, we review them based on size (downloads), how easy it is to pick up, what kind of data they generate, where it’s displayed, frequency of updates, and other similar factors to try and figure out the best fit for you among tool choices…all while trying to stay impartial as possible.
Are there really any performance testing tools that are 100% open source? Measuring the performance of a Web site involves analyzing the performance data obtained. So how can you measure and/or analyse the performance of a Web site using 100% open source tools?
Performance testing is becoming an integral part of the development process, so it is essential to know what tools are out there. This post contains a list of top 10 open source performance testing tools.
Open Source Performance Testing Tools
JMeter
The Apache JMeter application is open source software. It is a pure Java application designed to load test an application and measure its performance. Read More »
Gatling
Gatling is a highly capable load testing tool. It is designed for ease of use, maintainability and high performance. Read More »
Locust
Locust is an easy-to-use, distributed, user load testing tool. It is intended for load-testing websites (or other systems) and figuring out how many concurrent users a system can handle. Read More »
Tsung
Tsung is an open-source multi-protocol distributed load testing tool. It can be used to stress HTTP, WebDAV, SOAP, PostgreSQL, MySQL, LDAP, MQTT, and Jabber/XMPP servers. Read More »
Siege
Siege is an HTTP load testing and benchmarking utility. Siege supports basic authentication, cookies, HTTP, HTTPS and FTP protocols. It lets its user hit a server with a configurable number of simulated clients. Read More »
Httperf
Httperf is a tool for measuring web server performance. It provides a flexible facility for generating various HTTP workloads and for measuring server performance. Read More »
Taurus
Although not specifically related to Perf testing, Taurus provides an automation-friendly framework for continuous testing, including functional and performance. Read More »
Artillery
Artillery is a modern, powerful & easy-to-use load testing and functional testing toolkit. Use it to ship scalable applications that stay performant & resilient under high load. Read More »
Goad
Goad takes full advantage of the power of Amazon Lambdas for distributed load testing. You can use goad to launch HTTP loads from up to four AWS regions at once. Each lambda can handle hundreds of concurrent connections, able to achieve peak loads of up to 100,000 concurrent requests. Read More »
Apache Bench
ab is a tool for benchmarking your Apache Hypertext Transfer Protocol (HTTP) server. It is designed to give you an impression of how your current Apache installation performs. Read More »
Remember that top engineering organizations consider performance not as nice-to-have, but as a crucial feature of their products. But most engineering teams do not regularly test the performance and scalability of their infrastructure, and most lack the tools to properly do so.
Although you can find plenty of commercial tool options out there, for the right organization, free and open-source tools may be a good alternative—or the perfect complement to your commercial tool set.
The following is a list of open source performance testing tools. The tools listed here are the tools we used ourselves, at Conversant, and can confidently recommend. Keep in mind that our testing tools are constantly evolving, so this list will update periodically.
The value of performance
Before talking tools, let’s consider load times and the value of performance. When you understand that performance is key to a great user experience, you need tools that will do one thing very well, and that’s to measure the user’s perceived load time.
There are many performance rules out there, but ultimately, load time is the only performance metric that matters.
How fast is fast enough for a web application? Here’s a quick overview of key performance metrics:
- Under 100 milliseconds is perceived as instantaneous.
- A 100ms to 300ms delay is perceptible.
- One second is about the limit for the user’s flow of thought to stay uninterrupted.
- Users expect a site to load in 2 seconds.
- After 3 seconds, 40% of visitors will abandon your site.
- 10 seconds is around the limit for keeping the user’s attention.
So, as you can see, most software users want instant response.
Because of this, the importance of performance is increasing, and engineering teams need to treat performance as a feature. The goal of a performance load test is to understand how your applications behave under heavy load conditions.
To get started, you need to understand the baseline performance of your application and know that the performance of each transaction is unique.
For example, in an e-commerce application, a home page transaction is likely highly cached and very fast, whereas a checkout transaction is more complicated and must talk to a payment service, shipping service, etc.
To ensure that users have a great experience, you must test the most common flows for your users and understand performance both in the browser and on the server. To get the job done, you’ll need server-side, client-side, and performance tools, and you can find free and open-source options that fall into each of these categories.
But before jumping into load-testing tools, don’t make the mistake of overlooking how your application performs with just one user.
Don’t ignore one-user performance
Believe it or not, most folks skip the most obvious place to start with performance—verifying that their application can perform well for one user before even starting concurrent load or stress testing.
James Pulley, practice manager of performance engineering and test at TEKsystems Global Services and a podcaster at PerfBytes, reminded me in our last TestGuild performance podcast that most engineers focus only on how their applications perform for many concurrent users.
How many, though, can answer whether the app scales for only one person? You have to ask that question before even thinking about a load test.
Tools for understanding client-side performance
Most modern applications spend more time in the browser than on the server side. The reason is that newer applications now are using JavaScript frameworks such as React and AngularJS.
These feature-rich front ends add a new layer that needs to be measured. So measuring the rendering time of an application that is under test on a client’s local machine becomes critical.
Two of the best tools to use to understand client-side performance are:
- Google PageSpeed Insights, a service that analyzes the content of a web page and generates suggestions to make your pages load faster. Reducing page load times reduces bounce rates and increases conversion rates.
- Google Lighthouse is an open-source, automated tool for improving the quality of web pages. Your front-end developer should at a minimum be using Lighthouse metrics, which are available in Google Chrome tools.
Tools for understanding real-world performance
Sitespeed.io is my favorite tool for evaluating client-side performance from real browsers.
This open-source tool analyzes your website’s speed and performance based on performance best practices and timing metrics. You can analyze one site, analyze and compare multiple sites, or let your continuous integration server break your build when you have exceeded your performance budget.
It is not always possible for teams to modify the applications to optimize client-side performance. Fortunately, Google invested in making ngx_pagespeed and mod_pagespeed as web server extensions to automate performance improvements without the need for code changes.
Here’s a description of each, as well as one other useful tool:
- Google ngx_pagespeed speeds up your site and reduces page load time. And this open-source nginx server module for this Pagespeed tool automatically applies web performance best practices to pages and associated assets (CSS, JavaScript, images) without requiring you to modify your existing content or workflow.
- Google mod_pagespeed also speeds up your site and reduces page load time. This open-source Apache HTTP server module automatically applies web performance best practices to pages and associated assets (CSS, JavaScript, images) without requiring that you modify your existing content or workflow.
- WebPagetest.org provides deep insights into the performance of the client side in a variety of real browsers. This utility will test a web page in any browser, from any location, over any network condition—and it’s free.
Reuse your functional tests to get performance data
When getting started with performance testing, don’t overlook another asset your software team probably already has—a functional testing script.
Some open-source functional test tools can be leveraged to gather some high-level performance info that should help your team.
- Cypress has a method that can capture Google Chrome’s performance data for test runs. This allows you to find and fix common performance issues such as slow-time-to-first-byte.
- Selenium has a few ways to capture performance data as well. One common approach is using a HAR file. This is done using the BrowserMob Proxy (BMP). BMP lets you manipulate HTTP requests and responses, capture HTTP content, and export performance data as a HAR file.
- To use JMeter with Selenium Webdriver you can install the JMeter Webdriver Sampler. This is useful for testing the performance of AJAX, GWT-based web applications, and simulated user actions.
Once you have a good understanding of your client-side front-end performance, it’s time to move on to some server-side load testing tools.
Tools for understanding server-side performance
Apache Bench and Siege are great for quick load tests from a single endpoint. If you just need to get a sense of the requests per second for an endpoint, these are great solutions.
A more advanced approach—and my personal preference—is Locust.io, an open-source load-testing framework that allows complex transactions and can generate high levels of concurrency with ease. I’ve included it in the list below, along with a few other tools to consider.
- Locust.io is great for understanding the performance on the server side.
- Bees with Machine Guns‘ authors describe it as “a utility for arming (creating) many bees (micro EC2 instances) to attack (load test) targets (web applications).”
- Multi-Mechanize is an open-source framework for performance and load testing that runs concurrent Python scripts to generate load (synthetic transactions) against a remote site or service. It’s commonly used for web performance and scalability testing, but you can also use it to generate a workload against any remote API accessible from Python.
- Siege is an HTTP load-testing and benchmarking utility designed to let web developers measure code under duress, to see how it will stand up to load on the Internet. Siege supports basic authentication, cookies, and HTTP and HTTPS protocols, and lets the user hit a web server with a configurable number of simulated web browsers. Those browsers place the server “under siege.”
- Apache Bench is useful for benchmarking your Apache HTTP server, to get an idea of how Apache performs.
- Httperf provides a flexible facility for generating varied HTTP workloads and measuring web server performance. The focus is not on implementing a particular benchmark but on providing a robust, high-performance tool that facilitates the construction of both micro- and macro-level benchmarks. The three distinguishing characteristics of httperf are its robustness, which includes the ability to generate and sustain server overload; its support for the HTTP/1.1 and SSL protocols; and its extensibility to new workload generators and performance measurements.
- Apache JMeter is for testing performance both on static and dynamic resources (files, servlets, Perl scripts, Java objects, databases and queries, FTP servers, and more). You can also use it to simulate a heavy load on a server, network, or object to test its strength or analyze overall performance under different load types. Finally, consider using it to make a graphical analysis of performance or to test your server/script/object behavior under a heavy concurrent load.
- Element is an open-source Puppeteer node library that uses a browser-based load-testing tool. You create scripts using Typescript, and can use them against your web app in the same way that your customers do, opening a browser and interacting with page elements.
Choices and trade-offs
While you have many open-source tools from which to choose for performance management, it’s not always wise to build and manage your own performance testing tools and infrastructure.
You may determine that it’s not worth the engineering resources required to load test when you can pay for a proven commercial product or service. Such tools let you more easily build, execute, and analyze performance tests, but you have to have the budget pay for it.
Should you go with open source, or will a commercial tool provide more value? Only you can decide if it makes sense to use open source—and if your organization has the engineering resources required. If you do, many of the tools I’ve discussed above should go right into your toolbox.
Taken together, these free and open-source tools offer a path to get started on capacity planning and load testing the server side, optimizing and performance testing the client side, and monitoring performance from end to end to derive meaningful insights from performance tests.