What is pnv testing?

What is Agility? Why is everyone talking about Agile these days? What are the outcomes we get after we implement Agile practices? Do we. This kind of testing could be testing the functionality of the system or it could be testing the usability or both. During each subsequent prenatal visit (PNV), urine is again checked for protein and sugar.

This practical book focuses on how to use Apache JMeter to meet your testing needs. It starts with a quick introduction on performance testing and then presents more in detail topics like recording test scripts or monitoring system resources. It also explores related activities like using the cloud for testing or extending Apache JMeter with plugins. You will also get the basic knowledge on how to use tools such as Vagrant, Puppet and Apache Tomcat.

This book is well written and provides a solution-oriented perspective on performance testing with JMeter with a use case that is explained throughout the book. I will recommend it to every software tester or software developer that needs to perform some performance or load testing activities. If you don’t want to install JMeter on your machine, you can also explore the free pricing options available from hosted load testing tools vendors that are based on JMeter in our article Free Web Load Testing Services.

Reference: Performance Testing with JMeter 2.9, Bayo Erinle, Packt Publishing, ISBN 978-1-78216-584-2

Quotes

At a very high level, performance testing is always almost conducted to address one or more risks related to expense, opportunity costs, continuity, and/or corporate reputation. Conducting such tests help give insights to software application release readiness, adequacy of network and system resources, infrastructure stability, and application scalability, just to name a few. Gathering estimated performance characteristics of application and system resources prior to the launch helps to address issues early and provides valuable feedback to stakeholders, helping them make key and strategic decisions.

Performance testing is usually a collaborative effort between all parties involved. Parties include business stakeholders, enterprise architects, developers, testers, DBAs, system admins, and network admins. Such collaboration is necessary to effectively gather accurate and valuable results when conducting testing. Monitoring network utilization, database I/O and waits, top queries, and invocation counts, for example, helps the team find bottlenecks and areas that need further attention in ongoing tuning efforts.

Monitoring servers during test executions helps identify potential bottlenecks in the application or system resources. It can draw focus to long-running queries, insufficient thread and data source pools, insufficient heap size, high I/O activity, server capacity inadequacies, slow-performing application components, CPU usage, and so on. All these are important to troubleshooting performance issues and attaining the targeted goals.

There will come a time when running your test plans on a single machine won’t cut it any longer performance-wise, since resources on the single box are limited. For example, this could be the case when you want to spin-off a thousand users for a test plan. Depending on the power and resources of the machine you are testing on, and the nature of your test plans, a single machine can probably spin-off with 300-600 threads before starting to error out or causing inaccurate test results. There are several reasons why this may happen. One is because there is a limit to the amount of threads you can spin-off on a single machine. Most operating systems guard against complete system failure by placing such limits on hosted applications. Also, your use case may require you to simulate requests from various IP addresses. Distributed testing allows you to replicate tests across many low-end machines, enabling you to start more threads and thereby simulating more load on the server.

It is none of the above. My friend Mazdak Abtin, one of the greatest Agile Coaches in Melbourne, uses a very intriguing metaphor to explain agility. Which one do you think is more Agile – A kangaroo, a car, or a bullet train?

And the answer of course is the Kangaroo. Agility is not about speed, it is about changing direction. It’s about being nimble and light-footed. Of course the bullet train is the fastest of all, but it is the least agile.

The goal of agility in organizations is to make it easier to change direction. The environments that we are operating in are changing so fast that Agility today is not an option, it is required for survival.

When we want to change direction fast, especially when we have large scale and complex software solutions, the ability to automate testing becomes imperative.

We want to experiment fast, and implement changes fast. So we need to make sure not only regression testing can be done as quickly as possible, but also progression testing can be done quickly. In addition, we need to be able to maintain those automated tests with ease. At the end of the day, it’s all about being light-footed and nimble 😉

In Agile we talk about the concept of shift left in software testing. In this approach, we try to shift testing to as far left as possible. This shifting left is normally achieved via automation of testing and early collaboration.

The diagram below shows how the simplified V-Model of how software testing looks like when we apply the shift left mentality (as opposed to the shift right approach):

Agile takes test automation to the next level by merging the testing process with the requirements definition process. When we talk about shift left we are not only talking about writing our automated tests first and then writing code. We are actually talking about using tests to define the features that we are building.

We call this specification by example. One of the methodologies we use to define specification by example is Acceptance Test Driven Development (ATDD).

In Acceptance Test Driven Development, our Agile team (which may include testers, developers and analysts) works closely and collaboratively with stakeholders or customers to define the requirements using acceptance tests. In this technique, numerous workshops are held before the coding starts, where the team writes acceptance tests for the features that are going to be built. Then these acceptance tests are going to be used as the requirements definitions to write code against it.

Then these acceptance tests are going to be automated while the code is being written or even perhaps before the code is written. So initially because there is no code yet, all the tests are going to be failing. As our features get built one by one, these tests start to pass. Until they are all green 🚦

This means we can use the percentage of the passing test also as a reporting metric to figure out what percentage of the features have been built.

If we simply follow the process of automating our acceptance tests first and then writing code, it does not mean we are doing Acceptance Test Driven Development. Acceptance Test Driven Development has the following four tenets as defined by Large Scale Scrum.

In Agile environments, we should be using some variation of ATDD process for all levels of testing. For example, we could use Test Driven Development (TDD) for our unit testing. And on the system level, we can use Behaviour Driven Development (BDD) to write our automated system tests.

With BDD we could use given/when/then format to define our requirements and test cases on a system level. BDD helps us have conversations between developers, testers, customers, and other stakeholders using a common language and aligns us with our system behaviors. Tests written in this format can also be easily automated using various test automation tools that are available in the market.

As it was mentioned at the beginning of this article, to maximize agility, it is imperative to automate as much as possible. Automation is not only for unit tests, system tests, and acceptance tests.

Elisabeth Hendrickson, the author of the mini-book Exploratory Testing in an Agile Context, dares to state that: “I do think that if you can write a manual script for a test, you can automate it.”

We need to be automating on all levels for both regression and progression tests. We need to bring in the culture of automation and shift left to everything from Security testing, to PnV, QA, and Compliance testing.

This practice is also called “building quality in”. Testing done at the end will not ensure quality. Testing at the end is an inspection process to prevent low quality products from being released. But it will not ensure our product is built with quality. By using the techniques we have defined here, we will be building quality in our product.

But which performance testing types should you conduct, what’s the difference between load testing and stress testing, and which test is suitable for which situation? In this blog post, we’ll cover the answers to these questions and more.

Table of Contents:

While there are those who compare all three types of testing, the more popular comparisons that testers make include:

🚀 Try BlazeMeter today for performance testing at scale >>

Let us take a closer look at the above comparisons.

Load testing is the process of checking the behavior of the system under test under the anticipated load. For example, the piece of software under test is designed to serve X users (because it is an internal product of an enterprise and there are no more employees), so it does not make sense to conduct testing under a higher load. Therefore, it is sufficient to check if the performance is good enough and matches non-functional requirements or service level agreements.

So, load testing in a nutshell consists of:

While load testing simulates real-life application load, the goal of software stress testing is to identify the saturation point and the first bottleneck of the application under test.

An ideal application behaves in the following manner:

The above points are true to a certain extent. However, at some point you will see that while you are adding more and more virtual users, the number of requests per second remains the same or even goes down due to increased response time. Bottlenecks can happen during this stage with errors and even stop serving incoming requests entirely.

Therefore, the main way to differentiate between these two types of testing is by focusing on their end goal.

To summarize, load testing and stress testing are two popular performance testing types that each focus on different application behaviors, such as a system’s general behavior under load or the upper limits of a system’s load capacity.

With the comparisons out of the way, let us explore each type of testing on its own.

Different types of performance testing provides you with different data, as we will further detail.

Before performance testing, it is important to determine your system’s business goals, so you can tell if your system behaves satisfactorily or not according to your customers’ needs.

After running performance tests, you can analyze different KPIs, such as the number of virtual users, hits per second, errors per second, response time, latency, and bytes per second (throughput), as well as the correlations between them. Through different test reports, you can identify bottlenecks, bugs, and errors, then decide what needs to be done.

Run performance tests when you want to check your website and app performance, which may extend to testing servers, databases, networks, etc. If you follow the waterfall methodology, test at least once before you release a new version of your application. If you’re shifting left and going agile, you should test continuously.

The following figure shows an example of a performance testing report on BlazeMeter. This is a good test, given the growing number of users does not affect the response time, the error rate remains low, and the hits per second are in line with the number of virtual users.

In other words, the test measures how systems handle expected load volumes. There are a few types of open-source load testing tools, JMeter being the most popular.

Load test when you want to determine whether your system can support the anticipated number of concurrent users. You can configure tests to simulate various user scenarios which can focus on different parts of your system (such as a checkout page, for example).

You can determine how the load behaves when coming from different geo-locations or how the load might build up, then level out to a sustained level.

Load tests should be performed all the time in order to ensure your system is always on point, which is why it should be integrated into your continuous integration cycles (utilizing tools such as Jenkins and Taurus.)

A stress test is a type of performance test that checks the upper limits of your system by testing it under extreme loads, a simple task with a tool like BlazeMeter. Stress tests examine how the system behaves under intense loads and how it recovers when going back to normal usage. Are the KPIs like throughput and response time the same as before spike in load? Stress tests also look for eventual denials of service, slowdowns, security issues, and data corruption.

Stress testing can be conducted through load testing tools by defining a test case with a very high number of concurrent virtual users.

Just as a stress test is a type of performance test, there are types of load tests as well. If your stress test includes a sudden, high ramp-up in the number of virtual users, it is called a Spike Test. If you stress test for a long period of time to check the system’s sustainability over time with a slow ramp-up, it’s called a Soak Test.

The following example shows how to create a traffic spike using JMeter’s “Ultimate Thread Group” component. We presume the system will be under traffic three minutes into the test. We define more threads to be added within fixed time windows using the “Initial Delay” setting.

Run stress tests against your website or app before major events, like Black Friday, ticket selling for a popular concert with high demand, or elections. We recommend stress testing every once in a while so you know your system’s endurance capabilities. This ensures you’re always prepared for unexpected traffic spikes and gives you more time and resources to fix your bottlenecks.

Another possible positive outcome of stress testing is reducing operating costs. When it comes to cloud providers, they tend to charge for CPU and RAM usage or more powerful instances that cost more. For on-premise deployments, resource-intensive applications consume more electricity and produce more heat. So, identifying bottlenecks not only improves perceived user experience but also saves money and trees.

While load testing and stress testing are two of the most popular performance testing types, they are far from the only performance testing options available.

Let us explore three other types of performance tests: soak tests, spike tests, and scalability tests.

Also known as endurance testing, capacity testing, or longevity testing, soak testing tracks how an application performs under a growing number of users or draining tasks happening over an extended period.

Soak tests are especially known for their extended duration. Once you go through a ramp-up process and reach the target load that you want to test, soak tests maintain this load for a longer timeframe, ranging from a few hours to a few days. The main goal of soak testing is to detect memory leaks.

Spike testing assesses performance by quickly increasing the number of requests up to stress levels and decreasing it again soon after. A spike test will then continue to run with additional ramp-up and ramp-down sequences in either random or constant intervals to ensure continued performance.

Spike tests are great to use for scenarios like auto-scaling, failure recovery, and peak events like Black Friday.

Scalability tests measure how an application can scale certain performance test attributes up or down. When running a scalability test based on a factor like the number of user requests, testers can determine the performance of an application when the user requests scale up or down.

The main metric is whether the scaling out is proportional to the applied load. If not, this is an indication of a performance problem, since the scalability factor should be as close to the load multiplier as possible.

Running your performance tests is an important part of the development process. Here are the different steps you should take for performance testing your application:

Decide on the metrics you want to test. For example, determine your acceptable response time or non-acceptable error rate. These KPIs should be derived based on product requirements and business needs. If you're running these tests continuously, you can use baseline tests to enforce these SLAs.

Detail which scenarios you will be testing. For example, if you have an e-commerce site, you might test the checkout flow.

There are many excellent open source solutions out there, like JMeter, Taurus, and Gatling. You can also use BlazeMeter to get additional capabilities like more geolocations, test data, and advanced reporting.

Build the script in the performance testing tool. Simulate the expected load, the capabilities you are testing, test frequency, ramp-up, and any other part of the scenario. To simplify the process, you can record the scenarios and then edit them for accuracy. If you need test data, add it to the script.

Execute the tests. This is the simple part. Usually you click “run."

Analyze the test results to identify any bottlenecks, performance issues, or other problems. You can use the dashboards provided by the performance testing tool or you can look at solutions like APMs for more information.

Fix the performance issues and retest the application until it meets the performance requirements.

Performance testing and performance engineering are related concepts but they mean different things.

Performance testing evaluates the stability, responsiveness, reliability, speed, and scalability of a system or application under varying workloads. The performance of the system or application is tested and analyzed to ensure that it meets the performance requirements.

Performance engineering, on the other hand, is a proactive approach to software development that identifies and mitigates performance issues early in the development cycle, from the design. By addressing issues earlier, engineering organizations prevent issues and accelerate time-to-market.

Performance testing is one of the steps taken when performing performance engineering.

Performance testing tools are platforms that evaluate and analyze the speed, scalability, robustness and stability of the system under tests. These solutions help ensure that applications and websites can handle the expected level of user traffic and function reliably under different loads. As a result, they are an important component of the software development lifecycle. Many such platforms can integrate with CI/CD tools, so that performance tests are run automatically as part of the integration and deployment pipelines.

One such leading performance testing tool is BlazeMeter. BlazeMeter is a continuous testing platform that enables developers and testers to test the performance of their web and mobile applications under different user loads. It provides a comprehensive range of testing capabilities, including load testing, stress testing, and endurance testing that is open-source compatible. BlazeMeter also supports functional testing and API testing, and provides capabilities like mocking and test data.

Utilize each of the performance testing types detailed in this blog to ensure you are always aware of any issues and can have a plan for dealing with them.

With BlazeMeter, teams can run their performance testing at a massive scale against all your apps, including web and mobile apps, microservices, and APIs. With advanced analytics, teams using BlazeMeter can validate their app performance at every software delivery stage.

BlazeMeter lets you simulate over two million virtual users from 56 locations across the globe (Asia Pacific, Europe, North, and South America) to execute performance tests continuously from development to production.

See for yourself how you can easily build, scale, analyze, and automate performance tests.

START TESTING NOW