Why you should execute your performance tests gradually?

Let’s assume that we have a planned performance testing run with 10,000 users.

In this article I will discuss with you why we should start our execution gradually , from my point of view you should not start the run with the mentioned number of users , 10,000 users couldn’t be the start , for the following reasons.

  1. Frustration
    Most of the times when this is the first run ever , it will fail and may be failed dramatically with very high response time and tons of errors.
  2. Observation
    When we start the run gradually let’s say we started with the 10% of the load which is 1000 users , we can observe while increasing the load the response time changes from run to another also we can observe the backend metrics (CPU , Memory) changes between the runs.
    This will absolutely give you a better understanding how the system under test behaving with the user increase.

    3. Cost Benefit analysis
    Referring to the previous point , it will be clear which part of the system will be worth investing in , here I mean which is more cost saving (enhancing the software or buy/rent more hardware).

In the following I will suggest a simple execution plan , you can tweak it to match your case.

#User %User Number
1101,000
2202,000
3505,000
410010,000

As shown on the previous table i suggest 4 steps to reach 100% of the required load , you can absolutely skip steps or decreasing the number of steps if you observe in your results that the difference in response time is minimal when raising the load.

That’s for today 🙂 , hope you find it useful.

Please share your tips, experience, comments, and questions for further enriching this topic of discussion.

OWASP ZAP – add-ons that will enrich your discovery

In this article i will refer to number of add-ons that i think may help enrich the vulnerability discovery and also give a clear understanding about the system under test.

First of all what is OWASP ZAP?

Zap is a famous open source automated pen testing tool , created and maintained by the OWASP community. Pen testing is mostly on the application level , but you can extend the testing to cover things like open ports as an example.

I will list some add-ons that enrich the testing capabilities and you may find it useful.

  1. Wappalyzer – Technology Detection :

    This add-on can gather information about the system used technology like the application server , used frameworks , hosting service and also the OS used.

As shown above , this add-on can give you an overview about the used technologies on the system under test.

2. Port Scanner

This add-on can gather information about the open ports on the destination application/ web server , this will help you close all unnecessary ports to eliminate attacks.

3. FuzzDB Files

In Zap you have the ability to fuzz a parameter and with Fuzz here i mean to replace a parameter value with lots of patterns to test things like SQL Injection , XSS , OS Injection and more patterns that help you discover more vulnerabilities.

Fuzz DB files provide you with lots of patterns categorized by the test type as the following screen shots.

In the following example , i used the fuzzDB XSS patterns to exploit a vulnerable search field , and here are the results.

That’s for today 🙂 , hope you find it useful.

Please share your tips, experience, comments, and questions for further enriching this topic of discussion.

API Security Testing With Postman and OWASP Zap

Most of the content around API testing is about functional testing or recently about API automation testing , so what about Security Testing?

We’re going to use Postman and consume our existing collections.

The idea here is to send the Postman requests to OWASP Zap to be able to start automated pen-testing.

Why?
Sometimes we don’t have the proper API definition file that can be imported to OWASP Zap , so this is an easy workaround.

Step 1 :

Open OWASP Zap , go to application settings and look for “Local Proxy” as shown on the following screen shot.

On Windows : Go to “Tools” -> “Options”

On Mac OS : Go to “Zap” -> “Preferences”

In our case the local proxy is on port 8081 remember this number because we will use it very soon.

*Port number may be different on your machine , use the number displayed in your setting on the next step.

Step 2 :

Open “Postman” , go to application settings and press “Proxy” tab as shown on the following screen shot.

On Windows : Go to “File” -> “Settings”

On Mac OS : Go to “Postman” -> “Preferences”

Select “Add custom proxy configuration” and fill the following values :

  • Proxy Server : Localhost
  • Port : 8081 (the port acquired from the OWASP Zap settings in step 1)

Step 3 :

On Postman start to send API requests from the desired API collection as shown on the following Postman example.

All api calls you just did from Postman should be added to OWASP Sites list as the following screen shot.

*Don’t forget to return the Postman proxy settings to the previous / default settings after you finish.

Step 4 :

It is time to start the scan on OWASP ZAP.

Right Click on the main directory and select “Attack” then select “Active Scan”

The scan will be started and you should notice some findings under “Alerts” section as the following screen shot.

Indeed you can do more than active scan on OWASP Zap , and this may be another post I will do in the future to dig more deeper into ZAP 🙂

The API used in this demo is called VAmPI , it is a vulnerable api and you can find it on the following Github link :

https://github.com/erev0s/VAmPI

I hope you find it useful and I really enjoyed the time I was trying it.

Please share your tips, experience, comments, and questions for further enriching this topic of discussion.

Performance Testing background noise – What , why and how?

The main goal for any Performance Test Engineer is to make the perfect simulation , because this will lead to accurate test results and that’s what we need from this kind of test.

What is background noise?

The idea is to record and play some users behaviors which is not meant to be measured or not that important to make some noise , and this simulates the user behaviors in any system in real life.

You may too concerned about the “Login” , “Register” or any other user actions , but what is happening in real life is different.

Why?

Because while you as a user try to login or register there are some other users who opening the “contact us” page , “site map” , “subscribe to news letter” and may be “about us” page.

I know that those are the least functionality you want to measure but you don’t have to , you just want to play this in the background or in a more accurate way in parallel with your designed tests.

How?

What you need is to record some user behaviors as described above and run them in parallel with your current performance test.

You don’t have to assign a large number of users to these tests , you can assign a percentage 5% or 10% as example of your original load just to let this noise affect the system.

As you can see in the above JMeter script , one more Thread group is added with different transactions , this ThreadGroup will be executed in parallel with the original script to make some noise , or you can run it as a separate test from another machine in parallel with the original script.

As mentioned before , it is not important to measure those extra transactions response time , the job here is to send more request.

In the end , we always searching for a way to make our test results as representative as possible , because this will increase the reliability of the system and make our customers happy.

Please share your tips, experience, comments, and questions for further enriching this topic of discussion.

Web performance testing – an overview (presentation)

During 2021 I’ve presented this presentation different times to a different crowds in a different regions.

In this presentation I was trying to knock different doors that are related to performance testing without digging deeper , giving the audience the freedom to search and decide which part is more important to them to look into.

And I think it is the time now to publish it for two reasons actually:
1. To force myself not to use it or present it again 🙂
2. It is an overview in a simple form that can be easily consumed.

I hope you find it useful and I really enjoyed the time I was presenting it this year.

Please share your tips, experience, comments, and questions for further enriching this topic of discussion.

Forecasting Number of users in performance testing

How many users should be applied in our performance tests ?

in a previous article we discussed some ways that may simplify the process of setting system SLA , so if we can say that we have now our SLA or simply what is the response time we are hoping to operate under , what about the number of users.

The simple answer is it should be a given requirement , but sometime we don’t have or actually we don’t know.

The following are some ideas that may help :

– If the site is up and running , you can get the users numbers and distributions (users percentage per function or page) through analytical tools like Google Analytics.

– If you are selling a product to a company or organization you can forecast based on the number of employees , it can be a percentage of them or all of them as the highest load possible.

– Adding to the previous point , if the system is license based you can forecast by the number of sold licenses and also based on the max number of users per license.

– If the site / service is completely new and you’re in a lunching process , you can get the numbers based on the market research and the sales forecasting for the first 3 – 6 months and you can adapt your tests and infrastructure when needed during the 1st year of the lunch.

No.of users is an important factor within the performance testing activities and that’s why it is important to set them carefully and as close as possible to real world scenario to have accurate results and to have confidence in your running system.

Please share your tips, experience, comments, and questions for further enriching this topic of discussion.

Running performance tests from the cloud , why?

Does it necessary , why , for what cost and limitations ,all these questions are valid and make totally sense.

Let me first define what is meant by running performance tests from the cloud , in the simple form it is running performance test scripts from a rented virtual machine(s) from a cloud service provider like (AWS or Azure) or testing platform which offers a cloud service run like (Blazemeter , Microfocus,..)

Does it necessary ? , the answer is yes in some situations like the following :

  • Generate huge amount of users which requires a highly equipped machines which cost fortune if you build it locally.
  • Simulate different regions and countries which is required in your test run , this can be provided by multiple cloud service platforms which has different server locations around the world.
  • Reliability and consistency for the used machines , most cloud platforms guarantee more than 90% availability for their services.

Why ? , as I mentioned above in some cases massive resources are required and this can be easily provided by cloud platforms than build your own load generators on-premise.

For what cost ? , you are billed based on hourly rate according to the selected machine and its hardware , sometimes is cheaper to rent than build your own lab. But in some cases based on the usage and frequency , it is cheaper to build your own lab.

Limitations ? , if your application or software is not available online or it is only accessible through internal organization / company network , the cloud testing will not be the best fit in this case.

Also if you have some resources that you can use instead of renting.
example : use the company desktops to generate the users instead of create a cloud load generators.

The final reason is the cost , if you don’t have the budget to rent cloud machines , in this case use your company assets is the best solution to complete the task.

To sum up , running performance tests from the cloud is not a must or a trend , it is a go to solution if need it. It saves time , configuration hassle and sometimes money but it is not the magic solution for every performance run.

Please share your tips, experience, comments, and questions for further enriching this topic of discussion.

Performance Tuning – A Team Effort

Days ago, I was discussing the idea of performance tuning with a number of fellow testers and the discussion direction was heading mainly towards the responsibilities than the methodology.

The main question was who is responsible for the performance tuning?

I think we have to define what is performance tuning in the beginning, Performance tuning is the improvement of system performance , either to solve a performance problem or to achieve a goal.

Back to the main point about who is responsible for the performance tuning, it is a team responsibility.

In a tuning process, you will need different expertise to achieve the tuning goal.

You will need the following people:  

Architects to review the system design and suggest which part could be replaced or enhanced to improve the system performance. 

Performance testers, to design and execute tests to find and point out where is the performance bottlenecks. 

Developers to assess the code and suggest which methods or block of codes can be improved to help solve the performance bottlenecks.

Management to set priorities and clearly elaborate which part of the system is more vital or important to the customer and business to start with.

With the collaboration of all of the mentioned people, you can proceed with a productive performance tuning, achieve the best results and avoid reworks in my opinion.

Please share your tips, experience, comments, and questions for further enriching this topic of discussion.

Availability Testing | what , how and why?

A while ago , a colleague of mine was asking me about the availability test and my first answer was , do you mean soak/endurance test? , but I was wrong , both tests have something in common but they are totally different in the objective of the test.

What is Availability Testing?

As a general idea, availability is a measure of how often the application is available for use. More specifically, availability is a percentage calculation based on how often the application is actually available to handle service requests when compared to the total, planned, available runtime.

So the idea here is to run tests for longer period of time and collect failures , logs and any other metrics that represent the system availability.

But there is a one more thing to consider , how long it takes a system to switch between active and backup servers , wether it is application or database server , more important is what is the system actual downtime.

How to run availability test?

  1. You have to design a test which can be run for a longer period with a moderate number of users , the number of users is not a key factor here as we are not going to collect performance metrics.
  2. It is time to down one of your working server(s) , in this case will be your active/primary server wether it is a application or database based on the target of your test.You should start receiving errors in your tool and here you can start to count the number of failures and how long it takes your system to move to the secondary / backup node.
  3. Once your system is up again , note all of the errors and time it takes your system to work normally again.
  4. you can repeat the operation to switch again from the backup to primary server or servers.

Why we do Availability Testing?

The target here is to measure and collect data in case of application / database failure , and to make sure that your application setup is properly configured and with a reasonable downtime which will not affect your customers badly in case of unplanned failures or downtime.

Please share your tips, experience, comments, and questions for further enriching this topic of discussion.

Software Testing : Being a specialized or do anything a tester can do

We all have or had been in this debate even inside ourselves , which is better do everything a software tester can do , and here I mean to do manual , automation , performance , security , and anything else may be related to software testing or being a specialized and here I mean that you’re focusing on one or two of the above categories and be expert / specialized.

I am not discussing here exploration vs exploitation , as here we still within the same field which is software testing.

The reason that drove me to start this article , that I was asking a fellow software tester what she look forward in her career , which track she likes the most and I got the following answer :

I can do anything , I am a fast learner

This answer made me think a lot , this is not an answer , yeah we all started , and let me not say all , most of us started from the same point being a manual software tester. Writing , executing test cases and reporting bugs.

Wether you were testing a web , desktop or mobile application, it was the same starting point.

But by time we start to like a specific track or tracks more than the others and I think this is normal , and here I am not forcing the specialization point of view.

So the question here , am I against the do anything a software tester can do , the answer is no.

but you must be specialized in one of them if not on purpose it will be as you do it frequently.

A software tester who is testing mobile application for years and do other things beside it , can do anything as well but he/she is more experienced in testing mobile applications.

To be honest , being specialized has some pros but off course has a cons as well and from a personal experience it is harder to get a job that satisfies you and fulfill your track when you specialized in one track than being a software tester who can do anything.

The job market now forces some standards ,drive people to learn or practice some aspects of software testing like Automation testing , but does this make you automation testing expert , I think the answer is no.

Let me summarize here , I am not here to judge or to say what is right or what is wrong , I am in the same boat have the same struggles.

But what I can say here and also to myself that , it is okay and also good thing to do anything a software tester can do , but I don’t want to say must but try to be a specialized in one of the software testing tracks , this will give you an edge and confidence in your career path.

Please share your tips, experience, comments, and questions for further enriching this topic of discussion.