The Test Therapist

Does it necessary , why , for what cost and limitations ,all these questions are valid and make totally sense.

Let me first define what is meant by running performance tests from the cloud , in the simple form it is running performance test scripts from a rented virtual machine(s) from a cloud service provider like (AWS or Azure) or testing platform which offers a cloud service run like (Blazemeter , Microfocus,..)

Does it necessary ? , the answer is yes in some situations like the following :

• Generate huge amount of users which requires a highly equipped machines which cost fortune if you build it locally.
• Simulate different regions and countries which is required in your test run , this can be provided by multiple cloud service platforms which has different server locations around the world.
• Reliability and consistency for the used machines , most cloud platforms guarantee more than 90% availability for their services.

Why ? , as I mentioned above in some cases massive resources are required and this can be easily provided by cloud platforms than build your own load generators on-premise.

For what cost ? , you are billed based on hourly rate according to the selected machine and its hardware , sometimes is cheaper to rent than build your own lab. But in some cases based on the usage and frequency , it is cheaper to build your own lab.

Limitations ? , if your application or software is not available online or it is only accessible through internal organization / company network , the cloud testing will not be the best fit in this case.

Also if you have some resources that you can use instead of renting.
example : use the company desktops to generate the users instead of create a cloud load generators.

The final reason is the cost , if you don’t have the budget to rent cloud machines , in this case use your company assets is the best solution to complete the task.

To sum up , running performance tests from the cloud is not a must or a trend , it is a go to solution if need it. It saves time , configuration hassle and sometimes money but it is not the magic solution for every performance run.

Please share your tips, experience, comments, and questions for further enriching this topic of discussion.

Days ago, I was discussing the idea of performance tuning with a number of fellow testers and the discussion direction was heading mainly towards the responsibilities than the methodology.

The main question was who is responsible for the performance tuning?

I think we have to define what is performance tuning in the beginning, Performance tuning is the improvement of system performance , either to solve a performance problem or to achieve a goal.

Back to the main point about who is responsible for the performance tuning, it is a team responsibility.

In a tuning process, you will need different expertise to achieve the tuning goal.

You will need the following people:  

Architects to review the system design and suggest which part could be replaced or enhanced to improve the system performance. 

Performance testers, to design and execute tests to find and point out where is the performance bottlenecks. 

Developers to assess the code and suggest which methods or block of codes can be improved to help solve the performance bottlenecks.

Management to set priorities and clearly elaborate which part of the system is more vital or important to the customer and business to start with.

With the collaboration of all of the mentioned people, you can proceed with a productive performance tuning, achieve the best results and avoid reworks in my opinion.

Please share your tips, experience, comments, and questions for further enriching this topic of discussion.

Which performance testing tool you’re going to use in your next project , each one of us has a favorite tool or a go to tool , one that always in your mind.

But this is not the case always , we may have some limitations which affect which tool we should use , I will list some of it below :

1. Corporate decision : some corporates / organizations are not preferring to use an open source tool and also some of them already invested heavily in one tool and they are not going to use something else.
2. Financial decision : This is the quite opposite from the previous point , we don’t have a budget and we’re going to use an open source tool.
3. Technical constraints : This is the core of this blog post that we are fine to use anything but which tool is more suitable to our project , an open source tool or a paid tool ?

So why we commit to a tool that may not fit later in the project or wasting time try to make it work.

Let’s do a POC (Proof of concept) , we can try the basic application functionality like login , register , …. to make sure that we don’t have a limitation , and if the current tool is not working properly we can switch to another tool.

Sometimes the limitations is more complex than a basic functionality ,like the application protocol is not supported by the current tool , some of the following protocols are somehow complex that it has a modules available in a specific performance testing tools :

• Citrix
• Oracle
• Siebel
• SAP

In summary , don’t commit to a tool in the beginning of a performance test project except you’re 100% sure that it is going to work , take your time and make your own POC to make sure that you don’t and you will not have a limitation or unsolvable complexity during your project.

Please share your tips, experience, comments, and questions for further enriching this topic of discussion.

Not all performance test types are famously equal , some types are used less often than other.

That doesn’t mean of course that they are not important , they are and in some cases so important but unusual tests needs unusual situations.

I am going to focus on the following performance test types , we will discuss them in details and I will try to give an example.

• Endurance Testing
• Spike Testing
• Volume Testing

Endurance Testing : The less unusual type , in this test type we are executing performance runs for longer period of times (8 , 12 or 24 hrs) to test system availability and also to make sure that we don’t have some issues like “memory leaks”

The execution time should be based on the system operating time , a hosted web application which is available 24/7 , it is not operating 24/7.

Business / service operating time , is the duration the web application is actually functioning not only available , this vary according to the business domain

Example : Online Delivery web site which accepting orders from 10 AM to 10 PM is not functioning 24 hrs.

Spike Testing : Some people are confusing this test with the load testing , but they are different in design and the impact as well.

As shown on the following graph , the application / service is facing an unusual users hit for a specific period of time and after that we are returning to the normal application load.

Example : An e-commerce application which promote 1 hour exceptional discount / offer should have a user spike for 1 hour and after that it will return to the normal user load or slightly higher.

Volume Testing : The idea here is to perform your load testing but with different database sizes (volumes) to be sure that the system performance and behavior is not affected by the expected increases of system size.

Most of the time this type of test is needed when your system is dealing or storing a big amount of data , and we suspect that we may have a big increase in system size in a short time period.

Example : Governmental application which allow users to submit data with large volume (Ex.scanned documents , official documents) are likely hood to have a large set of data base size in a small period of time and achieve massive data size in the near future.

Please share your tips, experience, comments, and questions for further enriching this topic of discussion.

There is always a debate about how important is the speed to the software industry. They say and I quote “if the customer/user is not complaining , there is no issue and we are fine”.

I will try here to discuss this in a different way.

what is the hidden cost you pay when your site/service is slow and how you are losing although the customer is not complaining.

Speed-Revenue dilemma

“The speed of the site negatively impacts a user’s session depth, no matter how small the delay…The data suggests, both in terms of user experience and financial impact, that there are clear and highly valued benefits in making the site even faster. 

Users get even MORE impatient when it comes to website speed. Want proof? Have a look at the Financial Times Case study:

They add a 5-second delay to each page load time. Notable facts they found:

• The first-second delay resulted in a 4.9% drop in the number of articles a visitor read
• The three-second delay resulted in a 7.9% drop
• Visitors read less when delays occurred

• Effect on Sale: 79% of customers who report dissatisfaction with website performance are less likely to buy from that same site again.
• Speed Affects Revenue: If your site makes $100,000/month, a one second improvement in page speed brings $7,000 month

Speed-Satisfaction dilemma

Also customer loyalty can be affected by the site / service speed 👇

To sum up , there will be always a cost for slowness , as described here in the article. Revenue , satisfaction and loyalty are the price you pay or the correct word is you lose when you neglect the speed of your site/service , and yes your customer may not complain but also this is not a proof that he/she is satisfied with your service.

Sources :

How Does Page Load Time Affect Your Site Revenue?

https://www.websitebuilderexpert.com/building-websites/website-load-time-statistics/

Please share your tips, experience, comments, and questions for further enriching this topic of discussion.