Availability Testing | what , how and why?

A while ago , a colleague of mine was asking me about the availability test and my first answer was , do you mean soak/endurance test? , but I was wrong , both tests have something in common but they are totally different in the objective of the test.

What is Availability Testing?

As a general idea, availability is a measure of how often the application is available for use. More specifically, availability is a percentage calculation based on how often the application is actually available to handle service requests when compared to the total, planned, available runtime.

So the idea here is to run tests for longer period of time and collect failures , logs and any other metrics that represent the system availability.

But there is a one more thing to consider , how long it takes a system to switch between active and backup servers , wether it is application or database server , more important is what is the system actual downtime.

How to run availability test?

  1. You have to design a test which can be run for a longer period with a moderate number of users , the number of users is not a key factor here as we are not going to collect performance metrics.
  2. It is time to down one of your working server(s) , in this case will be your active/primary server wether it is a application or database based on the target of your test.You should start receiving errors in your tool and here you can start to count the number of failures and how long it takes your system to move to the secondary / backup node.
  3. Once your system is up again , note all of the errors and time it takes your system to work normally again.
  4. you can repeat the operation to switch again from the backup to primary server or servers.

Why we do Availability Testing?

The target here is to measure and collect data in case of application / database failure , and to make sure that your application setup is properly configured and with a reasonable downtime which will not affect your customers badly in case of unplanned failures or downtime.

Please share your tips, experience, comments, and questions for further enriching this topic of discussion.

Software Testing : Being a specialized or do anything a tester can do

We all have or had been in this debate even inside ourselves , which is better do everything a software tester can do , and here I mean to do manual , automation , performance , security , and anything else may be related to software testing or being a specialized and here I mean that you’re focusing on one or two of the above categories and be expert / specialized.

I am not discussing here exploration vs exploitation , as here we still within the same field which is software testing.

The reason that drove me to start this article , that I was asking a fellow software tester what she look forward in her career , which track she likes the most and I got the following answer :

I can do anything , I am a fast learner

This answer made me think a lot , this is not an answer , yeah we all started , and let me not say all , most of us started from the same point being a manual software tester. Writing , executing test cases and reporting bugs.

Wether you were testing a web , desktop or mobile application, it was the same starting point.

But by time we start to like a specific track or tracks more than the others and I think this is normal , and here I am not forcing the specialization point of view.

So the question here , am I against the do anything a software tester can do , the answer is no.

but you must be specialized in one of them if not on purpose it will be as you do it frequently.

A software tester who is testing mobile application for years and do other things beside it , can do anything as well but he/she is more experienced in testing mobile applications.

To be honest , being specialized has some pros but off course has a cons as well and from a personal experience it is harder to get a job that satisfies you and fulfill your track when you specialized in one track than being a software tester who can do anything.

The job market now forces some standards ,drive people to learn or practice some aspects of software testing like Automation testing , but does this make you automation testing expert , I think the answer is no.

Let me summarize here , I am not here to judge or to say what is right or what is wrong , I am in the same boat have the same struggles.

But what I can say here and also to myself that , it is okay and also good thing to do anything a software tester can do , but I don’t want to say must but try to be a specialized in one of the software testing tracks , this will give you an edge and confidence in your career path.

Please share your tips, experience, comments, and questions for further enriching this topic of discussion.

Don’t commit to a tool and start with POC

Which performance testing tool you’re going to use in your next project , each one of us has a favorite tool or a go to tool , one that always in your mind.

But this is not the case always , we may have some limitations which affect which tool we should use , I will list some of it below :

  1. Corporate decision : some corporates / organizations are not preferring to use an open source tool and also some of them already invested heavily in one tool and they are not going to use something else.
  2. Financial decision : This is the quite opposite from the previous point , we don’t have a budget and we’re going to use an open source tool.
  3. Technical constraints : This is the core of this blog post that we are fine to use anything but which tool is more suitable to our project , an open source tool or a paid tool ?

So why we commit to a tool that may not fit later in the project or wasting time try to make it work.

Let’s do a POC (Proof of concept) , we can try the basic application functionality like login , register , …. to make sure that we don’t have a limitation , and if the current tool is not working properly we can switch to another tool.

Sometimes the limitations is more complex than a basic functionality ,like the application protocol is not supported by the current tool , some of the following protocols are somehow complex that it has a modules available in a specific performance testing tools :

  • Citrix
  • Oracle
  • Siebel
  • SAP

In summary , don’t commit to a tool in the beginning of a performance test project except you’re 100% sure that it is going to work , take your time and make your own POC to make sure that you don’t and you will not have a limitation or unsolvable complexity during your project.

Please share your tips, experience, comments, and questions for further enriching this topic of discussion.

Unusual performance tests for unusual situations

Not all performance test types are famously equal , some types are used less often than other.

That doesn’t mean of course that they are not important , they are and in some cases so important but unusual tests needs unusual situations.

I am going to focus on the following performance test types , we will discuss them in details and I will try to give an example.

  • Endurance Testing
  • Spike Testing
  • Volume Testing

Endurance Testing : The less unusual type , in this test type we are executing performance runs for longer period of times (8 , 12 or 24 hrs) to test system availability and also to make sure that we don’t have some issues like “memory leaks”

The execution time should be based on the system operating time , a hosted web application which is available 24/7 , it is not operating 24/7.

Business / service operating time , is the duration the web application is actually functioning not only available , this vary according to the business domain

PerfMatrix: Do you really know all type of Performance Tests (Non-Functional Tests)?

Example : Online Delivery web site which accepting orders from 10 AM to 10 PM is not functioning 24 hrs.

Spike Testing : Some people are confusing this test with the load testing , but they are different in design and the impact as well.

As shown on the following graph , the application / service is facing an unusual users hit for a specific period of time and after that we are returning to the normal application load.

Stating the obvious - Coding the Architecture

Example : An e-commerce application which promote 1 hour exceptional discount / offer should have a user spike for 1 hour and after that it will return to the normal user load or slightly higher.

Volume Testing : The idea here is to perform your load testing but with different database sizes (volumes) to be sure that the system performance and behavior is not affected by the expected increases of system size.

Most of the time this type of test is needed when your system is dealing or storing a big amount of data , and we suspect that we may have a big increase in system size in a short time period.

Example : Governmental application which allow users to submit data with large volume (Ex.scanned documents , official documents) are likely hood to have a large set of data base size in a small period of time and achieve massive data size in the near future.

Please share your tips, experience, comments, and questions for further enriching this topic of discussion.

The hidden cost of slowness

There is always a debate about how important is the speed to the software industry. They say and I quote “if the customer/user is not complaining , there is no issue and we are fine”.

I will try here to discuss this in a different way.

what is the hidden cost you pay when your site/service is slow and how you are losing although the customer is not complaining.

Speed-Revenue dilemma

“The speed of the site negatively impacts a user’s session depth, no matter how small the delay…The data suggests, both in terms of user experience and financial impact, that there are clear and highly valued benefits in making the site even faster. 

Users get even MORE impatient when it comes to website speed. Want proof? Have a look at the Financial Times Case study:

They add a 5-second delay to each page load time. Notable facts they found:

  • The first-second delay resulted in a 4.9% drop in the number of articles a visitor read
  • The three-second delay resulted in a 7.9% drop
  • Visitors read less when delays occurred
  • Effect on Sale: 79% of customers who report dissatisfaction with website performance are less likely to buy from that same site again.
  • Speed Affects Revenue: If your site makes $100,000/month, a one second improvement in page speed brings $7,000 month

Speed-Satisfaction dilemma

Also customer loyalty can be affected by the site / service speed 👇

customer loyalty statistics

To sum up , there will be always a cost for slowness , as described here in the article. Revenue , satisfaction and loyalty are the price you pay or the correct word is you lose when you neglect the speed of your site/service , and yes your customer may not complain but also this is not a proof that he/she is satisfied with your service.

Sources :

How Does Page Load Time Affect Your Site Revenue?

https://www.websitebuilderexpert.com/building-websites/website-load-time-statistics/

Please share your tips, experience, comments, and questions for further enriching this topic of discussion.

How to Set your performance testing acceptance criteria

This is always a question for people who are doing performance testing (as a general term) for the first time and also those who don’t have a specific performance requirements.

  • What numbers to compare to?
  • what are the current response time means?
  • is it good or bad?
  • How to set an acceptance criteria?

All the above questions we used to hear in the beginning of any performance testing project or even a discussion about a future need for a testing.

I will list here some ideas that will help you determine and simplify the process of setting a test acceptance criteria.

  • Check websites/services working in the same domain : gather information about how their services response times are , and compare it to your current response times and if you’re way higher than them you have to plan enhancements for your current operating service(s).
    *Start with your local competitors as both of you are operating in the same market.
  • Some organizations / sites publish a yearly report about the web performance in general and categorized it by business domain , this will help you have an overview about the response times trend and have at least numbers that you don’t want to exceed anyway.
    *The full article link can be found in the end of this article.
IndustryUnited StatesUnited KingdomGermanyJapan
Automotive9.5 sec12.3  sec11.0 sec10.3 sec
Business & Industrial Markets8.7 sec8.3 sec8.2 sec8.1 sec
Classifieds & Local7.9 sec8.3 sec7.0 sec8.3 sec
Finance8.3 sec8.0 sec8.6 sec7.6 sec
Media & Entertainment9 sec8.8 sec7.6 sec8.4 sec
Retail9.8 sec10.3 sec10.3 sec8.3 sec
Technology11.3 sec10.6 sec8.8 sec10sec
Travel10.1 sec10.9 sec7.1 sec8.2 sec
While the average of the values in the table is 8.66 sec, the recommendation for 2018 is to be under 3 seconds.
  • If you are doing a revamp or replacement to an old system/service try to achieve at the least the same old system performance (in case the performance wasn’t the reason for the revamp 🙂 ) and then you can plan for 20 – 30% better performance than the old system , off-course you can plan for a higher performance achievement but it should be specific to not wasting a lot of time chasing unclear goal.

To summarize , it is ok if you don’t have a specific performance requirements , you can set your requirements based on how you’re operating comparing to the others and also having initial goal is a good step to start plan your performance enhancements and for sure those goals will be more ambitious by time.

Sources :

Average Page Load Times for 2018 – How does yours compare?
Average Page Load Times for 2020 – Are you faster?

Please share your tips, experience, comments, and questions for further enriching this topic of discussion.

What a performance test report saying about your system.

When evaluating a performance test report most of the times we are looking for the response time and specifically the Average response time.

But if you take a deeper look , the performance test report elaborates more information.

In this article I will use one of JMeter basic reports “Summary Report” as example to explain what I mean.

The focus in this article will be on the following terms/values

  • Standard Deviation
  • Min Response Time
  • Max Response Time

Standard Deviation :

The Standard Deviation is a measure of how response time is spread out around the Mean. Simply say, the smaller the Standard Deviation, the more consistent the response time.

Transaction NameRT
(I1)		RT
(I2)		RT
(I3)		RT
(I4)		RT
(I5)		AvgSD90th %ile
Login46348526
Search32151455.74
Logout5564550.75
“Logout” transaction having lowest Standard Deviation (0.7) it shows response times are more consistent than other two.

Standard Deviation in your test tells whether the response time of a particular transaction is consistent throughout the test or not? The smaller the Standard Deviation, the more consistent transaction response time and you will be more confident about particular page/request.

Min.Response Time:

The shortest time taken by a sample for specific label. If we look at Min value for Label 1 then, out of 20 samples shortest response time one of the sample had was 584 milliseconds.

Max.Response Time:

The longest time taken by a sample for specific label. If we look at Max value for Label 1 then, out of 20 samples longest response time one of the sample had was 2867 milliseconds.

Sources :

https://www.perfmatrix.com/standard-deviation-in-performance-testing/http://www.testingjournals.com/understand-summary-report-jmeter/

Please share your tips, experience, comments, and questions for further enriching this topic of discussion.

How to add security checks to your manual / automation test suite

Let me tell you that you can have a basic / moderate security checks in your manual test suite by extending your test cases in two different areas :

  • Input validation
  • Authentication

Most of test suites if not all of them are already testing the sections mentioned above but mostly just a basic checks like if the field accepting numbers we try characters and alpha numerics. What I am suggesting here is to test any input field against major web app vulnerabilities like XSS & SQL Injection

The same case for authentication instead of trying different combinations for right / wrong usernames and passwords .You can extend your test against major web app vulnerability like SQL Injection

XSS : XSS attacks enable attackers to inject client-side scripts into web pages viewed by other users. A cross-site scripting vulnerability may be used by attackers to bypass access controls such as the same-origin policy.

We’re going to use demo.testfire.net as a safe place to practice on.

Example :

The field we’re going to test here is the Search field , I am assuming that we already executed test cases with character , numbers , alphanumeric and also very long characters / numbers input to validate the input boundary.

Lets try an XSS payload as input.The input will be “<ScRipT>alert(“XSS”);</ScRipT>”

According to the above screen shot it seems that the web app under test is vulnerable to XSS attacks.

XSS payloads example :

  • </script><script>alert(1)</script>
  • <IMG SRC=jAVasCrIPt:alert(‘XSS’)>
  • <iframe %00 src=”&Tab;javascript:prompt(1)&Tab;”%00>
  • <form><isindex formaction=”javascript&colon;confirm(1)”

You can find XSS payload list in the following URL :
https://github.com/pgaijin66/XSS-Payloads/blob/master/payload.txt

SQL Injection : SQL injection is a code injection technique, used to attack data-driven applications, in which malicious SQL statements are inserted into an entry field for execution.

The element we’re going to test here is the Login form , I am assuming that we already executed test cases with valid and invalid username , password also very long characters / numbers input to validate the input boundary.

Lets try SQL Injection payload in the “username” field and any characters in the password field.

username value will be = ‘ ‘
password value will be = test

Lets try more advanced input as below

username value will be = ‘ UNION SELECT sum(columnname ) from tablename —
password value will be = test

According to the above screen shot it seems that the web app under test is vulnerable to SQL Injection attacks.

You can find SQL Injection payload list in the following URL :
https://github.com/payloadbox/sql-injection-payload-list

Conclusion :

By adding more test cases to your existing test suite you can help discover security vulnerabilities in the system under test without the need to learn a new tool.Of course it will increase the testing execution time but the benefit here is to catch those issues as soon as the testing started.

Of course this not a replacement for security testing a web app but the idea here is cover at least some basic security checks in the normal testing process.

Please share your tips, experience, comments, and questions for further enriching this topic of discussion.

Performance Test Script Validation – Why & How?

The main goal for any performance test script simply is to work , but is this enough? I mean is it enough that your script has no errors ?

No errors do not mean that your script is working flawlessly , you may get a 200 response code but the script functionality is not working and in this case all your results are not correct.

The process of checking if you receive the correct response is called Validation.

In this article I will demonstrate the validation process using JMeter as one of the performance testing tools that is widely being used.

How to use validations in JMeter

In JMeter context menu we have a whole section called Assertions as in the image below

As you can see above there are lots of assertions available to use but we will focus on one particular called “Response Assertion”

Response Assertion

Before we start to dig more deeper let’s have an example to show when the response code doesn’t mean that the script is working correctly.

Example

Our script should do the following :

1- Open “demo.testfire.net”

2- Open the Login page

3- Do the Login with the credentials (admin/admin)

In the following 2 screen shots we will Show that having a success repose code doesn’t mean that the scenario went well.

The do login has a 200 response code

According to the above screen shot the do Login should be done successfully and user should be already logged-in

But Actually the login didn’t happen , so this step of our scenario is not a successful one.

The reason that this step is failing is because I disabled the “HTTP cookie Manager” which is in most cases required in the login scenarios.

Let’s try to use the Response Assertion we mentioned earlier in this article and try to validate our scenario but before we do this let’s enable the Cookie manager to choose which text we can use in our validation step.

Now we have a successful login , so I think we can use the “Sign off” text as our asserion , because the sign off link will not be displayed if the user is not logged-in.

I added a response assertion as a child to the do login request , use the Text response and also put “Sign off” as the text to search for in the response.

I will do a trial with the Cookie manager on , then I will re-run the test with the cookie manager off to check that our validation (assertion) is working.

When I execute the test with the Cookie manager disabled , now we have a failed request although we have a 200 response code as shown on the following image.

Text Assertion is not the only assertion we can use but I think it is the mostly used one , and it will help you validate from the script side that you test is doing what should be done , help you have accurate results and have look about how your script and system under test is behaving.

*The JMX used in this article is uploaded here , feel free to use.

Download

Factors that affect your performance test results

The performance test process is not a complex process but it has lots of things to keep an eye on , in this article I will focus on factors that personally I think it will affect your test results. I will try to demonstrate the effect in most of them and I will put some references if exist.

Think Time :

By definition think time is the time between the completion of one request and the start of the next request.

So we can say that it is a kind of delay. So obviously when you have long think time value , it means long delays and less pressure on the system under test and also means if we have no think time this means more pressure on the system under test.

Short / No think time = More pressure

Too Long / Long think time = Less / no pressure

The question will be how to determine the suitable think time for your system? , you can determine how long time user spend on your system pages from analytical tool like “Google Analytics”

Google Analytics – Audience Overview

From the above screenshot we can roughly determine how long time the user spend on each page during his session (the the value is the average of all users in a specific range of dates)

What you can do , you can randomize a value between lower and upper think time values to have different think time value per request / per user.

The more close to real life the values you use , the more realistic results you will get.

Generating Users :

In most of performance testing tools if not all of them you have at least two options for user generation :

  • Constant Load : which means that all users will start to hit the server the same moment you will start the test.
  • step-up load / ramp-up : which means that we will introduce new user/thread every specified amount of time.
Micro Focus Load Runner – constant user generation
How to use Controller in LoadRunner
Micro Focus Load Runner – Ramp-up user generation

In most cases the ramp-up user generation will be the better approach except you want to test a specific scenario , because hitting the system under test with all users at the same moment is not an ideal scenario and sometimes It is not realistic. It will affect the test response time badly if the system is not design to sustain this kind of users hit.

There is no ideal number for the step duration so it can be tweaked during the test run or try to get this information from the analytics the same we described with the think time above.

Test Data :

Data used during the performance test run is important , the more close to real life the data the more accurate results you will get.

Also avoiding using the same data for all generated users like (user credentials , search keywords , etc) will eliminate the factor that caching may affect the test results.

Use unique data for each generated user and make sure that you have enough data to use during your test run.

Latency

Latency is the time from simply sending out the request until the first byte of response is accepted, it is also called as Time to First Byte.

You will always have a latency in your test if you are not testing in an ideal test environment.

But you can reduce the latency value by placing your remote machines in the closest region to your hosted application.

You have to initiate your test from the same region or close to the region your real users will access the application from.This will lead to more accurate test results.

Load generators

It is normal that most of test executions are initiated from a one machine / server if the number of users generated is not a large number.

But it is recommended to distribute the load generation among different machines / servers even the number of users is not that large.

This will help to balance the load on the system under test and avoid some security restriction for the hitting frequency from the same host.

Have you faced problems before related to mentioned factors? How you managed it? Please share your tips, experience, comments, and questions for further enriching this topic of discussion.