What a performance test report saying about your system.

When evaluating a performance test report most of the times we are looking for the response time and specifically the Average response time.

But if you take a deeper look , the performance test report elaborates more information.

In this article I will use one of JMeter basic reports “Summary Report” as example to explain what I mean.

The focus in this article will be on the following terms/values

  • Standard Deviation
  • Min Response Time
  • Max Response Time

Standard Deviation :

The Standard Deviation is a measure of how response time is spread out around the Mean. Simply say, the smaller the Standard Deviation, the more consistent the response time.

Transaction NameRT
(I1)
RT
(I2)
RT
(I3)
RT
(I4)
RT
(I5)
AvgSD90th %ile
Login46348526
Search32151455.74
Logout5564550.75
“Logout” transaction having lowest Standard Deviation (0.7) it shows response times are more consistent than other two.

Standard Deviation in your test tells whether the response time of a particular transaction is consistent throughout the test or not? The smaller the Standard Deviation, the more consistent transaction response time and you will be more confident about particular page/request.

Min.Response Time:

The shortest time taken by a sample for specific label. If we look at Min value for Label 1 then, out of 20 samples shortest response time one of the sample had was 584 milliseconds.

Max.Response Time:

The longest time taken by a sample for specific label. If we look at Max value for Label 1 then, out of 20 samples longest response time one of the sample had was 2867 milliseconds.

Sources :

https://www.perfmatrix.com/standard-deviation-in-performance-testing/http://www.testingjournals.com/understand-summary-report-jmeter/

Please share your tips, experience, comments, and questions for further enriching this topic of discussion.

Performance Test Script Validation – Why & How?

The main goal for any performance test script simply is to work , but is this enough? I mean is it enough that your script has no errors ?

No errors do not mean that your script is working flawlessly , you may get a 200 response code but the script functionality is not working and in this case all your results are not correct.

The process of checking if you receive the correct response is called Validation.

In this article I will demonstrate the validation process using JMeter as one of the performance testing tools that is widely being used.

How to use validations in JMeter

In JMeter context menu we have a whole section called Assertions as in the image below

As you can see above there are lots of assertions available to use but we will focus on one particular called “Response Assertion”

Response Assertion

Before we start to dig more deeper let’s have an example to show when the response code doesn’t mean that the script is working correctly.

Example

Our script should do the following :

1- Open “demo.testfire.net”

2- Open the Login page

3- Do the Login with the credentials (admin/admin)

In the following 2 screen shots we will Show that having a success repose code doesn’t mean that the scenario went well.

The do login has a 200 response code

According to the above screen shot the do Login should be done successfully and user should be already logged-in

But Actually the login didn’t happen , so this step of our scenario is not a successful one.

The reason that this step is failing is because I disabled the “HTTP cookie Manager” which is in most cases required in the login scenarios.

Let’s try to use the Response Assertion we mentioned earlier in this article and try to validate our scenario but before we do this let’s enable the Cookie manager to choose which text we can use in our validation step.

Now we have a successful login , so I think we can use the “Sign off” text as our asserion , because the sign off link will not be displayed if the user is not logged-in.

I added a response assertion as a child to the do login request , use the Text response and also put “Sign off” as the text to search for in the response.

I will do a trial with the Cookie manager on , then I will re-run the test with the cookie manager off to check that our validation (assertion) is working.

When I execute the test with the Cookie manager disabled , now we have a failed request although we have a 200 response code as shown on the following image.

Text Assertion is not the only assertion we can use but I think it is the mostly used one , and it will help you validate from the script side that you test is doing what should be done , help you have accurate results and have look about how your script and system under test is behaving.

*The JMX used in this article is uploaded here , feel free to use.

How to write data from JMeter response to a csv file

I though about it when i wanted to execute a data prepration script to generate some system ids and use them in a another script , but how can i get a certain value from the response and write to a file , CSV file specifically.

In this post i will tell you how i did it 🙂

Lets try to make it that way , we will use the random article function in wikipedia website to write the article name to a csv file , so everytime the random article is triggered JMeter will write the new article name to a CSV file.

The following will be added :

  • Thread Group
  • HTTP Sampler as shown below
  • View results tree
  • Regular expression extractor as a child to the HTTP Sampler
  • BeanShell PostProcessor as a Child to the HTTP Sampler

Every time “https://en.wikipedia.org/wiki/Special:Random” will requested the value in title will be written to the CSV file

Regular Expression Extractor configuration will be as shown below :

One step is remaining , to write the “Article_Name” parameter value to a CSV file

BeanShell PostProcessor code will be as the following :

artname = vars.get("Article_Name");
f = new FileOutputStream("Results.csv", true);
p = new PrintStream(f);
this.interpreter.setOut(p);
print(artname);
f.close();

Our last step is to run the script with more than one iteration , let’s execute it with 3 iterations and then go to see the CSV fie contents.

Hope you find this article useful 🙂

Increase Number of users generated from your local machine

Number of user generated from your local machine depends on your local machine specs but what if you can generate more users with number of tricks.

1. Don’t use listners

Listners consumes a lot of memory to be able to display information and do the necessary calculations , the less listners you use the more memory is available to generate users.

Use Simple Data Wriiter to store all of your run data and after the execution is finished you can get all information you need from .jtl file created.

*We have an article explaining how to use Simple data writer , see the following link

https://thetesttherapist.com/2019/01/27/save-run-results-with-simple-data-writer-in-apache-jmeter

2. Increase Java Heap Size Limit

We’re going to change the amount of memory reserved to JMeter by default to a larger size which allow JMeter to generate more users but you cannot set the memory size to be > 80% of your total system memory

Here is the default value in APache JMeter 5 , the default value is 256m

Screen Shot 2019-02-17 at 12.28.56 PM

You can set a new value which is not larger than 80% of your system memory , JMeter will not lunch if you set a memory which is much higher than possible.

*You can change the heap size value when you edit  jmeter.bat file and search for “set HEAP” in notepad or any other editor.

3. Use x64 JDK and Windows

With x64 Windows and JDK you can consume more memory and have a better memory management that will help you generate more users than before.

*With all the above tricks your machine specs cannot exceed a certain boundary related to your hardware , and unfortunately we cannot calculate precisely how much a machine with a specific specs can generate and also it is differ from test to test.But these tricks help you get the max out of your machine before thinking to add a new machine to generate more or move to the cloud.

Stop JMeter test run when reaching a specific number of requests

You want to execute a JMeter performance test but you don’t want to exceed a specific number of requests , is it possible?

Yes it is 🙂

What if we can get the value of the current iteration and make a condition to stop the test when reaching a specific values. This can be done as the following

Create a basic JMeter test plan with the essential samplers and listeners

Test plan will have the following :

  1. Thread Group
  2. Http Sampler
  3. If controller
  4. Test Action Sampler
  5. Summary Report Listener

Thread Group run settings will be as the following :

Thread group settings is just an example you can use what is suitable to your test

We will put If controller and a test action as a child for it in the beginning of the test just under the Thread Group.

Everytime you call ${__counter(,)} you got the current iteration number.
in the above example we need to set a maximum of 300 requests (iterations).

“${__counter(,)}” >= 301

So why i put the condition operator to be >=301 because in case of concurrency it will be easier to check for number of iteration after you already finished your 300 iterations , give it a try if you put 300 the test will be stopped at the iteration number 299.

Test Action will be a child to the IF Controller

When the condition is true the Test Action will stop the test.

Then you have to add http request , in the example we used “https://en.wikipedia.org/wiki/Main_Page”

At the end add summary report to check the number of request and run the test

Summary

With IF Controller and Test Action you can limit your test execution maximum number of requests While you executing a duration based test with multiple threads.

Save Run results with “Simple Data Writer” in Apache JMeter

One of the things that any new person who is dealing with JMeter is facing that if you save a jmx project which is executed before the run results is not saved.

So if you didn’t know this you may lose your run results and you have to repeat again.

One of the best ways to save your run data is using “Simple Data Writer” it not only make you able to save your run data but also enable you to use the data file on any listner or graph you want or need.

Data Files

CSV is one of the file extensions you can use to save your run results as the following example.

savedatatable

In the above example the results table will be saved to a .CSV file which is good in case you want only the data shown in the table but if you want more the Simple data writter will be a better option.

What are JTL files?

JMeter can create text files containing the results of a test run.

These are normally called JTL files, as that is the default extension – but any extension can be used.

Simple Data Writer

simpledatawritter

 You can configure the results save configuration the more you save the more listners you can use.

simpledatawritterconf

I used to select them all , sometimes this leads to larger size data file but this is safer to use in different graphs and listners as i will show you later.

Lets add simple data writter to thread group and disable all the other listners and execute test as the following.

simpledatawritterwithfile

datafile

After the run is finished i’m going to open the jtl file in the following listners :

  • View Results Tree
  • Summary Report
  • Active Threads over time graph
  • Hits per second graph

viewresultsjtl

summaryreportjtl

activethreadsjtl

As shown above the results data saved using Simple Data Writer can be used in any of the default JMeter listners and also extra listners which can be added to JMeter in the feature , .JTL extension is the default saving extensions which can be re-saved to CSV file if needed , see the following Wiki Page

https://wiki.apache.org/jmeter/JtlFiles

 

 

Get Max Number of User Concurrency in JMeter

In JMeter default lisnters some values are not provided by default and one of the examples is the max number of user concurrency which sometimes important if you’re executing a test which is not duration based.

In this article we’re going to execute a test with 200 users ramping 1 user each 2 sec and let’s see what is the max user concurrency we can get.

But first we need to install a plugin and add a listner as the following :

  1. install jmeter plugin manager from the link below
    https://jmeter-plugins.org/wiki/PluginsManager/
  2. After installation , open the plugin manager from jmeter and select the following plugin “3 Basic Graphs” from “Available Plugins” section
    Screen Shot 2018-07-14 at 11.52.10 AM
  3.  Make sure that you have a new listner under called “Active Threads Over Time” as the following picture
    Screen Shot 2018-07-14 at 12.02.43 PM

Now everything is in place lets make a quick run.

we’re going to use this fake API website “http://jsonplaceholder.typicode.com/” in our test

Screen Shot 2018-07-14 at 12.10.07 PM

Screen Shot 2018-07-14 at 12.10.34 PM

Screen Shot 2018-07-14 at 12.12.04 PM

Let’s execute and see the graph results 😉

Screen Shot 2018-07-14 at 12.19.43 PM

If we look at the graph above , we got a max of 1 concurrent user during the test run , so lets change the ramping up values and see the changes.

we will modify the thread group to be 200 threads / ramp-up Period : 50 as the following and execute again.

Screen Shot 2018-07-14 at 12.25.50 PM

Screen Shot 2018-07-14 at 12.27.09 PM

If we look at the graph above , in 2 times during the test we had more than 1 concurrent users. Because this is a simple http call with no business behind we got a smooth run and nearly constant results but in more real life test this graph will help you know the max number of concurrent users during your test.

 

 

Generate GUID in Apache JMeter

Although there is no dedicated sampler in JMeter to generate GUIDs but you can use the built-in functions to generate them.

In this article i will share with you how to generate GUID that is unique by user and iteration and use it through your JMeter thread group or test plan.

Steps:

  1. Open JMeter and add “Bean Shell Sampler”under your Thread Group

Screen Shot 2018-07-13 at 10.49.31 AM

2. Add a user defined variable that will hold the GUID value , set the varible name and leve the value empty

Screen Shot 2018-07-13 at 10.53.15 AM

3. Enter the following command in beanshell sampler
“vars.put(“VariableName”, “${__UUID}”);” where variable name is the name of variable created in the previous step.

Screen Shot 2018-07-13 at 11.01.10 AM

4. Now we need to verify that it is working so , we will add “Debug Samper” & “View Results Tree”

Screen Shot 2018-07-13 at 11.03.50 AM.png

Screen Shot 2018-07-13 at 11.05.45 AM

5. Save , Run and check the debug sampler values in the results tree

Screen Shot 2018-07-13 at 11.08.50 AM

6. now you can place the ${GUID} variable wherever you want in the thread group and it will generate a unique value for each user and each iteration (whenever the beanshell sampler executed).

*you can genrate more than one GUID everytime by pasting the line more than on time in the beanshell sampler and also add more variables in the user defined variables to hold the values.