How to Set your performance testing acceptance criteria

This is always a question for people who are doing performance testing (as a general term) for the first time and also those who don’t have a specific performance requirements.

  • What numbers to compare to?
  • what are the current response time means?
  • is it good or bad?
  • How to set an acceptance criteria?

All the above questions we used to hear in the beginning of any performance testing project or even a discussion about a future need for a testing.

I will list here some ideas that will help you determine and simplify the process of setting a test acceptance criteria.

  • Check websites/services working in the same domain : gather information about how their services response times are , and compare it to your current response times and if you’re way higher than them you have to plan enhancements for your current operating service(s).
    *Start with your local competitors as both of you are operating in the same market.
  • Some organizations / sites publish a yearly report about the web performance in general and categorized it by business domain , this will help you have an overview about the response times trend and have at least numbers that you don’t want to exceed anyway.
    *The full article link can be found in the end of this article.
IndustryUnited StatesUnited KingdomGermanyJapan
Automotive9.5 sec12.3  sec11.0 sec10.3 sec
Business & Industrial Markets8.7 sec8.3 sec8.2 sec8.1 sec
Classifieds & Local7.9 sec8.3 sec7.0 sec8.3 sec
Finance8.3 sec8.0 sec8.6 sec7.6 sec
Media & Entertainment9 sec8.8 sec7.6 sec8.4 sec
Retail9.8 sec10.3 sec10.3 sec8.3 sec
Technology11.3 sec10.6 sec8.8 sec10sec
Travel10.1 sec10.9 sec7.1 sec8.2 sec
While the average of the values in the table is 8.66 sec, the recommendation for 2018 is to be under 3 seconds.
  • If you are doing a revamp or replacement to an old system/service try to achieve at the least the same old system performance (in case the performance wasn’t the reason for the revamp 🙂 ) and then you can plan for 20 – 30% better performance than the old system , off-course you can plan for a higher performance achievement but it should be specific to not wasting a lot of time chasing unclear goal.

To summarize , it is ok if you don’t have a specific performance requirements , you can set your requirements based on how you’re operating comparing to the others and also having initial goal is a good step to start plan your performance enhancements and for sure those goals will be more ambitious by time.

Sources :

Please share your tips, experience, comments, and questions for further enriching this topic of discussion.

Leave a Comment

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.