Load testing with a specific number of users

I apologize in advance for this basic question but I am new to test automation.

I would like to know how to set up a load test with a predefined number of users (50). For now I have setup test cases and test suites for each of these users and execute the test suites within a test suite collection.

The problem is in the future, if I want to run that load test with 100 users I need to add more test cases and test suites. Since the test steps are about the same for each user, there has to be a more intelligent way to do this but I don’t know where to start.

Any help would be greatly appreciated.

Thank you.

You can overcome this by calling the same test cases multiple times in its loop, so that you just need to change loop variables to match with your desired. Below is a sample code which will loop 100 times to execute Login test case:

(1..100).each {
	WebUI.callTestCase(findTestCase('Common Test Cases/Login'), ['Username' : 'John Doe', 'Password' : 'ThisIsNotAPassword'])
}

Thanks a lot for the response Vinh. I have thought about using a loop but will this allow me to run my test for all my users simultaneously? I have a feeling if a implement a loop, it will do TC for user1, then TC for user2 and so on. I need to be able to run them at the same time to test the capacities of my AUT.

Hi! I have the same question. Any solution?

There is a way, but it’s a bit messy. You can add every single TC to a single Test Suite. Then add TSs to a Test Suite Collection and then you can run the TSC with a parallel execution option. See, told you it’s messy.

As for ‘load testing’ simulating 50 or 100 or more users, I would rather use JMeter (https://jmeter.apache.org/).
See https://octoperf.com/blog/2017/12/14/multiple-user-login-jmeter/
JMeter provides a lot of “Response time” analysis (https://jmeter.apache.org/usermanual/generating-dashboard.html), this would be big plus for performace study.

Mate Mrse said:

There is a way, but it’s a bit messy. You can add every single TC to a single Test Suite. Then add TSs to a Test Suite Collection and then you can run the TSC with a parallel execution option. See, told you it’s messy.

I already use Test Suite Collection with a number of same TSs. They successfully run in parallel, but the problem is, every instance uses the same test data (same username and password), while I want it to use one username per one execution instance. Any ideas?

The only solution I found is manually creating a Test Suite for each user to be tested. They are absolutely same, except the “Data Iteration” field in Test Data Binding, in each Test Suite it’s different: 1, 2, 3, etc.
If I do it this way, it runs the collection in parallel, each instance test its username & password, no duplication. BUT it’s a pretty stupid operation to just keep copy&pasting your test suites to imitate 10 or 50 users. Not elegant at all.

I think there is another solution.

In my comment to the post
How can I run the same test case for different URLs? - Katalon Studio - Katalon Community
I explained a method to run a Test Suite in Katalon Studio for multiple target URLs. The key feature is Execution Profile, which is new since v5.4

Execution Profile would enable you to run a Test Suite Collection which consists of multiple Test Suites, each of which different Profile is applied.

Have a look at the following screenshot. I have a TestSuiteCollection named ‘New Test Suite Collection’. The collection consists of 3 activation of a single Test Suite named ‘Test Suites/TS1’. What is important is that 3** activations are assigned with unique Profiles**: ‘default’, ‘develop’ and ‘product’

In the Profile named “default” I have :

And in the Profile named “develop” I have :

As you see, I have different set of Username & Password here.

You can add any GlobalVariable in the Profile as you like. For example, you may want to abstract the name of “Data Iteration” as an GlobalVariable, and you may want to assign values to the GlobalVariable using Profiles.

You do not have to create 100 Test Suites by copy, paste, slightly-modifying. A single TestSuite can be activated 100 times with different Profiles applied.

You need to prepare 100 entries of Execution Profile per individual user. I admit, this task is not elegant if you already have a list of URL,Username,Password pairs in Excel.

TestSuiteCollection_assigning_different_profiles_to_testsuites.png

Profile_default.PNG

Profile_develop.png

Here is another way to accomplish a “Poor Man’s Load Test” using an Excel Worksheet, one Test Case, one Test Suite and one Test Suite Collection…

1. Create a Test Case that is driven by data in an Excel Worksheet using the template Test Case displayed below…

Each row of the Excel Worksheet can be a new combination / permutation of URL, User Name and Password:
Column 1 is the URL for the Login Page
Column 2 is the User Name
Column 3 is the Password

Then, all you have to determine is how many rows you need to enter in the Excel Worksheet in order to load test all of the combinations / permutations for a single Test Case for your desired duration. Create hundreds / thousands of rows in the Excel Worksheet to ensure the Test Case runs for your desired duration…

2. Create multiple copies of a Test Suite that calls the Test Case created in step #1…

3. Create a Test Suite Collection that runs the multiple Test Suites created in step #2, each with your favorite browser, in “Parallel” Execution Mode…

4. Deploy Katalon Studio and the above Project to multiple workstations (PC / Linux Server / MacOS / Other) to avoid bandwidth issues using a single network segment…

******************************************

Test Case:
import static com.kms.katalon.core.checkpoint.CheckpointFactory.findCheckpoint

import static com.kms.katalon.core.testcase.TestCaseFactory.findTestCase

import static com.kms.katalon.core.testdata.TestDataFactory.findTestData

import static com.kms.katalon.core.testobject.ObjectRepository.findTestObject

import com.kms.katalon.core.checkpoint.Checkpoint as Checkpoint

import com.kms.katalon.core.checkpoint.CheckpointFactory as CheckpointFactory

import com.kms.katalon.core.mobile.keyword.MobileBuiltInKeywords as MobileBuiltInKeywords

import com.kms.katalon.core.mobile.keyword.MobileBuiltInKeywords as Mobile

import com.kms.katalon.core.model.FailureHandling as FailureHandling

import com.kms.katalon.core.testcase.TestCase as TestCase

import com.kms.katalon.core.testcase.TestCaseFactory as TestCaseFactory

import com.kms.katalon.core.testdata.TestData as TestData

import com.kms.katalon.core.testdata.TestDataFactory as TestDataFactory

import com.kms.katalon.core.testobject.ObjectRepository as ObjectRepository

import com.kms.katalon.core.testobject.TestObject as TestObject

import com.kms.katalon.core.webservice.keyword.WSBuiltInKeywords as WSBuiltInKeywords

import com.kms.katalon.core.webservice.keyword.WSBuiltInKeywords as WS

import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUiBuiltInKeywords

import com.thoughtworks.selenium.webdriven.commands.RunScript as RunScript

import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUI

import internal.GlobalVariable as GlobalVariable

import net.sourceforge.htmlunit.corejs.javascript.ast.ForLoop as ForLoop

import org.openqa.selenium.Keys as Keys

for (def row = 1; row <= findTestData(‘Login’).getRowNumbers(); row++) {

WebUI.openBrowser('')



WebUI.navigateToUrl(findTestData('Login').getValue(1, row))



WebUI.setText(findTestObject('Page_Login/Username'), findTestData('Login').getValue(2, row))



WebUI.setText(findTestObject('Page_Login/Password'), findTestData('Login').getValue(3, row))



WebUI.click(findTestObject('Page_Login/Sign In'))



WebUI.delay(5)  
  
//Call a Test Case that does what you would like each user to do in the AUT that is being load tested  

WebUI.callTestCase(findTestCase('What you want to do next in your application'), \[:\], FailureHandling.STOP\_ON\_FAILURE)  



WebUI.closeBrowser()  

}

kazurayam said:

You can add any GlobalVariable in the Profile as you like. For example, you may want to abstract the name of “Data Iteration” as an GlobalVariable, and you may want to assign values to the GlobalVariable using Profiles.

I can’t add “Data Iteration” as a variable to a profile, because it doesn’t allow spaces in the variable names. How do I make “Data Iteration” work as a variable? It seems that Test Suite expects it to be a normal Number, I can’t find a way to make it a variable.

Roman,

You wrote:

They are absolutely same, except the “Data Iteration” field in Test Data Binding,

I do not see what “Data Iteration field” means here. Could you attach some screenshot that shows your “Data Iteration field”?

in each Test Suite it’s different: 1, 2, 3, etc.

Could you also attach some screenshot that shows how 1,2,3 appears

Data Iteration is a field to assign which row of the Test Data you are going to use in this Test Case. If multiple rows are chosen (or all), it will iterate through them (will run this test case sequentially once per each test data entry).

My task, again, is to assign one test data entry to one test suite execution instance, and have them all run in parallel with their own test data each.

image.png

I understand that you are given with following requirements:

(1) a Data file (excel,csv etc) which contains 50 lines. Each line records a credential pair (username and password)

(2) a Test case has been developed. It is designed to accept a single pair of username and password and runs all necessary assertions with the credential pair.

(3) You want to execute the tests for multiple (at most 50 ) credential pairs in parallel.

And I understand that you have done following stuff:

(a) You have registered your Data file into the Katalon Studio project as described in https://docs.katalon.com/display/KD/Manage+Test+Data

(b) You have created necessary test cases.

(c) You have created 50 test suites. You bind each test suites with a credential pair recorded in each line of the Data file.

(d) You have created a Test Suite Collection which bundles 50 test suites.

(e) You execute the Test Suite Collection in the Parallel mode.

… there has to be a more intelligent way to do this …

I think what you have done is the best possible solution as of now using the Katalon Studio’s supported features.

If I were to develop “a more tricky way”, I have an idea. Let me briefly describe it.

I want to generate Execution Profiles for 50 users and generate a Test Suite Collection. I would develop a script which reads your Data file and generates 2 XML files.

You can find examples of target XML files in your project directory:

1. %katalon_project_folder%/Test Suites/.ts
2. %katalon_project_folder%/Libs/internal/GlobalVariable.groovy

I think that only these 2 files need to be generated. No other files are to be concerned.

This is a hack, not a supported feature.
Sorry, I have never tried it. But I suppose it would work.

kazurayam said:
I think what you have done is the best possible solution as of now using the Katalon Studio’s supported features.

Oh ok then. Thank you for sharing your knowledge! I’ll also maybe try the trick you described, but will probably stick to the existing solution if you consider it the best possible so far.

Hello everyone,

Currently, you can integrate Katalon Studio with JMeter via the JMeter Integration plugin which is free. Alternatively, you can manually integrate those two tools yourself by following this guide.

Cheers

Jass