Random execution order of test cases for each run

Product Forums Katalon Studio
,
1

I’m looking for a way so that every time a test suite is executed, it will pick a random order without repeating any test case similar to <runOrder>random</runOrder> in Junit. Is there a way to achieve the same?

2

Interesting problem :sweat_smile: As far as I’m aware, there’s no built-in way to configure this. However, I think this might serve as a workaround:

Create a separate Test Case that aggregates all of the test cases in your current suite, and calls them in a random order:

import static com.kms.katalon.core.testcase.TestCaseFactory.findTestCase

import com.kms.katalon.core.testcase.TestCase as TestCase
import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUI

List<TestCase> testSuite = new ArrayList<TestCase>();
testSuite.add(findTestCase('path/to/first/testCase'));
testSuite.add(findTestCase('path/to/second/testCase'));
testSuite.add(findTestCase('path/to/third/testCase'));
.
.
.

while(testSuite.size > 0) {
	TestCase testCase = testSuite.get(new Random().nextInt(testSuite.size()));
	WebUI.callTestCase(testCase);
	testSuite.remove(testCase);
}

Then you would create a new Test Suite, and just add this one Test Case. Of course, this isn’t ideal. One of the major drawbacks here is that your report will be wonky, as there is only one “test case” running a bunch of other test cases.

Out of curiosity, what is the motivation behind the random execution order? In other words, what information do you get about the AUT from running the test cases randomly as opposed to sequentially?

If it’s a crucial feature that you would like to see in the studio, I would recommend posting a feature request here:

http://forum.katalon.com/c/katalon-studio/katalon-studio-feature-improvement-suggestions

2 Likes
3

So you could ‘assume’ that the test cases are executed randomly already.

4

Hey @Brandon_Hein,
Thanks for the solution. But that doesn’t suffice my requirement as we need detailed reports at the end of run. When we execute test cases in same order, we are aware of the device state before next test case. Many issues were identified when we execute test cases randomly, as device state will be different for each run before a particular test case. This is my motive behind random execution order.

5

Let me give you solution with example.

Lets say you have TestSuite as Suite_1. Now if you open explorer and navigate to Folder “Test suites” under your project directory. You will find 2 files named as below.

  1. Suite_2.groovy
  2. Suite_2.ts ( has list of test cases)

Now, Create Keyword which reads xml file and write xml - where you wrire logic to randamize test case sequence.

Call this keyword in Suite_2.groovy file in Setup methog.

Now Execute TestSuite.

This method is not verified. So please verify it before using it.

6

@Rudra

Thanks for the solution. But I haven’t find .groovy file for TestSuiteCollection. Can you please let me know if we should create .groovy file and add the code or will it be created on it’s own.

Also .ts file of TestSuiteCollection contains the list of TestSuites, not the TestCases. So even if the keyword which will update the .ts file is called in TestSuiteCollection, it will execute TestSuites randomly but not TestCases.

7

@Brian_Ducson They aren’t executing randomly. TestCases will be executed in the order mentioned in TestSuite. I didn’t get what you mean by ‘assume’ that the test cases are executed randomly already.

8

just kidding. I meant when you expected it run randomly, running in an order is also a case of running randomly. There should be no problem with that. Normally, user expect that the test cases should run in order, instead of randomly.

9

@Brian_Ducson as mentioned in above replies, we had found some issues when we execute test cases in random order because state of our device is unaware. That’s the reason I’m looking for such feature.

10

Hi @manikumar.pulipati

I see, if there is no .groovy file for SuiteCollection then we can directly write same logic in TestSuite.groovy I think I was too brief in my answer above. Let me give you solution with example.

Lets say you have TestSuite as Suite_1. Now if you open explorer and navigate to Folder “Test suites” under your project directory. You will find 2 files named as below.

  1. Suite_2.groovy
  2. Suite_2.ts ( has list of test cases)

Now, Create Keyword which reads xml file and write xml - where you wrire logic to randamize test case sequence.

Call this keyword in Suite_2.groovy file in Setup methog.

Now Execute TestSuite.

This method is not verified. So please verify it before using it.

11

@Rudra Thanks for the solution. But I’m looking for a solution where first test case will be from TestSuite1 and next will be from another TestSuite say TestSuite4. I want something to randomize in such a way.