Run multiple test cases simultaneously

Product Forums Katalon Studio
1

I have some testcase:

  • Testcase 1
  • Testcase 2
  • Testcase 3
  • Testcase 4
  • Testcase 5
    I want to run Testcase 1->4 at the same time and after that run Testcase 5.
    Is it possible with Katalon? And how can i do it?
1 Like
2

No, you can not run multiple test cases simultaneously.

3

Just for your interest, please have a look at a previous post of mine:

4

I believe you need to purchase a license for Katalon Runtime Engine in order to run scripts in parallel, at least in an official capacity in Katalon:

From there, it really depends on what your scripts are doing and how process-intensive they are, and how many resources you have to work with. It’s perfectly possible to run 4-8 threads simultaneously on a modern machine with 32G of memory and multiple cores. In a normal scenario, the largest resource hog is going to be browser tabs, so if you can run your scripts headlessly, even better.

Ultimately, @kazurayam’s advice is sound, if you really want to run your suites efficiently and quickly, you should invest in a distributed system where multiple machines can execute in parallel. This is where cloud services come in.

In the meantime, check out the above link for Katalon Runtime Engine.

Cheers!

1 Like
5

Hi @antt ,

In theory, it is possible.
Setup 4 testSuites with each testSuite one test case (from testCase 1 to testCase 4).
Add testCase 5 to any of those 4 testSuites, preferably to the one that takes the longest (e.g. to testSuite n° 4).

Add these testSuites to a test suite collection and enable “Parallel” Execution mode (set max. concurrent instances to 4).
This will make each testCase from 1 to 4 run in parallel. As soon the first testCase in testSuite n° 4 is done it will start up testCase 5 in that same testSuite.
But of course you want to be sure that testCase 1, 2 and 3 are also finished before you start this testCase 5. To do that, you could either write a very long sleep before you start testCase 5 (not so elegant) or use some cross-test-suite variable to flag whether all testCase 1-3 are finished. Example: for testSuite n°1, 2 and 3 write a tearDown script that will create variable to an excel. Example for testSuite 1 a property “testCase1Finished” with value “true”. (Why in a tearDown and not just at the end of your testCase? Imagine the testCase 2 fails at one point, it might never reach that line of code where you write your “true” and your testCase 5 will wait unnecessarily long)
In the testCase 5 you then need to write some script to read those values from the excel(s) in a while loop and only continue with your testCase 5 when all 3 values are “true” (testCase 4 was done sequentially in your testSuite 4). This also implies a reset to “false” of these 3 variables, preferably at the startup of each testSuite.

This is not ideal. Writing to 1 excel in parallel is not recommendable, so you should for each testCase have a dedicate excel to avoid concurrency issues. Not really sure anymore whether a read of the excel might interfere with the write… And in general, parallel runs might behave different than expected like mentioned above. But if your whole project just consist of these 5 (small) test cases you might pull it off this way (and I think it might be do-able without a Katalon Runtime Engine license, but not sure on that).

Note: You could also set testCase5 to a dedicated testSuite n°5 and set the max. concurrent instances to 4, but you’d still face the problem that a run of testCase 5 will be initiated before ALL other testCase 1to4 are done.

On how to write properties to an excel for potential use between test suites, see my post here:

6

If you use some Continuous Integration tool (Jenkins, CircleCI etc), you will be able to look at your problem with light from new angles.

See the following article for Jenkins Pipeline:

(I don’t know if Katalon provide any CI/CD producet competitive to Jenkins. I just don’t know it. No reason. I am lazy)

Here I would assume 2 things

  1. You have enough number of Katlaon Runtime Engine licenses

  2. You wrap your Test Cases with Test Suites, like this:

.
├── Testsuite1
│   └── Testcase1
├── Testsuite2
│   └── Testcase2
├── Testsuite3
│   └── Testcase3
├── Testsuite4
│   └── Testcase4
└── Testsuite5
    └── Testcase5

You can execute Testsuite1 from command line using KRE. You can execute Testsuite2 from command line using KRE, and so on.

Now I start writing Jenkins Pipeline so that is execute the stages of Testsuite1-4 in parallel, wait for all these 4 stages to finish; ant then execute the stage of Testsuite5.

A psued Jenkins Pipeline code will look like this:

pipeline {
    agent any
    stages {
        stage('Build') {
            steps {
                sh 'echo "Building the application"'
                // Add commands to build application
            }
        }
        stage('Test') {
            parallel {
                stage('first stage) {
                    steps {
                        sh 'katalonc --testsuite TestSuite1 ............'
                        // Add more commands if required
                    }
                }
                stage('second stage) {
                    steps {
                        sh 'katalonc --testsuite TestSuite2 ............'
                    }
                }
                stage('third stage) {
                    steps {
                        sh 'katalonc --testsuite TestSuite3 ............'
                    }
                }
                stage('fourth stage) {
                    steps {
                        sh 'katalonc --testsuite TestSuite4 ............'
                    }
                }
            }
        }
        stage('finalize') {
            steps {
                    sh 'katalonc --testsuite TestSuite5 ............'   
            }
        }
    }
}