Parallel execution of different types of tests(sequential for the same type)

#1

Hi,

I have two sets of testsuites that have the following constraints:
TSType1 (N number tests) - these TS can only run sequential - technically they are just one test suite with a different data iteration to workaround issues with report generation
TSType2 (O number tests)- these TS can only run sequential - same reason as the other TSType

However, you can run TSType1 test with TSType2 tests in parallel.

Normally, you can think of just running all sequentially. However, to optimize for time, you can technically run TSType1 and TSType2 together. I’m looking at the documentation and I can only see that you either run sequential or parallel.

In my configuration, I have 2 test collections. TCTSType1 and TCTSType2 which are both sequential. At first, I thought I can nest it to make the 2 collections parallel (within the inner collection as sequential) but it seems that you can’t do it(you can only select TS and not TC within a TC).

I’m not sure if I did present my case clearly, so please ask for clarification. Do you guys have recommendations for me to optimize for test execution? The tests needs to have report to indicate the data variation that succeded and failed.

#2

I am afraid not.

As far as I know, a Test Suite Collection can contain Test Suites only. A Test Suite Collection can not contain other Test Suite Collections in it.

#3

@duyluong

Any recommendation from Katalon Team?

  1. Does he need KRE?
  2. Should he distribute 2 processes for Test Suite Collections on 2 VM instances?
  3. Should he use TestOps to orchestrate 2 parallel executions of Test Suite Collection? Orchestration means starting 2 tasks at the same time with a trigger and consolidating the results (reports, etc).
#4

Or two docker containers running on the same machine / vm.
However, on both cases, two KRE licences are needed.

#5

@david.casiano,

For the above specifications, you can setup two different Test Suite Collection (TSC) that each one contains TSType1 or TSType2.

  • You can run those TSC in two different KRE sessions in parallel.
  • You can also run those TSC in a single KS/KSE session but we don’t recommend this way because it might not get better performance than KRE does.
#6

I thought about the 2 KS, but as you mentioned, we need 2 licenses. Before the workaround of splitting data across multiple TS(to improve performance and making sure reports will be generated), the solution will just be putting the 2 TS together in a test collection (and you make the execution parallel). However, in the splitted data(you spread the execution to multiple Test Suite), I’m not sure how to make sure that there is only one instance of that type running at the same time(because they need to run sequentially).

Thanks!

#7

I know KS/KSE might not get better performance than KRE does because KS/KSE has Log Viewer which could make performance worse. See Log Viewer slows down your tests. How to prevent it?. But I am not sure if @duylong mean that there could be any other factors (other than Log Viewer) that makes KS/KSE slower than KRE.

@duyluong

What is the factor(s) you can think of that makes KS/KSE slower than KRE?

#9

How about this?

  • A Test Suite Collection (TSC) contains TSType1 and TSType2.
  • TSC is configured to execute TSType1 and TSType2 parallel.
  • You have one machine, on it you have a KS/KSE, on it you run the TSC.
  • Then, TSType1 and TSType2 would run parallel.
  • You do not need to introduce multiple Test Suite Collections. You need only one TSC.

Did I misunderstand? I am not sure what is your problem.

#10

That was the idea before, to have both test suite (TSType1 and TSType2) in a test collection as a parallel test. However, I have to split the data of TSType1 to multiple test suites (thus, it can run in parallel in a test collection which will be forbidden-if I enable parallel execution in test collection). Same is true for TSType2. It is a single test suite split into multiple test suites(so you can run data in Data Files in sections so you can get a report after finishing N data items).
I’m not sure if there is a thread affinity type of feature where you can set a type of tests to let say 1, 2 (if you set your max. concurrency to 2). I have the misfortune of splitting the data of a data files to multiple suites so KS can run faster (yes, splitting data of a big data file in a test suite, e.g. 20K entries in CSV-split the test suite so TS1:1-500, TS2: 501-100, etc.)

#11

Perhaps you want to write a small Groovy script (or shell script with grep or awk) that splits the source data file into pieces (e.g. 20K entries in CSV-split the test suite so TS1:1-500, TS2: 501-1000, … etc as you wrote) before running the Test Suite Collection. Is it difficult to implement?

#12

Actually, I don’t know the format of KS tc, ts so I have not look into it. I inquired if they do have the script to do it and found not yet. It will be great if there is one.

1 Like