How to have the test suite drive its test case variables?

This question is probably an ongoing source of complaining about Katalon Studio, but…

I have this test suite, called Discount, with 42 test cases in it. Each test case accept, as variable, a practiceProfile, with the same default value: com.xxx.profiles.PracticeProfile.SMD_DEFAULT.

Unfortunately, somewhere along the multiple runs of that test suite, I caused a blocker of a bug on the AUT, and can’t run any more tests with the default profile.

I wish to install some control, in the test suite, via which I can change the practiceProfile passed to all the test cases.

I do have the following PracticeProfiles, and I know I could change it from here:

public final class PracticeProfiles {
	public static final PracticeProfile SMD_DEFAULT = new PracticeProfile(OrganizationNames.SMD_BETA, PracticeNames.DEMO_PRACTICE, PracticeURLs.DEMO_PRACTICE);
	public static final PracticeProfile SMD_IMPERIAL = new PracticeProfile(OrganizationNames.SMD_BETA, PracticeNames.IMPERIAL_MEDICAL_CENTER_II, PracticeURLs.IMPERIAL_MEDICAL_CENTER_II);
	public static final PracticeProfile PARAGON_DEFAULT = new PracticeProfile(OrganizationNames.PARAGON, PracticeNames.PARAGON_DEMO_PRACTICE, PracticeURLs.PARAGON_DEMO_PRACTICE);

	// the MDG practices
	public static final PracticeProfile MDG_1 = new PracticeProfile(OrganizationNames.SMD_BETA, PracticeNames.MDG_1, PracticeURLs.MDG_1, NPINumbers.MDG_1);
	public static final PracticeProfile MDG_10 = new PracticeProfile(OrganizationNames.SMD_BETA, PracticeNames.MDG_10, PracticeURLs.MDG_10, NPINumbers.MDG_10);
	public static final PracticeProfile MDG_18 = new PracticeProfile(OrganizationNames.SMD_BETA, PracticeNames.MDG_18, PracticeURLs.MDG_18, NPINumbers.MDG_18);
	public static final PracticeProfile MDG_19 = new PracticeProfile(OrganizationNames.SMD_BETA, PracticeNames.MDG_19, PracticeURLs.MDG_19, NPINumbers.MDG_19);
	public static final PracticeProfile MDG_20 = new PracticeProfile(OrganizationNames.SMD_BETA, PracticeNames.MDG_20, PracticeURLs.MDG_20, NPINumbers.MDG_20);
	public static final PracticeProfile MDG_21 = new PracticeProfile(OrganizationNames.SMD_BETA, PracticeNames.MDG_21, PracticeURLs.MDG_21, NPINumbers.MDG_21);
	public static final PracticeProfile MDG_22 = new PracticeProfile(OrganizationNames.SMD_BETA, PracticeNames.MDG_22, PracticeURLs.MDG_22, NPINumbers.MDG_22);
	public static final PracticeProfile MDG_23 = new PracticeProfile(OrganizationNames.SMD_BETA, PracticeNames.MDG_23, PracticeURLs.MDG_23, NPINumbers.MDG_23);
	public static final PracticeProfile MDG_24 = new PracticeProfile(OrganizationNames.SMD_BETA, PracticeNames.MDG_24, PracticeURLs.MDG_24, NPINumbers.MDG_24);
	public static final PracticeProfile MDG_27 = new PracticeProfile(OrganizationNames.SMD_BETA, PracticeNames.MDG_27, PracticeURLs.MDG_27, NPINumbers.MDG_27);
	public static final PracticeProfile MDG_28 = new PracticeProfile(OrganizationNames.SMD_BETA, PracticeNames.MDG_28, PracticeURLs.MDG_28, NPINumbers.MDG_28);
	public static final PracticeProfile MDG_40 = new PracticeProfile(OrganizationNames.SMD_BETA, PracticeNames.MDG_40, PracticeURLs.MDG_40, NPINumbers.MDG_40);
	public static final PracticeProfile MDG_41 = new PracticeProfile(OrganizationNames.SMD_BETA, PracticeNames.MDG_41, PracticeURLs.MDG_41, NPINumbers.MDG_41);
	public static final PracticeProfile MDG_50 = new PracticeProfile(OrganizationNames.SMD_BETA, PracticeNames.MDG_50, PracticeURLs.MDG_50, NPINumbers.MDG_50);
}

however, doing so would be incorrect.

Just a shot in the dark, is there any way I could have the test suite drive the test cases like this?

I’m not really sure what you’re asking, but the PracticeProfiles class could be imported (via its package) by every test case. Each Test Case could then access SMD_DEFAULT.

If you want to start out with different values, you could set that up in a Test Listener.

But, repeat, I don’t really understand what you’re having trouble with.

The issue isn’t lack of access to the PracticeProfiles test case.

It is that, every test case has variable called practiceProfile that defaults to PracticeProfiles.SMD_DEFAULT, and I would like the test suite to use a different practice profile for all the test cases (e.g. PracticeProfiles.SMD_IMPERIAL)

Also, moving to Test Listener is incorrect, as it would run with every test case being ran, not just the ones in the Discounts test suite.

Not for me, it isn’t. I have code in a listener that reads a suite config file – it’s easy to publish new contents to the config before execution of the suite.

In my case, I have setup and teardown steps, in an already-existing Test Listener, that every test case need to execute (e.g. opening browser window, running the login test case, and quitting the driver).

How do I make sure that, I can create another Test Listener, specific to these test suite concerns, and separate from the universal test case concerns?

Me too, I’m not really sure what you’re asking.

Could you show us your code or any screenshot of your Discount test suite and explain how the Discount passes the default value to the practiceProfile variable in each test cases?

It may be obvious to you, but not clear to others in the forum.

Right now, there’s no mechanism in the test suite to drive the variables of the test cases. The test cases are simply added to the test suite, and no more than that.

Each test case, however, has in its variables collection, a single variable called practiceProfile. Here’s the variables (script mode) for one of the test cases:

<?xml version="1.0" encoding="UTF-8"?>
<Entity xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="variableEntityWrapper">
   <description></description>
   <tag></tag>
   <variables>
      <defaultValue>com.xxx.constants.PracticeProfiles.SMD_DEFAULT</defaultValue>
      <description></description>
      <id>14590671-fc11-4215-a5c1-25e0d5943163</id>
      <masked>false</masked>
      <name>practiceProfile</name>
   </variables>
</Entity>

Still I am not sure if I understood your problem, but just a thought…

You should not use a Test Suite in this case, as you have pointed out

  1. a Test Suite can not pass parameters into comprising Test Cases.
  2. you can not implement in a Test Suite any control (if … then … else … , for, try … catch) over the the sequence of calling Test Cases.

You have an alternative. There is a WebUI keyword callTestCase

You can create a Test Case called Discount which calls another test case, namely WorkerTC, 42 times. The Discount will call WorkerTC with an argument named practiceProfile while specifying 42 variations of static instances of PracticeProfile class.

The following screenshot shows my idea:

A “Controller” test case calling another “Worker” test case enables you to solve these 2 problems. You can pass runtime parameters; your “Worker” can return any value to the “Controller”. The Controller can dynamically chose process sequence based on the returned value from the Worker.

The only concern about this design is that the Report becomes less informative. The Report will show you a log of only the single Test Case “Discount” = the caller test case. The Report won’t show you 42 sets of “callees” distinguished.

Alternatively, you can create a Test Suite Collection which comprises with 42 invokations of a Test Suite “Discount” while specifying 42 ExecutionProfiles. Each Excecution Profile species the property name of “PracticeProfile”. Your test case script will be informed of the property name via GlobalVariable. You can parameterise your test case script by this. The Report will show 42 entries of test case execution logs.

However a Test Suite Collection does not enable you to implement any “flow control”. Probably you would find it not enough.

You are wrong.
There is such mechanism, it is named Data Binding, part of the Data Driven testing feature.
see:

Although the documentation does not clearly say it, it is possible to use it also without a Test Data file.
To demo this, I created a test case having a variable with a default value:

When I run this, the output is (I have only a print in the code):

2022-06-04 10:17:10.345 DEBUG testcase.New Test Case                   - 1: comment("some comment")
2022-06-04 10:17:10.424 INFO  c.k.k.c.keyword.builtin.CommentKeyword   - some comment
2022-06-04 10:17:10.425 DEBUG testcase.New Test Case                   - 2: println(myvar)
default value
2022-06-04 10:17:10.444 INFO  c.k.katalon.core.main.TestCaseExecutor   - END Test Cases/New Test Case

Now, I made a Test Suite. In data binding I didn’t add a Test Data but I am just overriding the variable value by using script variable for Type:

When I run the suite, the output is:

2022-06-04 10:22:50.242 DEBUG testcase.New Test Case                   - 1: comment("some comment")
2022-06-04 10:22:50.324 INFO  c.k.k.c.keyword.builtin.CommentKeyword   - some comment
2022-06-04 10:22:50.325 DEBUG testcase.New Test Case                   - 2: println(myvar)
override value
2022-06-04 10:22:50.336 INFO  c.k.katalon.core.main.TestCaseExecutor   - END Test Cases/New Test Case
2022-06-04 10:22:50.507 INFO  com.kms.katalon.core.util.KeywordUtil    - Start generating HTML report folder at: /home/ibus/Katalon Studio/test/Reports/20220604_102247/New Test Suite/20220604_102247...
2022-06-04 10:22:50.578 INFO  com.kms.katalon.core.util.KeywordUtil    - HTML report generated
2022-06-04 10:22:50.668 INFO  c.k.katalon.core.main.TestSuiteExecutor  - --------------------
2022-06-04 10:22:50.668 INFO  c.k.katalon.core.main.TestSuiteExecutor  - END Test Suites/New Test Suite
2022-06-04 10:22:50.668 INFO  c.k.katalon.core.main.TestSuiteExecutor  - ====================

Is this what you need?

1 Like

In addition to the above, you can also use a certain Global Variable as ‘value’ for the Script Variable type:

So, you can create as many profile as you need and just run your suite with the desired one.
Top of that, you can create a Test Collection where you add the same Test Suite multiple times but set for each one the desired profile for execution.

1 Like

As a side note, you can set a certain global variable also straight at the test case level.
So it is only a matter of what workflow do you need.
If you need to override a test case variable no matter what is the test suite, set it at the test case level and store all defaults in the ‘default’ profile. Create as many profiles as you need.
If you need to do the override only for a/some selected test suite(s), either with a fixed value or with a global variable, use the above method.

This only works for one test case. Applied to my scenario, you’d have to do that for every last test case in the test suite, and even then, how can you use variable in the test suite, and where?

The test suite doesn’t drive all the test cases with this.

This may work. I don’t necessarily like it, as not only does it lift state up from the test suite itself, but it means that I have to go back and change configuration for every last test case in the test suite, but given my situation, I may accept it.

I think, ideally, you want a bunch of profiles (containing your vars) which you could instruct a suite to use by a simple setting – and somehow automate that.

So why not create the profiles (profile1, profile2, …) and have Jenkins (or some other shell process) copy-rename the target profile over the top of the default profile before the suite starts?

I want just one of those, different from SMD_DEFAULT, passed in each of the test cases, but that one, whichever it is, to be set by the test suite, or something similar.

I’m sorry, but what is difficult to understand about that?

To answer that I would need to understand what your problem is. If I understood your problem, it wouldn’t be difficult to understand. QED.

Yeah, why I said “Jenkins”, or a shell script – because there’s no facility to do what you’re trying to do from a suite (we believe).

Michael, with respect, take a step back and realize this – if we are not getting it, you’ve managed to confuse and confound 3 of the top “experts” on this forum. If the answers you’re getting are frustrating you, look to the way you’re asking your questions.

One last shot from me:

  1. Your suite(s) reads a profile/config file with a fixed, well-known name. Let’s call it MIKE.

  2. Build a bunch of unique profiles/config files with the unique contents you require.

  3. Using whatever launch mechanism you prefer (batch, Jenkins, whatever), copy profile1 over MIKE. Launch the suite.

  4. Copy profile2 over MIKE. Launch the suite.

And so on.

If that doesn’t float your boat, make a new Feature Request.

@mwarren04011990

Still I do not understand what your problem is. If you show more of your code, then we would understand it better…


I guessed and created another sample code. Let me show it to you.

  1. In the Execution Profile default, I created a GlobalVariable named PRACTICE_PROFILE of value type Null with initial value null.

WARNING The value type must be “Null”. You should not specify “String”. Otherwise this sample would not work.

NOTE The value type Null in the Execution Profile GUI effectively means java.lang.Object, which is the most generic type and can be substituted with any types of Java/Groovy objects like com.xxx.profiles.PracticeProfile.SMD_DEFAULT.

Further NOTE I would like KS to show, in the Execution Profile GUI, Object rather than Null, as Null looks to be something malicious.

  1. In the Keywords folder, I created a package com.xxx.profiles. In the package, I created an enum named PracticeProfile. It is a mimic of your PracticeProfile class you showed to us in the original post. The enum contains several instances: SMD_DEFAULT, MDG_10, etc. You can make more, of course.

  2. I made 3 Test Cases: init, TC01, TC42. The TCx test cases are meant to be a mimic of your test cases.

  3. The TC01 test case has the following code:

import internal.GlobalVariable as GlobalVariable
def practiceProfile = GlobalVariable.PRACTICE_PROFILE
println "[TC01] practiceProfile is " + practiceProfile.toString()

It substitutes the value of GlobalVariable.PRACTICE_PROFILE into the practiceProfile variable. Then it shows the value of a variable practiceProfile. The TC42 does quite similar.

  1. The init test case is the core of my proposal. Its code looks like this:
import com.xxx.profiles.PracticeProfile as PP

import internal.GlobalVariable

GlobalVariable.PRACTICE_PROFILE =
	PP.SMD_DEFAULT;
	// PP.SMD_IMPERIAL;
	//PP.PARAGON_DEFAULT;
	// PP.MDG_1;
	//PP.MDG_10;
	
println "[init] GlobalVariable.PRACTICE_PROFILE is set with PracticeProfile." + 
			GlobalVariable.PRACTICE_PROFILE.toString()

The init sets the GlobalVariable.PRACTICE_PROFILE with one of the enum instance declared in the PracticeProfile object.

  1. I made a Test Suite named Discount. The Test Suite calls 3 test cases. It calls the init test case first. And then it calls other test cases.

  2. When I write the init so that it sets SMD_DEFAULT into the GlobalVariable.PRACTICE_PROFILE and run the test suite, I got the following output:

[init] GlobalVariable.PRACTICE_PROFILE is set with PracticeProfile.SMD_DEFAULT
[TC01] practiceProfile is SMD_DEFAULT
[TC42] practiceProfile is SMD_DEFAULT
  1. I can easily change the init so that it sets MDG_10 into the GlobalVariable.PRACTICE_PROFILE and run the test suite, I got the following output:
[init] GlobalVariable.PRACTICE_PROFILE is set with PracticeProfile.MDG_10
[TC01] practiceProfile is MDG_10
[TC42] practiceProfile is MDG_10

Please note that you can switch the value of practiceProfile variable in TC01TC10 easily. All you need to do is to change a single code: the init test case.

A Test Suite can not supply values into a variable in its test case. A Test Suite is just a bunch of Test Cases. You shouldn’t expect much out of Test Suite.

Instead, I introduced a GlobalVariable.PRACTICE_PROFILE and a test case init.

Does this give you any hint?

I understand your pain.
That’s why is a good idea to read the documentation of a certain product entirely before actually starting to develop mega-projects so you can address various issues in due time.
Now, indeed you will have to update each test case plugged in all test suites, but hey … you only have to do it once.

LE: or, use the globalvariable approach at the testcase level. less to edit