Design a Test Suite


This is a companion discussion topic for the original entry at https://docs.katalon.com/katalon-studio/docs/design-a-test-suite.html

Hi,
I’m trying to setup a test suit but I want to reference to the index of the Test Data (“No.” column)
under Variable Binding there is an option for Type called Data Column Index but I cannot figure out that value to use since It doesn’t allow me to leave it blank, I also tried “No.” as it is on the column name but it does not work either.
I want to find a good solution, I do not want to create a whole other column just as index, thanks!

We are having difficulty using a data file with a Test suite, where multiple tests in the suite must use data from the data file.

If only 1 test in the suite uses test data, then it works.
If more than 1 test needs to use data from the test file only the last row of test data produces a passing test suite, all other rows will cause the tests pulling test data to fail.

The expectation is that each loop through the suite will use the next row of data from the data file.
What seems to be happening though is that each test in the suite that uses test data will loop through the entire test data sheet independently before progressing to the next test in the suite. This causes a failure in all but the last row of data, because the last step and the first step of the test case are not logically sequential

Example:
If we have two test cases (Test 1 and Test 2) in a test suite, each using test data with three rows of data (customer A, customer B, customer C).

Current behaviour / execution order
1A (fails)
1B (fails)
1C (pass)
2A (fails)
2B (fails)
2C (pass)

Expected behaviour / execution order
1A (pass)
2A (pass)
1B (pass)
2B (pass)
3C (pass)
3B (pass)

Being able to use test cases and test data this way within test suites allows the creation of modular test cases that can be chained together in different sequences in test suites, with each test case utilising the shared test data sheet. This in turn greatly reduces the maintenance required when there is a web site UI change while also providing the benefits of DDT.

2 Likes

yoooo how do I make it so I can run the same test case in a suite passing in different variables each time.

hello! is it possible that in a single test suite, it will generate one json file for all test cases?

You can put your data in a data file (spreadsheet) and upload it into Katalon. When you set up your test Suite, there is a button titled ‘Show Data Binding’. Here you can add the data file you are going to pull data from, and then specify which order to run the test. Once you add your test data, select ‘Map All’ and Katalon will map the data from the data files with the variables in your test case

In your test case, you will need to create your variables as such. Notice the Type and Default Value.

https://docs.katalon.com/katalon-studio/docs/run-test-case-external-data.html#bind-variables-to-test-data

Hi dave,
I’m having the same problem. Did you solve it someway?

Not really, and it seems that this is because we’re trying to use Suites in a way they’re not designed to work. Basically, our approach made some assumptions about the product that were not correct.

We were expecting the test suite to loop through the entire suite (test 1, then test 2, then test 3) using the first line from the data file, then repeat the entire suite with the next line of the data file.

The way suites appear to work in Katalon is that test 1 will loop through each line of the data file, then test 2, then test 3, etc.

This is because Katalon has been designed to work with a hermetic test approach (each test is completely independent and not reliant on any other test case), which excludes the modular approach to chaining tests together to complete a test scenario.

While hermetic tests are likely to increase maintenance if there’s a substantial change to the system under test that breaks/modifies something critical like navigation, hermatic tests do have the advantage of being able to be executed in any order, which is important in DevOps (according to experts like Gary Gruver).

To that end, a guideline for test suite creation is that the order of tests in the suite should not matter. Don’t use a suite to build a test scenario, use it merely as a grouping of tests that will use the same test data file.

It’s possible to create something similar by calling Test Cases
Create a Data Driven Test Suite