Data driven test suites does not support multiple test cases using the data file

Product Forums Katalon Studio
,
1

If only 1 test in the suite uses test data, then the all test cases in the test suite will/can pass.

If multiple tests needs to use data from the test file only the last row of test data produces passing test cases, all other rows will cause the test cases pulling test data to fail. This is because each test case in the suite that uses test data will loop through the entire test data sheet independently before progressing to the next test in the suite. This causes a failure for each test case using all but the last row of data, because the last step and the first step of the test case are not (typically) logically sequential. This makes test suites no better than test cases.

The same problem may (un-checked) occur if a test case in the middle of a test suite uses test data.

Steps to reproduce:
Create two test cases (e.g. Test 1 and Test 2) and add them to a test suite.
The suite must use the test cases to flow logically through an end-to-end process (e.g.

  • Test 1 - login
  • Test 2 - update account profile
    Each test case should depend on using test data.
    Create a test data file with three rows of data (for customer, A, B and C), to suit the full end-to-end test scenario e.g. customer name, username, password, old email, new email

Current behaviour / execution order
1A (fails)
1B (fails)
1C (pass)
2A (fails)
2B (fails)
2C (pass)

Expected behaviour / execution order
The expectation is that each loop through the suite (end-to-end) will use the next row of data from the data file.
1A (pass)
2A (pass)
1B (pass)
2B (pass)
3C (pass)
3B (pass)

Being able to use test cases and test data this way within test suites allows the creation of modular test cases that can be chained together in different sequences in test suites, with each test case utilising the shared test data sheet. This in turn greatly reduces the maintenance required when there is a web site UI change while also providing the benefits of DDT.

I humbly suggest Katalon offer options with a test suite along the lines of - Read the next row of the data file:

  • At the end of each test case (individual test looping) - current behaviour
  • After last test case in the test suite only (suite looping) - new behaviour

OS - Windows 10
Katalon Studio Version - 5.10.1
Browser versions - Latest Chrome. Latest Firefox

Katalon logs from Windows logs folder: \config.metadata.log

2

Hi @dave.furlani

Thank you for your detailed report. I would also invite @Russ_Thomas @kazurayam, @Brandon_Hein to give some thoughts.
What do you guys think ?

3

A Test Case can call other Test Cases.
https://docs.katalon.com/katalon-studio/docs/call-test-case.html

I would rather do as follows:

  1. make 3 Test Cases
  • Test 1 - login
  • Test 2 - update account profile
  • Controller - calls Test 1 and Test 2, passing customer data as parameters
  1. make a Test Suite, which is bound to the Data (Customer A, B, C). The Test Suite calls the Controller test case with param A, B or C.

When you execute, you will see the following behavior for example

Test Suite invoked Controller(A): fails
Test Suite invoked Controller(B): fails
Test Suite invoked Controller(C): pass

I don’t need this.

Controller Test Case pattern makes many problems simpler.

3 Likes
4

Thanks for the tip.
We’ll give the controller pattern a try. I’ll respond again if that fails for us or doesn’t result in successful completion of the full chain of test cases per row of data.

It sounds like my suggested test suite option " * After last test case in the test suite only (suite looping)" may be the same as having a controller test case. Providing the option in the UI may be more intuitive, basically building the controller file discreetly on behalf of the user for the test suite to call the other tests with.

That said, thanks again for the prompt response.

5

Not that @kazurayam’s solution needs it but, that gets my vote and support.

1 Like
6

While I don’t personally test things this way (I take a more hermetic approach most of the time), I’ve seen enough similar requests for this kind of feature to make me think it might be a good idea.

If it were implemented as a configurable option on the test suite level, and it was easy enough to implement without screwing with existing functionality, I’m all for it.

This is probably the strongest argument for providing this functionality.

2 Likes
7

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.