One Test Case or Many? Seeking Guidance

Archive
,
1

Hi,

I am seeking guidance on how Katalon is intended to be used, as it’s not clear how to structure the tests to get the requires result in the reports.

I’ll start with some background for context:

I have an application (AUT) that users log into and navigate through a series of tiles / menus to start a business process (BP), the business process contains several screens (typically 8-15, some are common across multiple business processes)

Based on this context I expect this to be similar to many other applications that others are testing.

Many Test Case Model:
I create separate test cases for login, logout, menu navigation, and individual BP screens I then sequence these test cases in a test suite and execute the test suite, I get a report showing 45 test total, 43 passed, 2 failed etc. So far all good.

Now in my AUT, there are a variety of roles, so BP’s can be run by all roles, some BP’s by selected roles, for this simple example we are using a BP that can be run by all roles, so I set up a data sheet (csv) with a list of logins and roles, I create a new Test Data item in the Data Files section, then using the data binding in the test suite I configure my login test case to use the login names and passwords and execute the test suite again expecting the test cases to be executed sequentially for each login but what actually happens is the login test case is executed once for each login and then the other test cases are executed once using the last login…

… ah, so maybe I need to use one big test case?

One test case model:
I create a new “Parent” test case and call all the smaller individual test cases from the many test case model, this is easy to set up with the call test case test step.

I then create a test suite and add this test case to it, I can easily link the test data I already created and pass the login details, I can see the whole BP will be executed for each login

I configure to run with the first row to start with, I execute the test case and look at the report:
1 test total, 0 passed, 1 failed etc (expected 45 test total, 43 passed, 2 failed etc)

so how many of the test cases that are now test steps failed? I expand the test case but there is no summary of passed/failed test steps for the parent test case

**Summary:
**

  • The One big test case model doesn’t provide the detail in the report that the test team requires.
  • The many test case model doesn’t allow the whole BP to be executed for each role.

What is the recommended way to use Katalon test cases / test suites?

2

I prefer the “One Test Case Model” that calls other Test Cases because the “data driven” execution does what I want at the Business Process Level…

3

Hi David, thanks for your reply, with the one test case model how do you handle the single pass / fail results, do you have some trick to get the additional summary information on the number of test steps that failed or don’t your testers care? just a pass / fail per test case with no additional data is not going to cut it here unfortunately.

4

Dave,

Have you tried using the Katalon Analytics capability?

This might be able to provide the Test Suite summary information for Test Cases you are looking for…

5

Unfortunately Katalon Analytics is not an option for us - it’s a cloud solution therefore we can’t use it.

6

I also miss the option to do a DataDriven run on TestSuite level:
I would suggest to upvote this FEATURE on:

7

Thanks Gerard, I did an up-vote for Peter’s issue, Unfortunately he seems to be happy with the one test case model as is, but that breaks reporting for me.

I have implemented the many test case solution for now, but can only run one data set per test suite, i’ll just need to create many test suites that are the same except the data being used, not ideal but workable. Hopefully this will be improved in a future version.

8

dave said:

Thanks Gerard, I did an up-vote for Peter’s issue, Unfortunately he seems to be happy with the one test case model as is, but that breaks reporting for me.

I have implemented the many test case solution for now, but can only run one data set per test suite, i’ll just need to create many test suites that are the same except the data being used, not ideal but workable. Hopefully this will be improved in a future version.

Hi,

How about to use “Profiles” ? You can create a profile for each kind of “user” and each TestSuite can have one or several Profile !

see : https://docs.katalon.com/pages/viewpage.action?pageId=13697476

(new feature on 5.4.2)

9

Hey Dave, calling all the Test Cases does work, but it’s not very easy to maintain when you have to add new tests or want to selectively run some, but not others. As you note, the reporting won’t be great. My tests output a lot of information to the Warning Log to compensate.

I hope for a better implementation in the future, but it does get the job done (in certain circumstances) for the time being,.