Using Test Listener to monitor status of Test Cases in a Test Suite

I have a Test Suite, which represents a Sanity Test Plan. The Test Suite is composed of several independent Test Cases. In the profile, there is a variable called “Test_is_ok”, which starts as true. As each Test Case is executed, the given Test Case may throw an exception, make Asserts or call markFailed.

In addition, a Test Listener is used to:

  1. In the AfterTestCase hook, the following groovy code checks test case status and sets the global variable accordingly:
    GlobalVariable.Test_is_ok = testCaseContext.getTestCaseStatus() == ‘PASSED’
  2. In the BeforeTestCase hook, the global variable Test_is_ok is checked, and ideally, the desired behavior is to skip all the remaining tests if an issue is detected:
    if (!GlobalVariable.Test_is_ok) {
    testCaseContext.skipThisTestCase()
    }

This looks good on paper, and yes, it does work, most of the time / some times. What I have found is that in between the AfterTestCase and the BeforeTestCase, for some unknown reason, the GlobalVariable.Test_is_ok goes from true to false. It seems to occur randomly. As such I have disabled the invocation to skipThisTestCase, and the entire Test Suite is always executed.

At this moment, I am just curious as to what is happening because I am perplexed as to how this could possibly happen.

Thanks in advance.

1 Like

Hi there,

Thank you very much for your topic. Please note that it may take a little while before a member of our community or from Katalon team responds to you.

Thanks!

I think that this code never perfoms as you expect.

The skipThisTestCase() method of TestCaseContext object — what is this method designed for?

I think that this method is designed for Katalon Studio’s GUI where you configure a Test Suite. The GUI allows you to toggle a component Test Case is to be executed or skipped. See the following screenshot:

In the GUI, when you check off 🔲 Run of the Test Cases/TC1, the GUI will call skipThisTestCase() to mark the TC1 should be skipped.

A TestCaseContext must be configured before you start a Test Suite. Once the Test Suite TS1 has started, a call to testCaseContext.skipThisTestCase() in a Test Listener will have no effect. It’s too late.

Have a look at my previous experiment where I proposed an alternative architecture to make a Test Suite to finish as soon as possible once a comprising Test Case failed:

This way worked for me.