What are the reasons you may run the Katalon "Test Listeners" VS tear downs test cases?

naomy.arnold · November 22, 2023, 12:14pm

What are the reasons you may run the Test Listeners?

Please give examples if you are using it…

Thank you

kazurayam · November 22, 2023, 12:59pm

Here is a sample TestListener implementation:

import java.time.LocalDateTime
import java.time.format.DateTimeFormatter

import com.kms.katalon.core.annotation.AfterTestCase
import com.kms.katalon.core.annotation.AfterTestSuite
import com.kms.katalon.core.annotation.BeforeTestCase
import com.kms.katalon.core.annotation.BeforeTestSuite
import com.kms.katalon.core.context.TestCaseContext
import com.kms.katalon.core.context.TestSuiteContext

class TL {
	
	File myLog = new File("myLog.txt")
	int passedTC
	int failedTC
	
	TL() {
		myLog.text = ""
		passedTC = 0
		failedTC = 0
	}
	
	@BeforeTestCase
	def beforeTestCase(TestCaseContext testCaseContext) {
		LocalDateTime now = LocalDateTime.now()
		myLog << now.format(DateTimeFormatter.ISO_LOCAL_DATE_TIME) + " " + testCaseContext.getTestCaseId() + " started" + "\n"
	}

	@AfterTestCase
	def afterTestCase(TestCaseContext testCaseContext) {
		LocalDateTime now = LocalDateTime.now()
		if (testCaseContext.getTestCaseStatus()) {
			passedTC += 1
			myLog << now.format(DateTimeFormatter.ISO_LOCAL_DATE_TIME) + " " + testCaseContext.getTestCaseId() + " passed" + "\n"
		} else {
			failedTC += 1
			myLog << now.format(DateTimeFormatter.ISO_LOCAL_DATE_TIME) + " " + testCaseContext.getTestCaseId() + " failed" + "\n"
		}
	}
	
	@BeforeTestSuite
	def beforeTestSuite(TestSuiteContext testSuiteContext) {
		LocalDateTime now = LocalDateTime.now()
		myLog << now.format(DateTimeFormatter.ISO_LOCAL_DATE_TIME) + " " + testSuiteContext.getTestSuiteId() + " started" + "\n"
	}
	
	@AfterTestSuite
	def afterTestSuite(TestSuiteContext testSuiteContext) {
		LocalDateTime now = LocalDateTime.now()
		myLog << now.format(DateTimeFormatter.ISO_LOCAL_DATE_TIME) + " " + testSuiteContext.getTestSuiteId() + " finished with passedTC=" + passedTC + ", failedTC=" + failedTC + "\n"

	}
}

When I executed a Test Suite, it created a file myLog.txt with the following content:

2023-11-22T21:55:52.074243 Test Suites/TSx started
2023-11-22T21:55:52.150577 Test Cases/blocker started
2023-11-22T21:55:57.467069 Test Cases/blocker passed
2023-11-22T21:55:59.520327 Test Cases/TC1 started
2023-11-22T21:56:01.562813 Test Cases/TC1 failed
2023-11-22T21:56:03.622608 Test Cases/blocker started
2023-11-22T21:56:08.655317 Test Cases/blocker passed
2023-11-22T21:56:10.695562 Test Cases/TC1 started
2023-11-22T21:56:12.704066 Test Cases/TC1 failed
2023-11-22T21:56:14.727859 Test Cases/blocker started
2023-11-22T21:56:19.761104 Test Cases/blocker passed
2023-11-22T21:56:21.793795 Test Suites/TSx finished with passedTC=3, failedTC=2

This looks so-so useful to me.

Russ_Thomas · November 22, 2023, 2:05pm

To gain more control over the Test Case runtime.

anon46315158 · November 22, 2023, 2:11pm

Mainly, to avoid code duplication (lazy coding)
Take a look here to get an idea how such works:

kazurayam · November 23, 2023, 2:18am

My sample TestListener implementation TL has an if statement as follows:

if (testCaseContext.getTestCaseStatus()) {
    ...
} else {
    ...
}

This is an example of “more control over the Test Case runtime”.

kazurayam · November 23, 2023, 2:22am

My sample TestListener implementation TL has a @AfterTestCase-annotated method, which is invoked just after 2 Test Cases blocker and TC1 finished.

Just imagin I have more Test Cases — 5, 10, 20 … The @AfterTestCase-annotaed method will be applied to all of those test cases. You do not need to duplicate the code.

kazurayam · November 23, 2023, 2:28am

Let me mention one more goodness of TestListener.

You can have 2 or more TestListeners to be applied to a single TestCase/TestSuite.

My sample TestListener TL creates a file myLog.txt. Just leave it as is. You can add one more TestListener TLx which creates another file failure_detail.txt. Or you can add even more TestListener TLy which send email message.

You can divide your concerns into pieces and implement them as indivisual TestListener codes. Modularization makes your code clean.

On the other hand, in the “Script” tab of a Test Suite, you can implement a set of @SetUp and @TearDown annotated methods for a Test Suite as follows:

Please note that you can have only single set of annotated methods in “Script mode” per a Test Suite. If you have 3 concerns for a single Test Suite, then you need to implemented all the 3 concerns together in a single @tearDown methods mixed. The code will be messy. I personally never use the “Script mode” of Test Suite.

anon46315158 · November 23, 2023, 9:37am

Those are in place for particular needs, e.g. if the method (whatever will be, tearUp, tearDown) has to run for a particular test suite (or each suite must have different methods).
TestListeners are more global (will run for each and every testsuite, testcase etc)
LE: i think I know at least one user using this … mhm, may be @mwarren04011990 …
In certain scenarios, this feature make sense …

mwarren04011990 · November 23, 2023, 3:59pm

I use both. I’m about to head to Thanksgiving meet up with family, so I will elaborate later…

mwarren04011990 · November 24, 2023, 6:02am

Per my previous comment, I actively use both Test Listeners and Test Suite Hooks in my project.

Test Listeners

As we already know, they have four hooks:

BeforeTestSuite
BeforeTestCase
AfterTestCase
AfterTestSuite

What they are good for?

Centralizing out any setup/teardown logic (e.g. WebDriver setup/teardown, test case error reporting, …) common to all your test cases

The problem with just Test Listeners

If you only use Test Listeners, you may be setting yourself up for multiple issues:

you may have multiple test suites that have different setup/teardown logic, especially if your App Under Test (AUT) is non-trivial
you’re going to be in a world of disappointment if you expect to control which hooks on which Test Listeners fire in which order.

Test Suite Hooks

As @kazurayam showed you via screenshot, each Test Suite has the following hooks:

SetUp : runs before Test Suite start
SetUpTestCase : runs before each Test Case
TearDownTestCase : runs after each Test Case
TearDown : runs after Test Suite finishes

Use case for the Test Suite Hooks

It separates the concerns of each Test Suite and allows for seamless execution of multiple test suites one-after-another . Each Test Suite can be made to clean up after itself in its own way, and set global variables so as to avoid stepping on the other’s toes.

The problem with just Test Suite Hooks

Obviously, we shouldn’t just run test suites. We should be able to run test cases by themselves, too.

If we only have cleanup logic in Test Suites, that leaves the test cases themselves to pollute the AUT with test data.

How do YOU use Test Listeners and Test Suite Hooks, Mike?

I use it like thus:

Test Listener

I only have one Test Listener, and this is its real code:

import static com.kms.katalon.core.testcase.TestCaseFactory.findTestCase

import org.openqa.selenium.WebDriver

import com.kms.katalon.core.annotation.AfterTestCase
import com.kms.katalon.core.annotation.BeforeTestCase
import com.kms.katalon.core.context.TestCaseContext
import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUI
import com.xxx.utils.SMDWebDriverUtils
import com.xxx.utils.TestCaseReporter
import com.xxx.utils.recordHandler.PracticeProfileHandler

class NewTestListener {
	private WebDriver driver;

	private List<String> getNoLoginList() {
		return [
			/^Test Cases\/New Zoho\/New Zoho - 00 Login$/,
			/^Test Cases\/New Member Site\/.+$/,
		];
	}
	
	private List<String> getNoBrowserOpenList() {
		return [
		    /^Test Cases\/Unit Tests\/.+$/,
		];
	}
	
	private boolean isOnTestCaseList(TestCaseContext testCaseContext, List<String> testCaseList) {
		for (String regex : testCaseList) {
			if ((testCaseContext.getTestCaseId() =~ regex).matches()) {
				return true;
			}
		}
		
		return false;
	}
	
	/*** Executes before every test case starts.
	 * @param testCaseContext related information of the executed test case.
	 */
	@BeforeTestCase
	def sampleBeforeTestCase(TestCaseContext testCaseContext) {
		if (this.isOnTestCaseList(testCaseContext, this.getNoBrowserOpenList()))
			return;
		
		SMDWebDriverUtils.SetUpDriver();

		if (this.isOnTestCaseList(testCaseContext, this.getNoLoginList()))
			return;
			
		WebUI.callTestCase(findTestCase("Test Cases/New Zoho/New Zoho - 00 Login"), null)
	}
	
	@AfterTestCase
	def sampleAfterTestCase(TestCaseContext testCaseContext) {
		if (!testCaseContext.getTestCaseStatus().equals("PASSED"))
			TestCaseReporter.GetInstance().report(testCaseContext);
		
		PracticeProfileHandler.GetInstance().close();
		
		SMDWebDriverUtils.CloseDriver();
	}
}

In the BeforeTestCase, multiple things are happening:

unit test cases run as normal. In the non-unit-test-cases, …
we set up the WebDriver in our custom way. This gets around Zoho’s hard-limit for sign-ins, and other restrictions it put on us, as well as do some quality-of-life things like maximizing the window on startup
we can test multiple AUTs with this (this was actual business requirement last year: I had to test a WordPress sign-up page as well as the main AUT, and access the latter after doing work on the former). The AUT that didn’t require log-in (the sign-up page), didn’t need to go to the log-in process

In the AfterTestCase, again, multiple things are happening:

if a test case fails/errors out, we report on it. We report the following:
- screenshot of the browser window
- the browser URL
- the entire DOM in a dom.xml file, so that we can, for example, inspect it with VSCode XML Tools extension
- the exception stack trace
we close out the handler for the practice profile records. This controls a spreadsheet that logs a practice name, a practice URL, and that practice’s parent organization name. This is backbone of our entire project.
we quit() on the WebDriver we set up. This prevents orphan processes on your computer.

Test Suite Hooks

I’ma do you a solid here and show you two the hooks for three different Test Suites:

`Discount.groovy`

import com.kms.katalon.core.annotation.SetUp
import com.kms.katalon.core.annotation.SetupTestCase
import com.kms.katalon.core.annotation.TearDown
import com.kms.katalon.core.annotation.TearDownTestCase
import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUI
import com.xxx.pages.practice.PracticePage
import com.xxx.profiles.PracticeProfile
import com.xxx.utils.SMDTestCaseUtils
import com.xxx.utils.SMDWebDriverUtils
import com.xxx.utils.recordHandler.PracticeProfileHandler

import groovy.transform.Field
import internal.GlobalVariable

/**
 * Some methods below are samples for using SetUp/TearDown in a test suite.
 */

@Field
final PracticeProfile defaultProfile = GlobalVariable.practiceProfile;

@Field
final String practiceProfileName = "WILBARGER GENERAL HOSPITAL";

/**
 * Setup test suite environment.
 */
@SetUp
def setUp() {
	GlobalVariable.practiceProfile = PracticeProfileHandler.GetInstance().getPracticeProfile(practiceProfileName);
}

/**
 * Clean test suites environment.
 */
@TearDown
def tearDown() {
	SMDWebDriverUtils.DoWebAction { 
		PracticePage.DeleteContractsIfNeeded(GlobalVariable.practiceProfile);
	}
	
	GlobalVariable.practiceProfile = defaultProfile;
	PracticeProfileHandler.GetInstance().close();
	
}
/**
 * Run before each test case starts.
 */
@SetupTestCase
def setupTestCase() {
	PracticePage initPage = new PracticePage(GlobalVariable.practiceProfile);
	initPage.go();
	
	initPage.createRateCardIfNeeded(
		{ return initPage.getNumberOfDiscounts() },
		{ SMDTestCaseUtils.CreateDummyRateCard(GlobalVariable.practiceProfile,
			null);
		},
		{ return WebUI.verifyEqual(initPage.getNumberOfContracts(), initPage.getNumberOfRateCards()); },
	);
}

/**
 * Run after each test case ends.
 */
@TearDownTestCase(skipped = true) // Please change skipped to be false to activate this method.
def tearDownTestCase() {
	// Put your code here.
}

/**
 * References:
 * Groovy tutorial page: http://docs.groovy-lang.org/next/html/documentation/
 */

We start by setting a Global Variable for the practice profile.

Then, before each test case, we create a dummy rate card if needed. This is crucial because, in our AUT, we can’t create discount without first having a rate card to assign it to.

After test suite finishes, we start the WebDriver back up, delete any dummy contracts (the parent-most model) we created during test suite run time, close it, set the practiceProfile Global Variable back, and close the practice profile handler. (The latter is probably overkill…)

`MemberLead.groovy`

This one is much simpler:

import com.kms.katalon.core.annotation.SetUp
import com.kms.katalon.core.annotation.SetupTestCase
import com.kms.katalon.core.annotation.TearDown
import com.kms.katalon.core.annotation.TearDownTestCase
import com.xxx.pages.list.BaseListPage
import com.xxx.pages.list.MemberLeadListPage
import com.xxx.pages.list.MemberListPage
import com.xxx.profiles.PracticeProfile
import com.xxx.utils.SMDDateUtils
import com.xxx.utils.SMDWebDriverUtils
import com.xxx.utils.recordHandler.PracticeProfileHandler

import groovy.transform.Field
import internal.GlobalVariable



/**
 * Some methods below are samples for using SetUp/TearDown in a test suite.
 */

@Field
final PracticeProfile defaultProfile = GlobalVariable.practiceProfile;

@Field
final String practiceProfileName = "WILBARGER GENERAL HOSPITAL";

/**
 * Setup test suite environment.
 */
@SetUp
def setUp() {
	GlobalVariable.practiceProfile = PracticeProfileHandler.GetInstance().getPracticeProfile(practiceProfileName);
	
	SMDDateUtils.GetStartTime();
}

/**
 * Clean test suites environment.
 */
@TearDown
def tearDown() {
	SMDWebDriverUtils.DoWebAction { 
		[
			new MemberListPage(),
			new MemberLeadListPage(),
		].each { BaseListPage page -> 
			page.go();
			
			page.applyCreationFilters();
			page.submitFilters();
			
			page.waitForTableLoad();
			
			page.deleteAllCreatedAfter(SMDDateUtils.GetStartTime());
		}
	}
	
	GlobalVariable.practiceProfile = defaultProfile;
}

/**
 * Run before each test case starts.
 */
@SetupTestCase(skipped = true) // Please change skipped to be false to activate this method.
def setupTestCase() {
	// Put your code here.
}

/**
 * Run after each test case ends.
 */
@TearDownTestCase(skipped = true) // Please change skipped to be false to activate this method.
def tearDownTestCase() {
	// Put your code here.
}

/**
 * References:
 * Groovy tutorial page: http://docs.groovy-lang.org/next/html/documentation/
 */

All we’re doing here, is the following:

before the test suite starts, we capture the timestamp and set the practiceProfile to our test one
after the test suite ends, we go to the member and member lead list pages, and delete all the members and member leads we created (i.e. the ones matching the creation filters, created by us, today) after that timestamp

You’re probably thinking: why would I ever want to tear down after each test case, and then after the test suite itself…

Hopefully this next test suite helps answer that:

`Practice Contract.groovy`

import com.kms.katalon.core.annotation.SetUp
import com.kms.katalon.core.annotation.SetupTestCase
import com.kms.katalon.core.annotation.TearDown
import com.kms.katalon.core.annotation.TearDownTestCase
import com.xxx.pages.practice.PracticePage
import com.xxx.profiles.PracticeProfile
import com.xxx.utils.SMDWebDriverUtils
import com.xxx.utils.recordHandler.PracticeProfileHandler

import groovy.transform.Field
import internal.GlobalVariable

/**
 * Some methods below are samples for using SetUp/TearDown in a test suite.
 */

@Field
final PracticeProfile defaultProfile = GlobalVariable.practiceProfile;

@Field
final String practiceProfileName = "WILBARGER GENERAL HOSPITAL";

@Field
final String contractFieldName = "contracts";

/**
 * Setup test suite environment.
 */
@SetUp
def setUp() {
	GlobalVariable.practiceProfile = PracticeProfileHandler.GetInstance().getPracticeProfile(practiceProfileName);
	SMDWebDriverUtils.DoWebAction { 
		PracticePage page = new PracticePage(GlobalVariable.practiceProfile);
		page.go();
		PracticePage.InitialRecordCount[contractFieldName] = page.getNumberOfContracts();
	}
}

/**
 * Clean test suites environment.
 */
@TearDown
def tearDown() {
	SMDWebDriverUtils.DoWebAction { 
		PracticePage.DeleteContractsIfNeeded(GlobalVariable.practiceProfile);
	}
	
	GlobalVariable.practiceProfile = defaultProfile;
	PracticeProfileHandler.GetInstance().close();
	
}

/**
 * Run before each test case starts.
 */
@SetupTestCase(skipped = true) // Please change skipped to be false to activate this method.
def setupTestCase() {
	// Put your code here.
}

/**
 * Run after each test case ends.
 */
@TearDownTestCase
def tearDownTestCase() {
	PracticePage page = new PracticePage(GlobalVariable.practiceProfile);
	page.go();
	PracticePage.NewContractsCreated = page.getNumberOfContracts() - PracticePage.InitialRecordCount[contractFieldName];
}

/**
 * References:
 * Groovy tutorial page: http://docs.groovy-lang.org/next/html/documentation/
 */

Here’ we have something special going on:

right before the test suite starts, we set the practiceProfile global variable and get the number of contracts already in the system
after each test case, we get the number of new contracts created
after test suite ends, we delete all those new contracts, set the practiceProfile back, and close the practice profile handler

Sorry for the long-winded post, but I hope it helps!

vinod · June 15, 2024, 11:55am

@TearDownTestCase(skipped = false)
def afterTestCase(TestCaseContext testCaseContext) {
String testCaseName
testCaseName = testCaseContext.getTestCaseId()
println(‘testCaseName===’ + testCaseName)

String ignoredTestCase = 'Login'

if (testCaseName != ignoredTestCase) {
    CustomKeywords.'qatest.Navigation.navigateTo'()
} else {
    println("Ignoring tear down for test case: $testCaseName")
}

}
I am using ‘Test Suite Hooks’ and I am getting an error like below
Cannot invoke method getTestCaseId()

Please help?

system · June 15, 2024, 11:55am

Hi there,

Thank you very much for your topic. Please note that it may take a little while before a member of our community or from Katalon team responds to you.

Thanks!

Topic		Replies	Views
How to run test suite of features via testopt? Katalon TestOps bugs-report	1	418	August 18, 2023
How do I Use Test Case Tags in Test Listener? Katalon Studio katalon-studio , test-listener	2	213	May 2, 2024
How to use the Test Listeners from the plugins Katalon Studio plugins , katalon-studio	12	2019	April 17, 2020
Has Anybody used following methods? Katalon Studio katalon-studio	4	563	November 26, 2019
How to handle Test Listeners based on the Test Case or Test Suite execution Katalon Studio katalon-studio , web-testing	14	5527	February 6, 2023

What are the reasons you may run the Katalon "Test Listeners" VS tear downs test cases?

Test Listeners

What they are good for?

The problem with just Test Listeners

Test Suite Hooks

Use case for the Test Suite Hooks

The problem with just Test Suite Hooks

How do YOU use Test Listeners and Test Suite Hooks, Mike?

Test Listener

Test Suite Hooks

Discount.groovy

MemberLead.groovy

Practice Contract.groovy

Related Topics

`Discount.groovy`

`MemberLead.groovy`

`Practice Contract.groovy`