What are the reasons you may run the Katalon "Test Listeners" VS tear downs test cases?

What are the reasons you may run the Test Listeners?

image

image

Please give examples if you are using it…

Thank you

2 Likes

Here is a sample TestListener implementation:

import java.time.LocalDateTime
import java.time.format.DateTimeFormatter

import com.kms.katalon.core.annotation.AfterTestCase
import com.kms.katalon.core.annotation.AfterTestSuite
import com.kms.katalon.core.annotation.BeforeTestCase
import com.kms.katalon.core.annotation.BeforeTestSuite
import com.kms.katalon.core.context.TestCaseContext
import com.kms.katalon.core.context.TestSuiteContext

class TL {
	
	File myLog = new File("myLog.txt")
	int passedTC
	int failedTC
	
	TL() {
		myLog.text = ""
		passedTC = 0
		failedTC = 0
	}
	
	@BeforeTestCase
	def beforeTestCase(TestCaseContext testCaseContext) {
		LocalDateTime now = LocalDateTime.now()
		myLog << now.format(DateTimeFormatter.ISO_LOCAL_DATE_TIME) + " " + testCaseContext.getTestCaseId() + " started" + "\n"
	}

	@AfterTestCase
	def afterTestCase(TestCaseContext testCaseContext) {
		LocalDateTime now = LocalDateTime.now()
		if (testCaseContext.getTestCaseStatus()) {
			passedTC += 1
			myLog << now.format(DateTimeFormatter.ISO_LOCAL_DATE_TIME) + " " + testCaseContext.getTestCaseId() + " passed" + "\n"
		} else {
			failedTC += 1
			myLog << now.format(DateTimeFormatter.ISO_LOCAL_DATE_TIME) + " " + testCaseContext.getTestCaseId() + " failed" + "\n"
		}
	}
	
	@BeforeTestSuite
	def beforeTestSuite(TestSuiteContext testSuiteContext) {
		LocalDateTime now = LocalDateTime.now()
		myLog << now.format(DateTimeFormatter.ISO_LOCAL_DATE_TIME) + " " + testSuiteContext.getTestSuiteId() + " started" + "\n"
	}
	
	@AfterTestSuite
	def afterTestSuite(TestSuiteContext testSuiteContext) {
		LocalDateTime now = LocalDateTime.now()
		myLog << now.format(DateTimeFormatter.ISO_LOCAL_DATE_TIME) + " " + testSuiteContext.getTestSuiteId() + " finished with passedTC=" + passedTC + ", failedTC=" + failedTC + "\n"

	}
}

When I executed a Test Suite, it created a file myLog.txt with the following content:

2023-11-22T21:55:52.074243 Test Suites/TSx started
2023-11-22T21:55:52.150577 Test Cases/blocker started
2023-11-22T21:55:57.467069 Test Cases/blocker passed
2023-11-22T21:55:59.520327 Test Cases/TC1 started
2023-11-22T21:56:01.562813 Test Cases/TC1 failed
2023-11-22T21:56:03.622608 Test Cases/blocker started
2023-11-22T21:56:08.655317 Test Cases/blocker passed
2023-11-22T21:56:10.695562 Test Cases/TC1 started
2023-11-22T21:56:12.704066 Test Cases/TC1 failed
2023-11-22T21:56:14.727859 Test Cases/blocker started
2023-11-22T21:56:19.761104 Test Cases/blocker passed
2023-11-22T21:56:21.793795 Test Suites/TSx finished with passedTC=3, failedTC=2

This looks so-so useful to me.

To gain more control over the Test Case runtime.

Mainly, to avoid code duplication (lazy coding)
Take a look here to get an idea how such works:

My sample TestListener implementation TL has an if statement as follows:

if (testCaseContext.getTestCaseStatus()) {
    ...
} else {
    ...
}

This is an example of “more control over the Test Case runtime”.

My sample TestListener implementation TL has a @AfterTestCase-annotated method, which is invoked just after 2 Test Cases blocker and TC1 finished.

Just imagin I have more Test Cases — 5, 10, 20 … The @AfterTestCase-annotaed method will be applied to all of those test cases. You do not need to duplicate the code.

Let me mention one more goodness of TestListener.

You can have 2 or more TestListeners to be applied to a single TestCase/TestSuite.

My sample TestListener TL creates a file myLog.txt. Just leave it as is. You can add one more TestListener TLx which creates another file failure_detail.txt. Or you can add even more TestListener TLy which send email message.

You can divide your concerns into pieces and implement them as indivisual TestListener codes. Modularization makes your code clean.


On the other hand, in the “Script” tab of a Test Suite, you can implement a set of @SetUp and @TearDown annotated methods for a Test Suite as follows:

Please note that you can have only single set of annotated methods in “Script mode” per a Test Suite. If you have 3 concerns for a single Test Suite, then you need to implemented all the 3 concerns together in a single @tearDown methods mixed. The code will be messy. I personally never use the “Script mode” of Test Suite.

Those are in place for particular needs, e.g. if the method (whatever will be, tearUp, tearDown) has to run for a particular test suite (or each suite must have different methods).
TestListeners are more global (will run for each and every testsuite, testcase etc)
LE: i think I know at least one user using this … mhm, may be @mwarren04011990
In certain scenarios, this feature make sense …

I use both. I’m about to head to Thanksgiving meet up with family, so I will elaborate later…

Per my previous comment, I actively use both Test Listeners and Test Suite Hooks in my project.

Test Listeners

As we already know, they have four hooks:

  • BeforeTestSuite
  • BeforeTestCase
  • AfterTestCase
  • AfterTestSuite

What they are good for?

Centralizing out any setup/teardown logic (e.g. WebDriver setup/teardown, test case error reporting, …) common to all your test cases

The problem with just Test Listeners

If you only use Test Listeners, you may be setting yourself up for multiple issues:

  • you may have multiple test suites that have different setup/teardown logic, especially if your App Under Test (AUT) is non-trivial
  • you’re going to be in a world of disappointment if you expect to control which hooks on which Test Listeners fire in which order.

Test Suite Hooks

As @kazurayam showed you via screenshot, each Test Suite has the following hooks:

  • SetUp : runs before Test Suite start
  • SetUpTestCase : runs before each Test Case
  • TearDownTestCase : runs after each Test Case
  • TearDown : runs after Test Suite finishes

Use case for the Test Suite Hooks

It separates the concerns of each Test Suite and allows for seamless execution of multiple test suites one-after-another . Each Test Suite can be made to clean up after itself in its own way, and set global variables so as to avoid stepping on the other’s toes.

The problem with just Test Suite Hooks

Obviously, we shouldn’t just run test suites. We should be able to run test cases by themselves, too.

If we only have cleanup logic in Test Suites, that leaves the test cases themselves to pollute the AUT with test data.

How do YOU use Test Listeners and Test Suite Hooks, Mike?

I use it like thus:

Test Listener

I only have one Test Listener, and this is its real code:

import static com.kms.katalon.core.testcase.TestCaseFactory.findTestCase

import org.openqa.selenium.WebDriver

import com.kms.katalon.core.annotation.AfterTestCase
import com.kms.katalon.core.annotation.BeforeTestCase
import com.kms.katalon.core.context.TestCaseContext
import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUI
import com.xxx.utils.SMDWebDriverUtils
import com.xxx.utils.TestCaseReporter
import com.xxx.utils.recordHandler.PracticeProfileHandler

class NewTestListener {
	private WebDriver driver;

	private List<String> getNoLoginList() {
		return [
			/^Test Cases\/New Zoho\/New Zoho - 00 Login$/,
			/^Test Cases\/New Member Site\/.+$/,
		];
	}
	
	private List<String> getNoBrowserOpenList() {
		return [
		    /^Test Cases\/Unit Tests\/.+$/,
		];
	}
	
	private boolean isOnTestCaseList(TestCaseContext testCaseContext, List<String> testCaseList) {
		for (String regex : testCaseList) {
			if ((testCaseContext.getTestCaseId() =~ regex).matches()) {
				return true;
			}
		}
		
		return false;
	}
	
	/*** Executes before every test case starts.
	 * @param testCaseContext related information of the executed test case.
	 */
	@BeforeTestCase
	def sampleBeforeTestCase(TestCaseContext testCaseContext) {
		if (this.isOnTestCaseList(testCaseContext, this.getNoBrowserOpenList()))
			return;
		
		SMDWebDriverUtils.SetUpDriver();

		if (this.isOnTestCaseList(testCaseContext, this.getNoLoginList()))
			return;
			
		WebUI.callTestCase(findTestCase("Test Cases/New Zoho/New Zoho - 00 Login"), null)
	}
	
	@AfterTestCase
	def sampleAfterTestCase(TestCaseContext testCaseContext) {
		if (!testCaseContext.getTestCaseStatus().equals("PASSED"))
			TestCaseReporter.GetInstance().report(testCaseContext);
		
		PracticeProfileHandler.GetInstance().close();
		
		SMDWebDriverUtils.CloseDriver();
	}
}

In the BeforeTestCase, multiple things are happening:

  • unit test cases run as normal. In the non-unit-test-cases, …
  • we set up the WebDriver in our custom way. This gets around Zoho’s hard-limit for sign-ins, and other restrictions it put on us, as well as do some quality-of-life things like maximizing the window on startup
  • we can test multiple AUTs with this (this was actual business requirement last year: I had to test a WordPress sign-up page as well as the main AUT, and access the latter after doing work on the former). The AUT that didn’t require log-in (the sign-up page), didn’t need to go to the log-in process

In the AfterTestCase, again, multiple things are happening:

  • if a test case fails/errors out, we report on it. We report the following:
    • screenshot of the browser window
    • the browser URL
    • the entire DOM in a dom.xml file, so that we can, for example, inspect it with VSCode XML Tools extension
    • the exception stack trace
  • we close out the handler for the practice profile records. This controls a spreadsheet that logs a practice name, a practice URL, and that practice’s parent organization name. This is backbone of our entire project.
  • we quit() on the WebDriver we set up. This prevents orphan processes on your computer.

Test Suite Hooks

I’ma do you a solid here and show you two the hooks for three different Test Suites:

Discount.groovy

import com.kms.katalon.core.annotation.SetUp
import com.kms.katalon.core.annotation.SetupTestCase
import com.kms.katalon.core.annotation.TearDown
import com.kms.katalon.core.annotation.TearDownTestCase
import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUI
import com.xxx.pages.practice.PracticePage
import com.xxx.profiles.PracticeProfile
import com.xxx.utils.SMDTestCaseUtils
import com.xxx.utils.SMDWebDriverUtils
import com.xxx.utils.recordHandler.PracticeProfileHandler

import groovy.transform.Field
import internal.GlobalVariable

/**
 * Some methods below are samples for using SetUp/TearDown in a test suite.
 */

@Field
final PracticeProfile defaultProfile = GlobalVariable.practiceProfile;

@Field
final String practiceProfileName = "WILBARGER GENERAL HOSPITAL";

/**
 * Setup test suite environment.
 */
@SetUp
def setUp() {
	GlobalVariable.practiceProfile = PracticeProfileHandler.GetInstance().getPracticeProfile(practiceProfileName);
}

/**
 * Clean test suites environment.
 */
@TearDown
def tearDown() {
	SMDWebDriverUtils.DoWebAction { 
		PracticePage.DeleteContractsIfNeeded(GlobalVariable.practiceProfile);
	}
	
	GlobalVariable.practiceProfile = defaultProfile;
	PracticeProfileHandler.GetInstance().close();
	
}
/**
 * Run before each test case starts.
 */
@SetupTestCase
def setupTestCase() {
	PracticePage initPage = new PracticePage(GlobalVariable.practiceProfile);
	initPage.go();
	
	initPage.createRateCardIfNeeded(
		{ return initPage.getNumberOfDiscounts() },
		{ SMDTestCaseUtils.CreateDummyRateCard(GlobalVariable.practiceProfile,
			null);
		},
		{ return WebUI.verifyEqual(initPage.getNumberOfContracts(), initPage.getNumberOfRateCards()); },
	);
}

/**
 * Run after each test case ends.
 */
@TearDownTestCase(skipped = true) // Please change skipped to be false to activate this method.
def tearDownTestCase() {
	// Put your code here.
}

/**
 * References:
 * Groovy tutorial page: http://docs.groovy-lang.org/next/html/documentation/
 */

We start by setting a Global Variable for the practice profile.

Then, before each test case, we create a dummy rate card if needed. This is crucial because, in our AUT, we can’t create discount without first having a rate card to assign it to.

After test suite finishes, we start the WebDriver back up, delete any dummy contracts (the parent-most model) we created during test suite run time, close it, set the practiceProfile Global Variable back, and close the practice profile handler. (The latter is probably overkill…)

MemberLead.groovy

This one is much simpler:

import com.kms.katalon.core.annotation.SetUp
import com.kms.katalon.core.annotation.SetupTestCase
import com.kms.katalon.core.annotation.TearDown
import com.kms.katalon.core.annotation.TearDownTestCase
import com.xxx.pages.list.BaseListPage
import com.xxx.pages.list.MemberLeadListPage
import com.xxx.pages.list.MemberListPage
import com.xxx.profiles.PracticeProfile
import com.xxx.utils.SMDDateUtils
import com.xxx.utils.SMDWebDriverUtils
import com.xxx.utils.recordHandler.PracticeProfileHandler

import groovy.transform.Field
import internal.GlobalVariable



/**
 * Some methods below are samples for using SetUp/TearDown in a test suite.
 */

@Field
final PracticeProfile defaultProfile = GlobalVariable.practiceProfile;

@Field
final String practiceProfileName = "WILBARGER GENERAL HOSPITAL";

/**
 * Setup test suite environment.
 */
@SetUp
def setUp() {
	GlobalVariable.practiceProfile = PracticeProfileHandler.GetInstance().getPracticeProfile(practiceProfileName);
	
	SMDDateUtils.GetStartTime();
}

/**
 * Clean test suites environment.
 */
@TearDown
def tearDown() {
	SMDWebDriverUtils.DoWebAction { 
		[
			new MemberListPage(),
			new MemberLeadListPage(),
		].each { BaseListPage page -> 
			page.go();
			
			page.applyCreationFilters();
			page.submitFilters();
			
			page.waitForTableLoad();
			
			page.deleteAllCreatedAfter(SMDDateUtils.GetStartTime());
		}
	}
	
	GlobalVariable.practiceProfile = defaultProfile;
}

/**
 * Run before each test case starts.
 */
@SetupTestCase(skipped = true) // Please change skipped to be false to activate this method.
def setupTestCase() {
	// Put your code here.
}

/**
 * Run after each test case ends.
 */
@TearDownTestCase(skipped = true) // Please change skipped to be false to activate this method.
def tearDownTestCase() {
	// Put your code here.
}

/**
 * References:
 * Groovy tutorial page: http://docs.groovy-lang.org/next/html/documentation/
 */

All we’re doing here, is the following:

  • before the test suite starts, we capture the timestamp and set the practiceProfile to our test one
  • after the test suite ends, we go to the member and member lead list pages, and delete all the members and member leads we created (i.e. the ones matching the creation filters, created by us, today) after that timestamp

You’re probably thinking: why would I ever want to tear down after each test case, and then after the test suite itself…

Hopefully this next test suite helps answer that:

Practice Contract.groovy

import com.kms.katalon.core.annotation.SetUp
import com.kms.katalon.core.annotation.SetupTestCase
import com.kms.katalon.core.annotation.TearDown
import com.kms.katalon.core.annotation.TearDownTestCase
import com.xxx.pages.practice.PracticePage
import com.xxx.profiles.PracticeProfile
import com.xxx.utils.SMDWebDriverUtils
import com.xxx.utils.recordHandler.PracticeProfileHandler

import groovy.transform.Field
import internal.GlobalVariable

/**
 * Some methods below are samples for using SetUp/TearDown in a test suite.
 */

@Field
final PracticeProfile defaultProfile = GlobalVariable.practiceProfile;

@Field
final String practiceProfileName = "WILBARGER GENERAL HOSPITAL";

@Field
final String contractFieldName = "contracts";

/**
 * Setup test suite environment.
 */
@SetUp
def setUp() {
	GlobalVariable.practiceProfile = PracticeProfileHandler.GetInstance().getPracticeProfile(practiceProfileName);
	SMDWebDriverUtils.DoWebAction { 
		PracticePage page = new PracticePage(GlobalVariable.practiceProfile);
		page.go();
		PracticePage.InitialRecordCount[contractFieldName] = page.getNumberOfContracts();
	}
}

/**
 * Clean test suites environment.
 */
@TearDown
def tearDown() {
	SMDWebDriverUtils.DoWebAction { 
		PracticePage.DeleteContractsIfNeeded(GlobalVariable.practiceProfile);
	}
	
	GlobalVariable.practiceProfile = defaultProfile;
	PracticeProfileHandler.GetInstance().close();
	
}

/**
 * Run before each test case starts.
 */
@SetupTestCase(skipped = true) // Please change skipped to be false to activate this method.
def setupTestCase() {
	// Put your code here.
}

/**
 * Run after each test case ends.
 */
@TearDownTestCase
def tearDownTestCase() {
	PracticePage page = new PracticePage(GlobalVariable.practiceProfile);
	page.go();
	PracticePage.NewContractsCreated = page.getNumberOfContracts() - PracticePage.InitialRecordCount[contractFieldName];
}

/**
 * References:
 * Groovy tutorial page: http://docs.groovy-lang.org/next/html/documentation/
 */

Here’ we have something special going on:

  • right before the test suite starts, we set the practiceProfile global variable and get the number of contracts already in the system
  • after each test case, we get the number of new contracts created
  • after test suite ends, we delete all those new contracts, set the practiceProfile back, and close the practice profile handler

Sorry for the long-winded post, but I hope it helps!

2 Likes