Test Listeners do not function in top down order

  • Katalon Studio Version: 7.9
  • OS Version: Windows 10
  • Browser Version: Chrome 87.0.4280.141

Steps to Reproduce:

  1. Create a few listeners and order them from top down and not in alphabetical order.
  2. Run a test case.
  3. Notice the listeners do not occur in top down order but randomly.

Expected behavior:
Listeners should occur from top down.

Screenshots:

Can you explain/show where your listeners are marked @Before/@AfterTestCase?

Mine are called Listener1_blah and Listener2_blah and they always execute in alphanumeric order (my tests/suites would all fail if they didn’t).

Basically all the ones in the screenshot have @AfterTestCase marked before the method. Should these be marked differently? I haven’t really deviated from the standard sample marking.

@Chase_Emerick

Can you provide an example project to reliably reproduce this issue ? Did the previous version work ?

This was also an issue in the previous version. This screen shot is just a bare bones project with a test case and listeners.

Which file in particular would you need?

Curious if there has been any traction on this issue?

Well over a year since a last response is there any update on this?

@Jass    

@Chase_Emerick You could use ONE @AfterTestCase and call methods instead.

@Chase_Emerick

You wrote your Test Listener like this:

import com.kms.katalon.core.annotation.AfterTestCase
import com.kms.katalon.core.context.TestCaseContext


class TL1 {
	
	@AfterTestCase
	def deletingTheDownloadDirectory() {
		println "delete the download directory"
	}
	
	@AfterTestCase
	def captureConsoleLogs() {
		println "capture console logs"
	}
	
	@AfterTestCase
	def captureCookieLogs() {
		println "capture cookie logs"
	}
	
	@AfterTestCase
	def closeBrowser() {
		println "close browser"
	}	
}

When I execute a Test case, I saw messages in the console:

delete the download directory
capture console logs
capture cookie logs
close browser

It looks that the @AfterTestCase-annotaded 4 methods are invoked in the order they are coded in the source.

However, I think that it is a bad idea to assume that 4 methods will be / should be in the order they are coded in the source.

I think we should expect NO order promised by Katalon Studio.

Moreover, I think that it is a bad idea to make 2 or more methods annotated with @AfterTestCase.

You only need one method annotated with @AfterTestCase.

See the following Test Listener.

import com.kms.katalon.core.annotation.AfterTestCase
import com.kms.katalon.core.context.TestCaseContext


class TL2 {
	
	@AfterTestCase
	def afterTestCase(TestCaseContext testCaseContext) {
		captureConsoleLogs()
		captureCookieLogs()
		closeBrowser()
		deletingTheDownloadDirectory()
	}
	
	def deletingTheDownloadDirectory() {
		println "delete the download directory"
	}
	
	def captureConsoleLogs() {
		println "capture console logs"
	}
	
	def captureCookieLogs() {
		println "capture cookie logs"
	}
	
	def closeBrowser() {
		println "close browser"
	}
	
	
}

In the TL2, there is only a single method annotated with @AfterTestCase, and it drives other 4 methods in the order as it want to.

Thanks @kazurayam and @Russ_Thomas I appreciate the help! God Bless!