Test Suite Collection isn't running all Test Suites for API Tests

[D_Adams]

I have two test suites in my collection. Test Suite A has 3 test cases, and Test Suite B has 2 test cases. When I run either of them on their own, the tests run successfully. When put into a test suite collection and executed, my test progress meter shows 5/5 tests executed, but my console only shows the 3 test cases from Test Suite A, with just the 3 passed tests, not 5 as I was expecting.

My first 3 test cases from Test Suite A send a POST and return an access token - I have my own Custom Keyword to hold onto this value and pass it into my remaining 2 test cases in Test Suite B.

Again…when run separately, both Test Suite A and B pass without issue. Has anyone out there had a similar issue, and is there a solution? I’m still pretty new with respect to API testing in Katalon, so any help would be appreciated.

Here are the details for my Test Suite A (3 test cases)

Here are the details for my Test Suite B (2 test cases)

Here are my test suite collection results. Note 5 of 5 tests were run (from the upper right). Log viewer is only showing 3 of 3 tests run with no failures.

Lastly, here’s my console log output (it’s consistent with the log viewer details, but I’m including it, just in case):
SLF4J: The requested version 1.7.16 by your slf4j binding is not compatible with [1.6]
SLF4J: See SLF4J Error Codes for further details.
2019-03-16 14:57:11.371 e[34mINFO e[0;39m e[36mc.k.katalon.core.main.TestSuiteExecutor -e[0;39m e[39mSTART Test Suites/TS_AuthenticateTestse[0;39m
2019-03-16 14:57:11.481 e[34mINFO e[0;39m e[36mc.k.katalon.core.main.TestSuiteExecutor -e[0;39m e[39mhostName = myhostnamee[0;39m
2019-03-16 14:57:11.481 e[34mINFO e[0;39m e[36mc.k.katalon.core.main.TestSuiteExecutor -e[0;39m e[39mos = Windows 10 64bite[0;39m
2019-03-16 14:57:11.481 e[34mINFO e[0;39m e[36mc.k.katalon.core.main.TestSuiteExecutor -e[0;39m e[39mhostAddress = myhostaddresse[0;39m
2019-03-16 14:57:11.481 e[34mINFO e[0;39m e[36mc.k.katalon.core.main.TestSuiteExecutor -e[0;39m e[39mkatalonVersion = 6.0.5.4e[0;39m
2019-03-16 14:57:12.824 e[34mINFO e[0;39m e[36mc.k.katalon.core.main.TestCaseExecutor -e[0;39m e[39m--------------------e[0;39m
2019-03-16 14:57:12.824 e[34mINFO e[0;39m e[36mc.k.katalon.core.main.TestCaseExecutor -e[0;39m e[39mSTART Test Cases/Auth/AuthenticateUser_FailOnBadUsernamee[0;39m
2019-03-16 14:57:15.440 e[39mDEBUGe[0;39m e[36mt.AuthenticateUser_FailOnBadUsername -e[0;39m e[39m1: request = findTestObject(“Auth/Authenticate User”)e[0;39m
2019-03-16 14:57:16.114 e[39mDEBUGe[0;39m e[36mt.AuthenticateUser_FailOnBadUsername -e[0;39m e[39m2: requestBody = "{“username”: “badusername”, “password”: “goodpassword”}"e[0;39m
2019-03-16 14:57:16.114 e[39mDEBUGe[0;39m e[36mt.AuthenticateUser_FailOnBadUsername -e[0;39m e[39m3: trye[0;39m
2019-03-16 14:57:16.130 e[39mDEBUGe[0;39m e[36mt.AuthenticateUser_FailOnBadUsername -e[0;39m e[39m1: request.setBodyContent(new com.kms.katalon.core.testobject.impl.HttpTextBodyContent(requestBody, UTF-8, application/json))e[0;39m
2019-03-16 14:57:16.130 e[39mDEBUGe[0;39m e[36mt.AuthenticateUser_FailOnBadUsername -e[0;39m e[39m6: response = sendRequest(request)e[0;39m
2019-03-16 14:57:25.102 e[34mINFO e[0;39m e[36mc.k.k.core.util.BrowserMobProxyManager -e[0;39m e[39mHAR: C:\Source\WebAPI Testing\Web API Testing\Reports\TS_AuthenticateTests\20190316_145706\requests\main\0.hare[0;39m
2019-03-16 14:57:25.634 e[39mDEBUGe[0;39m e[36mt.AuthenticateUser_FailOnBadUsername -e[0;39m e[39m7: getStatusCode()).isIn(Arrays.asList(200, 201, 202))e[0;39m
2019-03-16 14:57:26.076 e[39mDEBUGe[0;39m e[36mt.AuthenticateUser_FailOnBadUsername -e[0;39m e[39m8: verifyElementPropertyValue(response, “systemUserId”, “0”)e[0;39m
2019-03-16 14:57:27.399 e[39mDEBUGe[0;39m e[36mt.AuthenticateUser_FailOnBadUsername -e[0;39m e[39m9: getResponseText()).contains(“Invalid login ID or password”)e[0;39m
2019-03-16 14:57:27.717 e[34mINFO e[0;39m e[36mc.k.katalon.core.main.TestCaseExecutor -e[0;39m e[39mEND Test Cases/Auth/AuthenticateUser_FailOnBadUsernamee[0;39m
2019-03-16 14:57:27.833 e[34mINFO e[0;39m e[36mc.k.katalon.core.main.TestCaseExecutor -e[0;39m e[39m--------------------e[0;39m
2019-03-16 14:57:27.834 e[34mINFO e[0;39m e[36mc.k.katalon.core.main.TestCaseExecutor -e[0;39m e[39mSTART Test Cases/Auth/AuthenticateUser_FailOnBadPassworde[0;39m
2019-03-16 14:57:28.117 e[39mDEBUGe[0;39m e[36mt.AuthenticateUser_FailOnBadPassword -e[0;39m e[39m1: request = findTestObject(“Auth/Authenticate User”)e[0;39m
2019-03-16 14:57:28.263 e[39mDEBUGe[0;39m e[36mt.AuthenticateUser_FailOnBadPassword -e[0;39m e[39m2: requestBody = "{“username”: “goodusername”, “password”: “badpassword”}"e[0;39m
2019-03-16 14:57:28.465 e[39mDEBUGe[0;39m e[36mt.AuthenticateUser_FailOnBadPassword -e[0;39m e[39m3: trye[0;39m
2019-03-16 14:57:28.468 e[39mDEBUGe[0;39m e[36mt.AuthenticateUser_FailOnBadPassword -e[0;39m e[39m1: request.setBodyContent(new com.kms.katalon.core.testobject.impl.HttpTextBodyContent(requestBody, UTF-8, application/json))e[0;39m
2019-03-16 14:57:28.469 e[39mDEBUGe[0;39m e[36mt.AuthenticateUser_FailOnBadPassword -e[0;39m e[39m6: response = sendRequest(request)e[0;39m
2019-03-16 14:57:30.083 e[34mINFO e[0;39m e[36mc.k.k.core.util.BrowserMobProxyManager -e[0;39m e[39mHAR: C:\Source\WebAPI Testing\Web API Testing\Reports\TS_AuthenticateTests\20190316_145706\requests\main\1.hare[0;39m
2019-03-16 14:57:30.177 e[39mDEBUGe[0;39m e[36mt.AuthenticateUser_FailOnBadPassword -e[0;39m e[39m7: getStatusCode()).isIn(Arrays.asList(200, 201, 202))e[0;39m
2019-03-16 14:57:30.198 e[39mDEBUGe[0;39m e[36mt.AuthenticateUser_FailOnBadPassword -e[0;39m e[39m8: verifyElementPropertyValue(response, “systemUserId”, “0”)e[0;39m
2019-03-16 14:57:30.229 e[39mDEBUGe[0;39m e[36mt.AuthenticateUser_FailOnBadPassword -e[0;39m e[39m9: getResponseText()).contains(“Invalid login ID or password”)e[0;39m
2019-03-16 14:57:30.229 e[34mINFO e[0;39m e[36mc.k.katalon.core.main.TestCaseExecutor -e[0;39m e[39mEND Test Cases/Auth/AuthenticateUser_FailOnBadPassworde[0;39m
2019-03-16 14:57:30.229 e[34mINFO e[0;39m e[36mc.k.katalon.core.main.TestCaseExecutor -e[0;39m e[39m--------------------e[0;39m
2019-03-16 14:57:30.229 e[34mINFO e[0;39m e[36mc.k.katalon.core.main.TestCaseExecutor -e[0;39m e[39mSTART Test Cases/Auth/AuthenticateUser_Successe[0;39m
2019-03-16 14:57:30.914 e[39mDEBUGe[0;39m e[36mtestcase.AuthenticateUser_Success -e[0;39m e[39m1: request = findTestObject(“Auth/Authenticate User”)e[0;39m
2019-03-16 14:57:31.343 e[39mDEBUGe[0;39m e[36mtestcase.AuthenticateUser_Success -e[0;39m e[39m2: requestBody = "{“username”: “goodusername”, “password”: “goodpassword”}"e[0;39m
2019-03-16 14:57:31.358 e[39mDEBUGe[0;39m e[36mtestcase.AuthenticateUser_Success -e[0;39m e[39m3: trye[0;39m
2019-03-16 14:57:31.444 e[39mDEBUGe[0;39m e[36mtestcase.AuthenticateUser_Success -e[0;39m e[39m1: request.setBodyContent(new com.kms.katalon.core.testobject.impl.HttpTextBodyContent(requestBody, UTF-8, application/json))e[0;39m
2019-03-16 14:57:31.460 e[39mDEBUGe[0;39m e[36mtestcase.AuthenticateUser_Success -e[0;39m e[39m6: response = sendRequest(request)e[0;39m
2019-03-16 14:57:32.540 e[34mINFO e[0;39m e[36mc.k.k.core.util.BrowserMobProxyManager -e[0;39m e[39mHAR: C:\Source\WebAPI Testing\Web API Testing\Reports\TS_AuthenticateTests\20190316_145706\requests\main\2.hare[0;39m
2019-03-16 14:57:32.868 e[39mDEBUGe[0;39m e[36mtestcase.AuthenticateUser_Success -e[0;39m e[39m7: getStatusCode()).isIn(Arrays.asList(200, 201, 202))e[0;39m
2019-03-16 14:57:32.896 e[39mDEBUGe[0;39m e[36mtestcase.AuthenticateUser_Success -e[0;39m e[39m8: verifyElementPropertyValue(response, “systemUserId”, “40”)e[0;39m
2019-03-16 14:57:33.079 e[34mINFO e[0;39m e[36mc.k.katalon.core.main.TestCaseExecutor -e[0;39m e[39mEND Test Cases/Auth/AuthenticateUser_Successe[0;39m
2019-03-16 14:57:33.394 e[34mINFO e[0;39m e[36mc.k.katalon.core.main.TestSuiteExecutor -e[0;39m e[39m--------------------e[0;39m
2019-03-16 14:57:33.394 e[34mINFO e[0;39m e[36mc.k.katalon.core.main.TestSuiteExecutor -e[0;39m e[39mEND Test Suites/TS_AuthenticateTestse[0;39m
2019-03-16 14:57:33.394 e[34mINFO e[0;39m e[36mc.k.katalon.core.main.TestSuiteExecutor -e[0;39m e[39m====================e[0;39m

[devalex88]

Thank you for this report. Could you please try again with version 6.1.0 beta?

[D_Adams]

I attempted the same tests with the 6.1.0 beta, and they yielded the same results. I’m offering up my Custom Keywords and a sample test to see if this might shed some light on my results.

Here are the contents of my Keywords

package tokens

import static com.kms.katalon.core.testobject.ObjectRepository.findTestObject

import com.kms.katalon.core.annotation.Keyword
import com.kms.katalon.core.testobject.ConditionType
import com.kms.katalon.core.testobject.RequestObject
import com.kms.katalon.core.testobject.ResponseObject
import com.kms.katalon.core.testobject.TestObject
import com.kms.katalon.core.testobject.TestObjectProperty
import com.kms.katalon.core.webservice.keyword.WSBuiltInKeywords as WS

@Keyword
def String getToken(){
	def authResponse = WS.sendRequestAndVerify(findTestObject('Auth/Authenticate User'))
	def slurper = new groovy.json.JsonSlurper()
	def result = slurper.parseText(authResponse.getResponseBodyContent())
	return result.token
}

@Keyword
def ResponseObject sendAuthorizedRequest(TestObject tstObject, String access_token){

	// Build authentication header, send it, then pass the response back.
	def authRequest = (RequestObject)tstObject
	ArrayList<TestObjectProperty> httpHeader = new ArrayList<TestObjectProperty>()
	httpHeader.add(new TestObjectProperty('Authorization', ConditionType.EQUALS, "Bearer " + access_token))
	authRequest.setHttpHeaderProperties(httpHeader)
	def response = WS.sendRequest(authRequest)
	return response
}

And here is what one of my tests for success looks like…

import static com.kms.katalon.core.testobject.ObjectRepository.findTestObject
import com.kms.katalon.core.webservice.keyword.WSBuiltInKeywords as WS

def token = CustomKeywords.'tokens.TokenUtilities.getToken'()
def secureResponse = CustomKeywords.'tokens.TokenUtilities.sendAuthorizedRequest'(findTestObject('SystemUser/GetSystemUsers'), token)

WS.verifyResponseStatusCode(secureResponse, 200)
WS.verifyElementPropertyValue(secureResponse, 'data[0].registeredEmailAddress', 'samplemail@sample.coms')

To test this further, I reversed the order of execution in the test suite collection, and I’ve also duplicated a test suite so that the same test is run from within the test suite collection. In both cases, the first test suite in the collection is the test that’s executed; all other subsequent test suites are ignored.

Have I found a bug, or is there something obvious that I’m missing? I’m getting an access token and sending it as part of a request in the same test case; I would have expected that all of the test suites in the collection would be executed.

[D_Adams]

Browsing through the reports folder, the report shows a different story. It’s actually reporting the results correctly in the report summary. It’s just not showing them on the log viewer correctly.

Not sure how much more information I can provide to you on this issue.

[devalex88]

Sorry I thought you were mentioning a bug with Test Suite Collection execution.

For your case, please find this button in the bottom section:

[D_Adams]

Great! Many thanks! I needed to do more digging apparently