Web Service tests - REST API - XML Response : Problem with verification / getResponseBodyContent does not contain all of the response's body

#1

Hello!

My colleague is currently doing a proof of concept with Katalon Studio.
He wants to mainly test his REST API and a little of Web UI.
We think we encountered a bug with the tool, as after reading many pages of documentation, we did not found what we were doing wrong :frowning:
To be honest, should this problem not be solved, as this test is his main use case, we would abandon the POC (and try with another tool).

Environment

Operating System : Windows 10
Katalon Studio version 8.1.0

Quick Description of the problem

Verifying a response’s XML body text is failing, even though the keyword’s locator parameter was fetched using Ctrl+K shortcut in the “Response Body” view in the Object Repository (which displays the expected value of the response).
We noticed that the ResponseObject’s getResponseBodyContent method does not return all XML nodes but only ErrorCode node. Then, we tried to verify the ErrorCode text and it unexpectedly matched.

Our error logs :

09-06-2021 03:16:37 PM verifyElementText(response, "DocumentID", "A2")

Elapsed time: 0,259s

Unable to verify element text (Root cause: com.kms.katalon.core.exception.StepFailedException: Expected text is 'A2' but actual element text is: 1
at com.kms.katalon.core.keyword.internal.KeywordMain.stepFailed(KeywordMain.groovy:50)
at com.kms.katalon.core.webservice.keyword.builtin.VerifyElementTextKeyword$_verifyElementText_closure1.doCall(VerifyElementTextKeyword.groovy:52)
at com.kms.katalon.core.webservice.keyword.builtin.VerifyElementTextKeyword$_verifyElementText_closure1.call(VerifyElementTextKeyword.groovy)
at com.kms.katalon.core.keyword.internal.KeywordMain.runKeyword(KeywordMain.groovy:74)
at com.kms.katalon.core.webservice.keyword.builtin.VerifyElementTextKeyword.verifyElementText(VerifyElementTextKeyword.groovy:45)
at com.kms.katalon.core.webservice.keyword.builtin.VerifyElementTextKeyword.execute(VerifyElementTextKeyword.groovy:40)
at com.kms.katalon.core.keyword.internal.KeywordExecutor.executeKeywordForPlatform(KeywordExecutor.groovy:74)
at com.kms.katalon.core.webservice.keyword.WSBuiltInKeywords.verifyElementText(WSBuiltInKeywords.groovy:253)
at com.kms.katalon.core.webservice.keyword.WSBuiltInKeywords$verifyElementText$1.call(Unknown Source)
at TestTNRInquiry.run(TestTNRInquiry:28)
at com.kms.katalon.core.main.ScriptEngine.run(ScriptEngine.java:194)
at com.kms.katalon.core.main.ScriptEngine.runScriptAsRawText(ScriptEngine.java:119)
at com.kms.katalon.core.main.TestCaseExecutor.runScript(TestCaseExecutor.java:430)
at com.kms.katalon.core.main.TestCaseExecutor.doExecute(TestCaseExecutor.java:421)
at com.kms.katalon.core.main.TestCaseExecutor.processExecutionPhase(TestCaseExecutor.java:400)
at com.kms.katalon.core.main.TestCaseExecutor.accessMainPhase(TestCaseExecutor.java:392)
at com.kms.katalon.core.main.TestCaseExecutor.execute(TestCaseExecutor.java:273)
at com.kms.katalon.core.main.TestCaseMain.runTestCase(TestCaseMain.java:142)
at com.kms.katalon.core.main.TestCaseMain.runTestCase(TestCaseMain.java:133)
at com.kms.katalon.core.main.TestCaseMain$runTestCase$0.call(Unknown Source)
at TempTestCase1630934178960.run(TempTestCase1630934178960.groovy:25)

Steps to reproduce:

The test case is sending a POST request with a XML body to a REST API, fetching the response and then verifying the XML body of the response.
Test Case script (I removed the imports as there does not seem to be any issue with them):

response = WS.sendRequest(findTestObject('Stock-A25-XML-CAT')) // (1)
WS.verifyResponseStatusCode(response, 200)                     // (2)
WS.comment(response.getResponseText().toString())              // (3)
WS.verifyElementText(response, 'ErrorCode', '1')               // (4)
WS.verifyElementText(response, 'DocumentID', 'A2')             // (5)

The lines (3) and (4) were added for debugging purposes.
Line (3) returns, as expected, the full xml raw text:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ns2:quote_A2 xmlns:ns2="http://www.reifen.net">
<DocumentID>A2
</DocumentID>
<Variant>5
</Variant>
<ErrorHead>
<ErrorCode>1
</ErrorCode>
</ErrorHead>
<CustomerReference>
<DocumentID>
</DocumentID>
</CustomerReference>
<BuyerParty>
<PartyID>233082
</PartyID>
<AgencyCode>92
</AgencyCode>
</BuyerParty>
<OrderLine>
<LineID>1
</LineID>
<OrderedArticle>
<ArticleIdentification>
<EANUCCArticleID>3528708756786
</EANUCCArticleID>
</ArticleIdentification>
<RequestedQuantity>
<QuantityValue>2
</QuantityValue>
</RequestedQuantity>
<Error>
<ErrorCode>940
</ErrorCode>
<ErrorText>Incorrect supplier material number (not known on supplier side)
</ErrorText>
</Error>
</OrderedArticle>
</OrderLine>
</ns2:quote_A2>

But, getResponseBodyContent would only return <ErrorCode>1</ErrorCode>

The line (4) is unexpedly succeeding, while the line (5) is unexpectedly failing with the error code above.

Thank you for your help!

#2

Here is the source code of com.kms.katalon.core.testobject.ResponseObject. You can read how getResponseBodyContent and getResponseText are implemented. You may find some clue.

Beware, I have no insight about your problem.

#3
#4

Do you know if it is a problem that the XML response is expected to have two DocumentID tags at its root? I don’t know whether it could be related to the problem. Is it possible to specify the tag’s position in the locator? I saw that it is possible to use [0], [1] etc if the XML or JSON contains a list, but I don’t know if it is possible to locate the “first tag of its type” for example.
Thank you :slight_smile:

#5

@elodie.amato_ext

Change your Test Case script and try:

import static com.kms.katalon.core.testobject.ObjectRepository.findTestObject

import com.kms.katalon.core.testobject.RequestObject
import com.kms.katalon.core.testobject.ResponseObject
import com.kms.katalon.core.webservice.keyword.WSBuiltInKeywords as WS

ResponseObject response = WS.sendRequest((RequestObject)findTestObject('Stock-A25-XML-CAT')) // (1)
WS.verifyResponseStatusCode(response, 200)                     // (2)
WS.comment(response.getResponseText().toString())              // (3)
WS.verifyElementText(response, 'quote_A2.ErrorHead.ErrorCode', '''1
''')               // (4)
WS.verifyElementText(response, 'quote_A2.DocumentID', '''A2
''')             // (5)

  1. You wrote
    ErrorCode
    as a locator for a WS.verifyElementText keyword. The locator is wrong. You should write a “full path” locator which starts with the root element, tracing down each path layers to the target element, like
    quote_A2.ErrorHead.ErrorCode

  2. Your response XML contains a new-line character for each element’s content text; like

<ErrorCode>1
</ErrorCode>

which is not equivalent to the following markup:

<ErrorCode>1</ErrorCode>

It seems to me that WS.verifyElementText() keyword does strict equality check; it does not ignore redundant new line character silently. Therefore, as long as the response XML contains redundant new-line characters, your test case script must express new line character(s) as they appear in the response XML, like

WS.verifyElementText(response, locator, '''1
''')

By the way, you should imagine how difficult to write calls for WS.verifyElementText() keyword as required if the response XML contains redundant whitespaces (0x20) casually, like:

<ErrorCode>   1  
</ErrorCode>

Can you count how many 0x20 characters I wrote here? — Ans: 5

#6

@elodie.amato_ext

I have a doubt about your Response XML regarding how it uses XML Namespace.

This XML looks odd to me, because the XML Namespace http://www.reifen.net is not actually used at all in this XML document instance.

I guess, it should rather be:

<quote_A2 xmlns="http://www.reifen.net">
    <DocumentID>A2</DocumentID>
    ...

Or, alternatively

<ns2:quote_A2 xmlns:ns2="http://www.reifen.net">
    <ns2:DocumentID>A2</ns2:DocumentID>
    ...

Please suggest my doubt to your colleague.

#7

Thank you for your insights, we will investigate both !

#8

In another topic

I found that the WS.verifyElementText() keyword is problematic for XML. It is not documented clear enough how to use the keyword for XML properly. In fact this keyword only works for SOAP response without Head element. Therefore the keyword would not work for your <quote_A2> XML instance.

#9

Thank you for your help!

  1. I tried including the root tag but it did not work.
  2. My colleague looked at the XML validity and his opinion is that he sees no problem, the ns2 namespace is apparently here to be an additional namespace
  3. I did a keyword to (temporarily) replace verifyElementText in order to handle XML responses:
	public static String getXmlNodeText(String xmlText, String locator) {
		GPathResult node = new XmlSlurper().parseText(xmlText)
		def path = locator.split("\\.")
		for (String child : path) {
			node = node[child]
		}
		return node.text()
	}

	@Keyword
	public static boolean elementText(ResponseObject response, String locator, String text, FailureHandling flowControl) {
		if (response.isXmlContentType()) {
			String responseText = response.getResponseText()
			boolean res = BuiltinKeywords.verifyNotEqual(responseText, null, flowControl)
			return res && BuiltinKeywords.verifyEqual(getXmlNodeText(responseText, locator), text, flowControl)
		} else {
			return WS.verifyElementText(response, locator, text, flowControl)
		}
	}
	@Keyword
	public static boolean elementText(ResponseObject response, String locator, String text) {
		return elementText(response, locator, text, getDefaultFailureHandling())
	}

I tested it with

CustomKeywords.'s2s.Verify.elementText'(response, 'DocumentID', 'A2')
CustomKeywords.'s2s.Verify.elementText'(response, 'DocumentID[1]', '')
CustomKeywords.'s2s.Verify.elementText'(response, 'ErrorHead.ErrorCode', '1')
#10

I thought so. WS.verifyElementText() is problematic.

c/c @ThanhTo

Your colleague owns the right to define the validity of your XML. Entirely up to you. I wrote just what I guessed. Ignore me, please.

#11

Katalon Studio v8.1.0 bundles Groovy v2.4.x, which is old one. In the current Katalon Studio you can use groovy.util.slurpersupport.GPathResult.

However, the latest version of Groovy v3.x.x has deprecated this class.

The package is renamed:

In case Katalon Studio change the Groovy version from v2.4.x to v3.x.x in future, your custom keyword will not work. You have to rewrite the import statement. Please note this.

Is it likely to happen? — I don’t know. Ask Katalon Team.

#12

You showed an XML like this:

The text “A2” is followed by a new line character. It is 0x41 0x32 0x0D 0x0A in hex-decimal representation.

If it is really the case, your Custom Keyword had better to “normalize” whitespaces in the XML content text before testing equality to the text you expect; like this

(original)

        return res && BuiltinKeywords.verifyEqual(
                      getXmlNodeText(responseText, locator),
                      text, flowControl)

(proposed)

        return res && BuiltinKeywords.verifyEqual(
                      getXmlNodeText(responseText, locator).trim().replaceAll("\\s+", " "),
                      text, flowControl)

Here is my alternative test case implementation which uses javax.xml.xpath.XPath rather than Grovvy’s GPath.

import static com.kms.katalon.core.testobject.ObjectRepository.findTestObject

import javax.xml.parsers.DocumentBuilder;
import javax.xml.parsers.DocumentBuilderFactory;
import javax.xml.xpath.XPath
import javax.xml.xpath.XPathConstants;
import javax.xml.xpath.XPathFactory;

import org.w3c.dom.Document;

import com.kms.katalon.core.testobject.RequestObject
import com.kms.katalon.core.testobject.ResponseObject
import com.kms.katalon.core.webservice.keyword.WSBuiltInKeywords as WS
import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUI

ResponseObject response = WS.sendRequest((RequestObject)findTestObject('Stock-A25-XML-CAT - tuned')) // (1)
WS.verifyResponseStatusCode(response, 200)
WS.comment(response.getResponseText())

Document doc = createDocument(response.getResponseText())

WebUI.verifyEqual(evaluateXPathToString( doc, "normalize-space(//DocumentID[1])"), "A2")

/**
 * create a W3C Document object from a XML text
 */
Document createDocument(String xml) {
	DocumentBuilder builder     = DocumentBuilderFactory.newInstance().newDocumentBuilder()
	InputStream inputStream = new ByteArrayInputStream( xml.bytes )
	return builder.parse(inputStream)
}

/**
 * evaluate a XPath expression against the document, return a String
 */
String evaluateXPathToString(Document doc, String xpathExpression) {
	XPath xpath = XPathFactory.newInstance().newXPath()
	def result = xpath.evaluate( xpathExpression, doc, XPathConstants.STRING )
	
}

/**
 * evaluate a XPath expression against the document, return a list of Strings
 */
List<String> evaluateXPathToListOfString(Document doc, String xpathExpression ) {
	XPath xpath = XPathFactory.newInstance().newXPath()
	def nodes = xpath.evaluate( xpathExpression, doc, XPathConstants.NODESET )
	return nodes.collect { node -> node.textContent }
}

I think that this code is better than the one which uses GPath because I can utilise full power of the XPath technology, which is natively designed to process XML documents. For example, the XPath-builtin normalize-space() function solves the aforementioned “redundant white spaces” problem cleanly.

Not able to validate SOAP call response
#13

Thank you for both ideas. I think that normalizing the whitespaces is indeed better!
I like both Xpath and Gpath solutions, but I stayed with XmlSlurper and Gpath as we need to handle JSON responses and those can be handled with Gpath too

#14