Save API response (zip file) to local machine

Hello,

I perform a GET request to which the response is a zip file. I would like to save that to file to my local machine. How would I go about that?

My code now looks as follows:

def zipFile = WS.sendRequest(findTestObject('<testobject>'))
	println('Succesfully sent request for zipfile')
	String filepath = "<filepath>"
	FileOutputStream fileOut = new FileOutputStream(filepath.zip");
	ObjectOutputStream objectOut = new ObjectOutputStream(fileOut);
	objectOut.writeObject(zipFile);
	objectOut.close();
	System.out.println("The Object  was succesfully written to a file");

This creates the following error:

2020-11-12 16:55:48.191 ERROR c.k.katalon.core.main.TestCaseExecutor - :x: Test Cases/TESTCASE - testcase FAILED.
Reason:
java.io.NotSerializableException: com.kms.katalon.core.testobject.ResponseObject
at java_io_ObjectOutput$writeObject.call(Unknown Source)
at TESTCASE - testcase.run(TESTCASE - testcase:42)
at com.kms.katalon.core.main.ScriptEngine.run(ScriptEngine.java:194)
at com.kms.katalon.core.main.ScriptEngine.runScriptAsRawText(ScriptEngine.java:119)
at com.kms.katalon.core.main.TestCaseExecutor.runScript(TestCaseExecutor.java:339)
at com.kms.katalon.core.main.TestCaseExecutor.doExecute(TestCaseExecutor.java:330)
at com.kms.katalon.core.main.TestCaseExecutor.processExecutionPhase(TestCaseExecutor.java:309)
at com.kms.katalon.core.main.TestCaseExecutor.accessMainPhase(TestCaseExecutor.java:301)
at com.kms.katalon.core.main.TestCaseExecutor.execute(TestCaseExecutor.java:235)
at com.kms.katalon.core.main.TestCaseMain.runTestCase(TestCaseMain.java:114)
at com.kms.katalon.core.main.TestCaseMain.runTestCase(TestCaseMain.java:105)
at com.kms.katalon.core.main.TestCaseMain$runTestCase$0.call(Unknown Source)
at TempTestCase1605196540670.run(TempTestCase1605196540670.groovy:25)

I understand the error says the object “zipFile” is not serializable. I do not know how to fix this though.

Update:

I managed to fix the error I think. My code is now as follows:

ResponseObject zipFile = WS.sendRequest(findTestObject('<testobject>'))
println('Succesfully sent request for zipfile')
String filepath = "<filepath>"
File targetFile = new File(filepath + ".zip")
OutputStream outputStream = new FileOutputStream(targetFile)
zipFile.getBodyContent().writeTo(outputStream)

This leads to corrupt zipfiles however. I thought it might have something to do with the response itself so I tried a manual download of a zipfile through Postman. This lead to a non corrupt zipfile which I could access and extract. Hence I figured there must be something wrong with my script. I have no clue what though.

Update 2:

I managed to fix one problem. I don’t get corrupt zipfiles anymore. I had to specify in the HTTP header what the content type is, ie application/zip. However, the zipfiles I get now are empty. I did not change anything in the code. It is the same as in my previous update.

Hi @rewien,

Does your request contain Authorization headers?

If not, you can use Apache common library to download the file:

import org.apache.commons.io.FileUtils

FileUtils.copyURLToFile(new URL('your url'), new File('path to file'))

It does contain Authorization headers I’m afraid

@rewien

You have misunderstood: the ResponseObject object is NOT an Object which is serializable to a zip file. Please have a look at the API document of com.kms.katalon.core.testobject.ResponseObject

A call to ResponseObject.getBodyContent() returns an instance of com.kms.katalon.core.testobject.HttpBodyContent. And HttpBodyContent interface implements the writeTo(OutputStream outstream) method. You should try this method to serialise the byte array as the Http Body Content.

@kazurayam Thank you for your response. Do you have any tips on how to go about that?

Hi @duyluong

Just for testing purposes I tried your suggestion which led to the following exception:

javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

A little bit of googling did get me somewhere. If I understand correctly, I will have to install the certificate within the virtual machine in which Katalon is also located?

@rewien

I tried my idea but I was not successful.

I could not get the byte array contained in the com.kms.katalon.core.testobject.ResponseObject in its raw format. I found that com.kms.katalon.core.testobject.ResponseObject.getBodyContent() returns an instance of com.kms.katalon.core.testobject.HttpTextBodyContent. This means, the ResponseObject converts the byte array contained in the HTTP Response into a String regardless of the original Content-Type (application/octet-stream).

I think, you can not rely on the Katalon WS API to save a downloaded ZIP file from a URL locally. But you can use the Apache HttpClient library directly in your Test Case script. See the following sample code in Java.

Katalon Studio bundles the Apache HttpClient library so that you can call it in you test case scripts.

@ThanhTo

Here I copy&pasted my study.

import com.kms.katalon.core.testobject.ConditionType
import com.kms.katalon.core.testobject.RequestObject
import com.kms.katalon.core.testobject.ResponseObject

import com.kms.katalon.core.testobject.TestObjectProperty
import com.kms.katalon.core.testobject.impl.HttpTextBodyContent
import com.kms.katalon.core.webservice.keyword.WSBuiltInKeywords as WS
import com.kms.katalon.core.testobject.HttpBodyContent
import com.kms.katalon.core.testobject.impl.HttpFileBodyContent

RequestObject scriptedWso = new RequestObject('Download a zip file')
//scriptedWso.setBodyContent(httpBodyContent)
scriptedWso.setServiceType('REST')
//scriptedWso.setHttpHeaderProperties(httpHeaderProperties)
String url = 'https://github.com/kazurayam/VisualTestingInKatalonStudio/releases/download/1.23.2/vt-example-1.23.2.zip'
scriptedWso.setRestUrl(url)
scriptedWso.setRestRequestMethod('GET')

// Send the request and get the response
ResponseObject response = WS.sendRequest(scriptedWso)
	
// switch by the HTTP Status Code

switch (response.getStatusCode()) {
	case '200':
		break
	case '301':
	case '302':
		Map<String, List<String>> headerFields = response.getHeaderFields()
		for (key in headerFields.keySet()) {
			println("${key}:${headerFields.get(key)}")
		}
		String redirectedTo = response.getHeaderField('location')
		assert redirectedTo != null
		println("redirectedTo=${redirectedTo}")
		RequestObject altWso = new RequestObject('Download a zip file from redirected location')
		altWso.setServiceType('REST')
		altWso.setRestUrl(redirectedTo)
		altWso.setRestRequestMethod('GET')
		response = WS.sendRequest(altWso)
		assert response.getStatusCode() == 200
		break
	default:
		throw new IllegalStateException("reponse.getStatusCode()==" + response.getStatusCode())
}

WS.verifyResponseStatusCode(response, 200)
// Result: Test case successfully executed
println('Succesfully sent a request')

println("response.getHeaderFields" + response.getHeaderFields())

HttpBodyContent hbc = response.getBodyContent()
println("hbc.getContentEncoding=${hbc.getContentEncoding()}")
println("hbc.getContentLength=${hbc.getContentLength()}")
println("hbc.getContentType=${hbc.getContentType()}")


String filepath = "vt-example-1.23.2.zip"
File outFile = new File(filepath)
OutputStream outputStream = new FileOutputStream(outFile)
hbc.writeTo(outputStream)
outputStream.flush()
outputStream.close()
assert outFile.exists()
System.out.println("The Object  was succesfully written to a file")

//However, the created file is found malformed as a zip file.

The file created by this script was found mal-formed as a zip file. I can not open the file with any archivers/editors.

The Content-Length header of HTTP Response showed 31973, but the size of the created file was 49923. Why the size changed? This shouldn’t happen.

I think it is due to implicit data-type conversion by ResponseObject class.

Hi @kazurayam

Thanks again for your response and your help throughout this. I implemented the sample code you provided but it still seems to get stuck on certificate issues:

2020-11-24 12:47:14.791 ERROR c.k.katalon.core.main.TestCaseExecutor   - ❌ Test Cases/Test FAILED.
Reason:
java.lang.IllegalStateException: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at Downloader.download(Script1605280507559.groovy:121)
	at Downloader$download.call(Unknown Source)
	at Test.run(Test:105)
	at com.kms.katalon.core.main.ScriptEngine.run(ScriptEngine.java:194)
	at com.kms.katalon.core.main.ScriptEngine.runScriptAsRawText(ScriptEngine.java:119)
	at com.kms.katalon.core.main.TestCaseExecutor.runScript(TestCaseExecutor.java:339)
	at com.kms.katalon.core.main.TestCaseExecutor.doExecute(TestCaseExecutor.java:330)
	at com.kms.katalon.core.main.TestCaseExecutor.processExecutionPhase(TestCaseExecutor.java:309)
	at com.kms.katalon.core.main.TestCaseExecutor.accessMainPhase(TestCaseExecutor.java:301)
	at com.kms.katalon.core.main.TestCaseExecutor.execute(TestCaseExecutor.java:235)
	at com.kms.katalon.core.main.TestCaseMain.runTestCase(TestCaseMain.java:114)
	at com.kms.katalon.core.main.TestCaseMain.runTestCase(TestCaseMain.java:105)
	at com.kms.katalon.core.main.TestCaseMain$runTestCase$0.call(Unknown Source)
	at TempTestCase1606218423703.run(TempTestCase1606218423703.groovy:25)
Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:394)
	at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:353)
	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:134)
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
	at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:71)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:220)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:164)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:139)
	at org.apache.http.client.HttpClient$execute.call(Unknown Source)
	at Downloader.download(Script1605280507559.groovy:118)
	at Downloader$download.call(Unknown Source)
	at Script1605280507559.run(Script1605280507559.groovy:105)
	... 11 more
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	... 29 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	... 29 more

I think I know how to fix this. Does this mean I have to add the certificate belonging to the URL I perform a get request on to the cacerts file found in Katalon_Studio/jre/lib/security?
So basically, what they are saying here: https://stackoverflow.com/questions/21076179/pkix-path-building-failed-and-unable-to-find-valid-certification-path-to-requ/36427118#36427118

Asking to me? I don’t know your target URL, how it is secured, what type of certificate you need to add for it.

Why not you try what you think you should?

@kazurayam That sounds kind of rude to be honest… Anyways I did think about what has to be done and this is what I found to might be the solution. Since I am not proficient in Java and performing this solution requires resources I figured it would be wise to ask whether I’m on the right track. I will try it though and update this thread if I succeed in fixing this issue. That way others facing the same problem will at least have a possible solution.

Sorry, I wan’t polite.

may sound rude but is not.
You have to understand that without proper details and access to your AUT, most of the time we can only guess.
Kindly read this: [TIP] How To Help Us Help You!