java.lang.OutOfMemoryError multipart file upload

I’m receiving an out heap space memory error when calling a multipart file upload of 2GB file. The same test works in postman. Still I’m seeing the same error after increasing katalon’s xmx heap to 5G

Operating System

Windows 10

Katalon Studio Version

8.1.0 enterprise

Katalon Studio logs

Reason:
ERROR c.k.k.core.keyword.internal.KeywordMain - :x: Unable to send request (Root cause: java.lang.OutOfMemoryError
at org.apache.http.entity.mime.content.InputStreamBody.writeTo(InputStreamBody.java:93)
at org.apache.http.entity.mime.AbstractMultipartForm.doWriteTo(AbstractMultipartForm.java:134)
at org.apache.http.entity.mime.AbstractMultipartForm.writeTo(AbstractMultipartForm.java:157)
at org.apache.http.entity.mime.MultipartFormEntity.writeTo(MultipartFormEntity.java:103)
at com.kms.katalon.core.testobject.impl.HttpFormDataBodyContent.writeTo(HttpFormDataBodyContent.java:95)
at com.kms.katalon.core.webservice.common.RestfulClient.buildHttpRequestWithBody(RestfulClient.java:44)
at com.kms.katalon.core.webservice.common.RestfulClient.buildHttpRequest(RestfulClient.java:27)
at com.kms.katalon.core.webservice.common.RestfulClient.buildHttpRequest(RestfulClient.java:1)
at com.kms.katalon.core.webservice.common.BasicRequestor.send(BasicRequestor.java:55)
at com.kms.katalon.core.webservice.helper.WebServiceCommonHelper.sendRequest(WebServiceCommonHelper.java:28)
at com.kms.katalon.core.webservice.keyword.builtin.SendRequestKeyword$_sendRequest_closure1.doCall(SendRequestKeyword.groovy:42)
at com.kms.katalon.core.webservice.keyword.builtin.SendRequestKeyword$_sendRequest_closure1.call(SendRequestKeyword.groovy)
at com.kms.katalon.core.keyword.internal.KeywordMain.runKeyword(KeywordMain.groovy:74)
at com.kms.katalon.core.webservice.keyword.builtin.SendRequestKeyword.sendRequest(SendRequestKeyword.groovy:38)
at com.kms.katalon.core.webservice.keyword.builtin.SendRequestKeyword.execute(SendRequestKeyword.groovy:33)
at com.kms.katalon.core.keyword.internal.KeywordExecutor.executeKeywordForPlatform(KeywordExecutor.groovy:74)
at com.kms.katalon.core.webservice.keyword.WSBuiltInKeywords.sendRequest(WSBuiltInKeywords.groovy:40)
at com.kms.katalon.core.webservice.keyword.WSBuiltInKeywords$sendRequest.call(Unknown Source)
at Upload file (200) - Very Large File.run(Upload file (200) - Very Large File:20)
at com.kms.katalon.core.main.ScriptEngine.run(ScriptEngine.java:194)
at com.kms.katalon.core.main.ScriptEngine.runScriptAsRawText(ScriptEngine.java:119)
at com.kms.katalon.core.main.TestCaseExecutor.runScript(TestCaseExecutor.java:430)
at com.kms.katalon.core.main.TestCaseExecutor.doExecute(TestCaseExecutor.java:421)
at com.kms.katalon.core.main.TestCaseExecutor.processExecutionPhase(TestCaseExecutor.java:400)
at com.kms.katalon.core.main.TestCaseExecutor.accessMainPhase(TestCaseExecutor.java:392)
at com.kms.katalon.core.main.TestCaseExecutor.execute(TestCaseExecutor.java:273)
at com.kms.katalon.core.main.TestCaseMain.runTestCase(TestCaseMain.java:142)
at com.kms.katalon.core.main.TestCaseMain.runTestCase(TestCaseMain.java:133)
at com.kms.katalon.core.main.TestCaseMain$runTestCase$0.call(Unknown Source)
at TempTestCase1635282531159.run(TempTestCase1635282531159.groovy:25)
)
2021-10-26 17:09:03.276 ERROR c.k.katalon.core.main.TestCaseExecutor - :x: Test Cases/Upload file (200) - Very Large File FAILED.
Reason:
com.kms.katalon.core.exception.StepFailedException: Unable to send request (Root cause: java.lang.OutOfMemoryError
at org.apache.http.entity.mime.content.InputStreamBody.writeTo(InputStreamBody.java:93)
at org.apache.http.entity.mime.AbstractMultipartForm.doWriteTo(AbstractMultipartForm.java:134)
at org.apache.http.entity.mime.AbstractMultipartForm.writeTo(AbstractMultipartForm.java:157)
at org.apache.http.entity.mime.MultipartFormEntity.writeTo(MultipartFormEntity.java:103)
at com.kms.katalon.core.testobject.impl.HttpFormDataBodyContent.writeTo(HttpFormDataBodyContent.java:95)
at com.kms.katalon.core.webservice.common.RestfulClient.buildHttpRequestWithBody(RestfulClient.java:44)
at com.kms.katalon.core.webservice.common.RestfulClient.buildHttpRequest(RestfulClient.java:27)
at com.kms.katalon.core.webservice.common.RestfulClient.buildHttpRequest(RestfulClient.java:1)
at com.kms.katalon.core.webservice.common.BasicRequestor.send(BasicRequestor.java:55)
at com.kms.katalon.core.webservice.helper.WebServiceCommonHelper.sendRequest(WebServiceCommonHelper.java:28)
at com.kms.katalon.core.webservice.keyword.builtin.SendRequestKeyword$_sendRequest_closure1.doCall(SendRequestKeyword.groovy:42)
at com.kms.katalon.core.webservice.keyword.builtin.SendRequestKeyword$_sendRequest_closure1.call(SendRequestKeyword.groovy)
at com.kms.katalon.core.keyword.internal.KeywordMain.runKeyword(KeywordMain.groovy:74)
at com.kms.katalon.core.webservice.keyword.builtin.SendRequestKeyword.sendRequest(SendRequestKeyword.groovy:38)
at com.kms.katalon.core.webservice.keyword.builtin.SendRequestKeyword.execute(SendRequestKeyword.groovy:33)
at com.kms.katalon.core.keyword.internal.KeywordExecutor.executeKeywordForPlatform(KeywordExecutor.groovy:74)
at com.kms.katalon.core.webservice.keyword.WSBuiltInKeywords.sendRequest(WSBuiltInKeywords.groovy:40)
at com.kms.katalon.core.webservice.keyword.WSBuiltInKeywords$sendRequest.call(Unknown Source)
at Upload file (200) - Very Large File.run(Upload file (200) - Very Large File:20)
at com.kms.katalon.core.main.ScriptEngine.run(ScriptEngine.java:194)
at com.kms.katalon.core.main.ScriptEngine.runScriptAsRawText(ScriptEngine.java:119)
at com.kms.katalon.core.main.TestCaseExecutor.runScript(TestCaseExecutor.java:430)
at com.kms.katalon.core.main.TestCaseExecutor.doExecute(TestCaseExecutor.java:421)
at com.kms.katalon.core.main.TestCaseExecutor.processExecutionPhase(TestCaseExecutor.java:400)
at com.kms.katalon.core.main.TestCaseExecutor.accessMainPhase(TestCaseExecutor.java:392)
at com.kms.katalon.core.main.TestCaseExecutor.execute(TestCaseExecutor.java:273)
at com.kms.katalon.core.main.TestCaseMain.runTestCase(TestCaseMain.java:142)
at com.kms.katalon.core.main.TestCaseMain.runTestCase(TestCaseMain.java:133)
at com.kms.katalon.core.main.TestCaseMain$runTestCase$0.call(Unknown Source)
at TempTestCase1635282531159.run(TempTestCase1635282531159.groovy:25)
)
at com.kms.katalon.core.keyword.internal.KeywordMain.stepFailed(KeywordMain.groovy:50)
at com.kms.katalon.core.keyword.internal.KeywordMain.stepFailed(KeywordMain.groovy)
at com.kms.katalon.core.keyword.internal.KeywordMain.runKeyword(KeywordMain.groovy:76)
at com.kms.katalon.core.webservice.keyword.builtin.SendRequestKeyword.sendRequest(SendRequestKeyword.groovy:38)
at com.kms.katalon.core.webservice.keyword.builtin.SendRequestKeyword.execute(SendRequestKeyword.groovy:33)
at com.kms.katalon.core.keyword.internal.KeywordExecutor.executeKeywordForPlatform(KeywordExecutor.groovy:74)
at com.kms.katalon.core.webservice.keyword.WSBuiltInKeywords.sendRequest(WSBuiltInKeywords.groovy:40)
at com.kms.katalon.core.webservice.keyword.WSBuiltInKeywords$sendRequest.call(Unknown Source)
at Upload file (200) - Very Large File.run(Upload file (200) - Very Large File:20)
at com.kms.katalon.core.main.ScriptEngine.run(ScriptEngine.java:194)
at com.kms.katalon.core.main.ScriptEngine.runScriptAsRawText(ScriptEngine.java:119)
at com.kms.katalon.core.main.TestCaseExecutor.runScript(TestCaseExecutor.java:430)
at com.kms.katalon.core.main.TestCaseExecutor.doExecute(TestCaseExecutor.java:421)
at com.kms.katalon.core.main.TestCaseExecutor.processExecutionPhase(TestCaseExecutor.java:400)
at com.kms.katalon.core.main.TestCaseExecutor.accessMainPhase(TestCaseExecutor.java:392)
at com.kms.katalon.core.main.TestCaseExecutor.execute(TestCaseExecutor.java:273)
at com.kms.katalon.core.main.TestCaseMain.runTestCase(TestCaseMain.java:142)
at com.kms.katalon.core.main.TestCaseMain.runTestCase(TestCaseMain.java:133)
at com.kms.katalon.core.main.TestCaseMain$runTestCase$0.call(Unknown Source)
at TempTestCase1635282531159.run(TempTestCase1635282531159.groovy:25)
Caused by: java.lang.OutOfMemoryError
at org.apache.http.entity.mime.content.InputStreamBody.writeTo(InputStreamBody.java:93)
at org.apache.http.entity.mime.AbstractMultipartForm.doWriteTo(AbstractMultipartForm.java:134)
at org.apache.http.entity.mime.AbstractMultipartForm.writeTo(AbstractMultipartForm.java:157)
at org.apache.http.entity.mime.MultipartFormEntity.writeTo(MultipartFormEntity.java:103)
at com.kms.katalon.core.testobject.impl.HttpFormDataBodyContent.writeTo(HttpFormDataBodyContent.java:95)
at com.kms.katalon.core.webservice.common.RestfulClient.buildHttpRequestWithBody(RestfulClient.java:44)
at com.kms.katalon.core.webservice.common.RestfulClient.buildHttpRequest(RestfulClient.java:27)
at com.kms.katalon.core.webservice.common.RestfulClient.buildHttpRequest(RestfulClient.java:1)
at com.kms.katalon.core.webservice.common.BasicRequestor.send(BasicRequestor.java:55)
at com.kms.katalon.core.webservice.helper.WebServiceCommonHelper.sendRequest(WebServiceCommonHelper.java:28)
at com.kms.katalon.core.webservice.keyword.builtin.SendRequestKeyword$_sendRequest_closure1.doCall(SendRequestKeyword.groovy:42)
at com.kms.katalon.core.webservice.keyword.builtin.SendRequestKeyword$_sendRequest_closure1.call(SendRequestKeyword.groovy)
at com.kms.katalon.core.keyword.internal.KeywordMain.runKeyword(KeywordMain.groovy:74)
… 17 more

Expected Behavior:

Actual Behavior:

Why not you try to allocate even larger Xmx if possible?
8G, 16G …


You can read the source of com.kms.katalon.core.testobject.impl.HttpTextBodyContent class here:

If you read it, you can see it makes a few copies of 2GB bytes in memory runtime. Unfortunately It is designed memory-inefficient. So, -Xmx=5G might be not large enough.

Thank you for a quick reply. I increased xms to 4G and xmx to 8G but it’s still failing . Seems like my limit is around 100MB before it crashes . Anything above that throws an outofmem exception.

How did you specified it? Any screenshot for evidence?

Reference:

Hi here is the specification we gave

-startup
plugins/org.eclipse.equinox.launcher_1.5.700.v20200207-2156.jar
–launcher.library
plugins/org.eclipse.equinox.launcher.win32.win32.x86_64_1.1.1200.v20200508-1552
-data
@noDefault
-vmargs
-XX:+UseG1GC
-XX:+UseStringDeduplication
-Xms4G
-Dfile.encoding=utf-8
-Xmx8G
Image

Image

How about trying even larger memory?

Same issue is occurring. Any file over around 200mb has the same issue of heap space.

Any other troubleshooting that we can apply if the larger memory is not helping with the heap space?

@sahib.s.virk

As the screenshot reveals, you seem to know the VisualVM tool, which would provide good enough troubleshooting features.

As far as I read the source of Katalon Studio’s WS keywords, they are not well-designed and well-tested to process multipart/form-data.

  1. As the following post tells, it has a bug in String encoding.

I would not be surprised if they raise OutOfMemoryError when you try to upload a multipart/form-data that comprises with binary data.

  1. I would rather advise you to stop using Katalon Studio if you seriously want to upload your files.

  2. @wade.ward mentioned that he successfully uploaded the 2GB file using Postman. Then, @sahib.s.virk, why not you use Postman? ==> @Jass @duyluong

  3. Katalon Studio bundles the Apache HttpClient libraries as .classpath file shows:

	<classpathentry kind="lib" path="/Applications/Katalon Studio.app/Contents/Eclipse/plugins/org.apache.httpcomponents.httpclient_4.5.1.jar"/>
	<classpathentry kind="lib" path="/Applications/Katalon Studio.app/Contents/Eclipse/plugins/org.apache.httpcomponents.httpcore_4.4.4.jar"/>
	<classpathentry kind="lib" path="/Applications/Katalon Studio.app/Contents/Eclipse/plugins/org.apache.httpcomponents.httpmime_4.5.1.jar"/>

Therefore you can write a Groovy script in Katalon Studio that uploads files using the Apache HttpClient libs directly. I guess that approach would be rather hopeful.

  1. The following code is my sample Test Case script in my Katalon Studio project.

This code shows how to write a HTTP client using Apache HttpClient library that uploads 1.8GB file as mime multipart/form-data. This client worked for me. However the HttpServer that I wrote was dum, and Timeout occurred while transferring a large file of 1.8GB. I failed to finish uploading the file using this pair of client/server :persevere: , but I am not going to work on it any longer

@duyluong

Could you look into this?

@sahib.s.virk

As @Jass wrote, Katalon Forum is volunteer-based, users help each other. The Katalon Support team is not committed to watch this and reply to the issues here .

If you are paying for KSE and KRE, you should create a ticket at https://support.katalon.com/hc/en-us and ask their immediate action.