java.lang.OutOfMemoryError multipart file upload

wade.ward · October 27, 2021, 4:03am

I’m receiving an out heap space memory error when calling a multipart file upload of 2GB file. The same test works in postman. Still I’m seeing the same error after increasing katalon’s xmx heap to 5G

Operating System

Windows 10

Katalon Studio Version

8.1.0 enterprise

Katalon Studio logs

Reason:
ERROR c.k.k.core.keyword.internal.KeywordMain - Unable to send request (Root cause: java.lang.OutOfMemoryError
at org.apache.http.entity.mime.content.InputStreamBody.writeTo(InputStreamBody.java:93)
at org.apache.http.entity.mime.AbstractMultipartForm.doWriteTo(AbstractMultipartForm.java:134)
at org.apache.http.entity.mime.AbstractMultipartForm.writeTo(AbstractMultipartForm.java:157)
at org.apache.http.entity.mime.MultipartFormEntity.writeTo(MultipartFormEntity.java:103)
at com.kms.katalon.core.testobject.impl.HttpFormDataBodyContent.writeTo(HttpFormDataBodyContent.java:95)
at com.kms.katalon.core.webservice.common.RestfulClient.buildHttpRequestWithBody(RestfulClient.java:44)
at com.kms.katalon.core.webservice.common.RestfulClient.buildHttpRequest(RestfulClient.java:27)
at com.kms.katalon.core.webservice.common.RestfulClient.buildHttpRequest(RestfulClient.java:1)
at com.kms.katalon.core.webservice.common.BasicRequestor.send(BasicRequestor.java:55)
at com.kms.katalon.core.webservice.helper.WebServiceCommonHelper.sendRequest(WebServiceCommonHelper.java:28)
at com.kms.katalon.core.webservice.keyword.builtin.SendRequestKeyword$_sendRequest_closure1.doCall(SendRequestKeyword.groovy:42)
at com.kms.katalon.core.webservice.keyword.builtin.SendRequestKeyword$_sendRequest_closure1.call(SendRequestKeyword.groovy)
at com.kms.katalon.core.keyword.internal.KeywordMain.runKeyword(KeywordMain.groovy:74)
at com.kms.katalon.core.webservice.keyword.builtin.SendRequestKeyword.sendRequest(SendRequestKeyword.groovy:38)
at com.kms.katalon.core.webservice.keyword.builtin.SendRequestKeyword.execute(SendRequestKeyword.groovy:33)
at com.kms.katalon.core.keyword.internal.KeywordExecutor.executeKeywordForPlatform(KeywordExecutor.groovy:74)
at com.kms.katalon.core.webservice.keyword.WSBuiltInKeywords.sendRequest(WSBuiltInKeywords.groovy:40)
at com.kms.katalon.core.webservice.keyword.WSBuiltInKeywords$sendRequest.call(Unknown Source)
at Upload file (200) - Very Large File.run(Upload file (200) - Very Large File:20)
at com.kms.katalon.core.main.ScriptEngine.run(ScriptEngine.java:194)
at com.kms.katalon.core.main.ScriptEngine.runScriptAsRawText(ScriptEngine.java:119)
at com.kms.katalon.core.main.TestCaseExecutor.runScript(TestCaseExecutor.java:430)
at com.kms.katalon.core.main.TestCaseExecutor.doExecute(TestCaseExecutor.java:421)
at com.kms.katalon.core.main.TestCaseExecutor.processExecutionPhase(TestCaseExecutor.java:400)
at com.kms.katalon.core.main.TestCaseExecutor.accessMainPhase(TestCaseExecutor.java:392)
at com.kms.katalon.core.main.TestCaseExecutor.execute(TestCaseExecutor.java:273)
at com.kms.katalon.core.main.TestCaseMain.runTestCase(TestCaseMain.java:142)
at com.kms.katalon.core.main.TestCaseMain.runTestCase(TestCaseMain.java:133)
at com.kms.katalon.core.main.TestCaseMain$runTestCase$0.call(Unknown Source)
at TempTestCase1635282531159.run(TempTestCase1635282531159.groovy:25)
)
2021-10-26 17:09:03.276 ERROR c.k.katalon.core.main.TestCaseExecutor - Test Cases/Upload file (200) - Very Large File FAILED.
Reason:
com.kms.katalon.core.exception.StepFailedException: Unable to send request (Root cause: java.lang.OutOfMemoryError
at org.apache.http.entity.mime.content.InputStreamBody.writeTo(InputStreamBody.java:93)
at org.apache.http.entity.mime.AbstractMultipartForm.doWriteTo(AbstractMultipartForm.java:134)
at org.apache.http.entity.mime.AbstractMultipartForm.writeTo(AbstractMultipartForm.java:157)
at org.apache.http.entity.mime.MultipartFormEntity.writeTo(MultipartFormEntity.java:103)
at com.kms.katalon.core.testobject.impl.HttpFormDataBodyContent.writeTo(HttpFormDataBodyContent.java:95)
at com.kms.katalon.core.webservice.common.RestfulClient.buildHttpRequestWithBody(RestfulClient.java:44)
at com.kms.katalon.core.webservice.common.RestfulClient.buildHttpRequest(RestfulClient.java:27)
at com.kms.katalon.core.webservice.common.RestfulClient.buildHttpRequest(RestfulClient.java:1)
at com.kms.katalon.core.webservice.common.BasicRequestor.send(BasicRequestor.java:55)
at com.kms.katalon.core.webservice.helper.WebServiceCommonHelper.sendRequest(WebServiceCommonHelper.java:28)
at com.kms.katalon.core.webservice.keyword.builtin.SendRequestKeyword$_sendRequest_closure1.doCall(SendRequestKeyword.groovy:42)
at com.kms.katalon.core.webservice.keyword.builtin.SendRequestKeyword$_sendRequest_closure1.call(SendRequestKeyword.groovy)
at com.kms.katalon.core.keyword.internal.KeywordMain.runKeyword(KeywordMain.groovy:74)
at com.kms.katalon.core.webservice.keyword.builtin.SendRequestKeyword.sendRequest(SendRequestKeyword.groovy:38)
at com.kms.katalon.core.webservice.keyword.builtin.SendRequestKeyword.execute(SendRequestKeyword.groovy:33)
at com.kms.katalon.core.keyword.internal.KeywordExecutor.executeKeywordForPlatform(KeywordExecutor.groovy:74)
at com.kms.katalon.core.webservice.keyword.WSBuiltInKeywords.sendRequest(WSBuiltInKeywords.groovy:40)
at com.kms.katalon.core.webservice.keyword.WSBuiltInKeywords$sendRequest.call(Unknown Source)
at Upload file (200) - Very Large File.run(Upload file (200) - Very Large File:20)
at com.kms.katalon.core.main.ScriptEngine.run(ScriptEngine.java:194)
at com.kms.katalon.core.main.ScriptEngine.runScriptAsRawText(ScriptEngine.java:119)
at com.kms.katalon.core.main.TestCaseExecutor.runScript(TestCaseExecutor.java:430)
at com.kms.katalon.core.main.TestCaseExecutor.doExecute(TestCaseExecutor.java:421)
at com.kms.katalon.core.main.TestCaseExecutor.processExecutionPhase(TestCaseExecutor.java:400)
at com.kms.katalon.core.main.TestCaseExecutor.accessMainPhase(TestCaseExecutor.java:392)
at com.kms.katalon.core.main.TestCaseExecutor.execute(TestCaseExecutor.java:273)
at com.kms.katalon.core.main.TestCaseMain.runTestCase(TestCaseMain.java:142)
at com.kms.katalon.core.main.TestCaseMain.runTestCase(TestCaseMain.java:133)
at com.kms.katalon.core.main.TestCaseMain$runTestCase$0.call(Unknown Source)
at TempTestCase1635282531159.run(TempTestCase1635282531159.groovy:25)
)
at com.kms.katalon.core.keyword.internal.KeywordMain.stepFailed(KeywordMain.groovy:50)
at com.kms.katalon.core.keyword.internal.KeywordMain.stepFailed(KeywordMain.groovy)
at com.kms.katalon.core.keyword.internal.KeywordMain.runKeyword(KeywordMain.groovy:76)
at com.kms.katalon.core.webservice.keyword.builtin.SendRequestKeyword.sendRequest(SendRequestKeyword.groovy:38)
at com.kms.katalon.core.webservice.keyword.builtin.SendRequestKeyword.execute(SendRequestKeyword.groovy:33)
at com.kms.katalon.core.keyword.internal.KeywordExecutor.executeKeywordForPlatform(KeywordExecutor.groovy:74)
at com.kms.katalon.core.webservice.keyword.WSBuiltInKeywords.sendRequest(WSBuiltInKeywords.groovy:40)
at com.kms.katalon.core.webservice.keyword.WSBuiltInKeywords$sendRequest.call(Unknown Source)
at Upload file (200) - Very Large File.run(Upload file (200) - Very Large File:20)
at com.kms.katalon.core.main.ScriptEngine.run(ScriptEngine.java:194)
at com.kms.katalon.core.main.ScriptEngine.runScriptAsRawText(ScriptEngine.java:119)
at com.kms.katalon.core.main.TestCaseExecutor.runScript(TestCaseExecutor.java:430)
at com.kms.katalon.core.main.TestCaseExecutor.doExecute(TestCaseExecutor.java:421)
at com.kms.katalon.core.main.TestCaseExecutor.processExecutionPhase(TestCaseExecutor.java:400)
at com.kms.katalon.core.main.TestCaseExecutor.accessMainPhase(TestCaseExecutor.java:392)
at com.kms.katalon.core.main.TestCaseExecutor.execute(TestCaseExecutor.java:273)
at com.kms.katalon.core.main.TestCaseMain.runTestCase(TestCaseMain.java:142)
at com.kms.katalon.core.main.TestCaseMain.runTestCase(TestCaseMain.java:133)
at com.kms.katalon.core.main.TestCaseMain$runTestCase$0.call(Unknown Source)
at TempTestCase1635282531159.run(TempTestCase1635282531159.groovy:25)
Caused by: java.lang.OutOfMemoryError
at org.apache.http.entity.mime.content.InputStreamBody.writeTo(InputStreamBody.java:93)
at org.apache.http.entity.mime.AbstractMultipartForm.doWriteTo(AbstractMultipartForm.java:134)
at org.apache.http.entity.mime.AbstractMultipartForm.writeTo(AbstractMultipartForm.java:157)
at org.apache.http.entity.mime.MultipartFormEntity.writeTo(MultipartFormEntity.java:103)
at com.kms.katalon.core.testobject.impl.HttpFormDataBodyContent.writeTo(HttpFormDataBodyContent.java:95)
at com.kms.katalon.core.webservice.common.RestfulClient.buildHttpRequestWithBody(RestfulClient.java:44)
at com.kms.katalon.core.webservice.common.RestfulClient.buildHttpRequest(RestfulClient.java:27)
at com.kms.katalon.core.webservice.common.RestfulClient.buildHttpRequest(RestfulClient.java:1)
at com.kms.katalon.core.webservice.common.BasicRequestor.send(BasicRequestor.java:55)
at com.kms.katalon.core.webservice.helper.WebServiceCommonHelper.sendRequest(WebServiceCommonHelper.java:28)
at com.kms.katalon.core.webservice.keyword.builtin.SendRequestKeyword$_sendRequest_closure1.doCall(SendRequestKeyword.groovy:42)
at com.kms.katalon.core.webservice.keyword.builtin.SendRequestKeyword$_sendRequest_closure1.call(SendRequestKeyword.groovy)
at com.kms.katalon.core.keyword.internal.KeywordMain.runKeyword(KeywordMain.groovy:74)
… 17 more

Expected Behavior:

Actual Behavior:

kazurayam · October 27, 2021, 4:15am

Why not you try to allocate even larger Xmx if possible?
8G, 16G …

You can read the source of com.kms.katalon.core.testobject.impl.HttpTextBodyContent class here:

github.com

katalon-studio/katalon-studio-testing-framework/blob/master/Include/scripts/groovy/com/kms/katalon/core/testobject/impl/HttpTextBodyContent.java

package com.kms.katalon.core.testobject.impl;

import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.nio.charset.Charset;
import java.nio.charset.IllegalCharsetNameException;

import org.apache.commons.lang.StringUtils;

import com.kms.katalon.core.testobject.HttpBodyContent;

/**
 * Represents the body content of a HTTP message (request/response) that obtains content from a {@link String}.
 * 
 * @since 5.4
 */
public class HttpTextBodyContent implements HttpBodyContent {

This file has been truncated. show original

If you read it, you can see it makes a few copies of 2GB bytes in memory runtime. Unfortunately It is designed memory-inefficient. So, -Xmx=5G might be not large enough.

wade.ward · October 27, 2021, 4:01pm

Thank you for a quick reply. I increased xms to 4G and xmx to 8G but it’s still failing . Seems like my limit is around 100MB before it crashes . Anything above that throws an outofmem exception.

kazurayam · October 27, 2021, 11:36pm

How did you specified it? Any screenshot for evidence?

Reference:
https://docs.katalon.com/katalon-studio/docs/execution-settings.html#allow-editing-jvm-parameters-in-execution-settings

sahib.s.virk · October 30, 2021, 4:31am

Hi here is the specification we gave

-startup
plugins/org.eclipse.equinox.launcher_1.5.700.v20200207-2156.jar
–launcher.library
plugins/org.eclipse.equinox.launcher.win32.win32.x86_64_1.1.1200.v20200508-1552
-data
@noDefault
-vmargs
-XX:+UseG1GC
-XX:+UseStringDeduplication
-Xms4G
-Dfile.encoding=utf-8
-Xmx8G
Image

Image

kazurayam · October 31, 2021, 8:04am

How about trying even larger memory?

sahib.s.virk · November 4, 2021, 7:56pm

Same issue is occurring. Any file over around 200mb has the same issue of heap space.

sahib.s.virk · November 9, 2021, 2:00pm

Any other troubleshooting that we can apply if the larger memory is not helping with the heap space?

kazurayam · November 9, 2021, 10:59pm

@sahib.s.virk

As the screenshot reveals, you seem to know the VisualVM tool, which would provide good enough troubleshooting features.

kazurayam · November 9, 2021, 11:15pm

As far as I read the source of Katalon Studio’s WS keywords, they are not well-designed and well-tested to process multipart/form-data.

As the following post tells, it has a bug in String encoding.

Katalon POSt Request with multipart/form-data and formdata fields conrtains utf-8 text not submittd the text in the same format

I would not be surprised if they raise OutOfMemoryError when you try to upload a multipart/form-data that comprises with binary data.

I would rather advise you to stop using Katalon Studio if you seriously want to upload your files.
@wade.ward mentioned that he successfully uploaded the 2GB file using Postman. Then, @sahib.s.virk, why not you use Postman? ==> @Jass @duyluong
Katalon Studio bundles the Apache HttpClient libraries as .classpath file shows:

	<classpathentry kind="lib" path="/Applications/Katalon Studio.app/Contents/Eclipse/plugins/org.apache.httpcomponents.httpclient_4.5.1.jar"/>
	<classpathentry kind="lib" path="/Applications/Katalon Studio.app/Contents/Eclipse/plugins/org.apache.httpcomponents.httpcore_4.4.4.jar"/>
	<classpathentry kind="lib" path="/Applications/Katalon Studio.app/Contents/Eclipse/plugins/org.apache.httpcomponents.httpmime_4.5.1.jar"/>

Therefore you can write a Groovy script in Katalon Studio that uploads files using the Apache HttpClient libs directly. I guess that approach would be rather hopeful.

The following code is my sample Test Case script in my Katalon Studio project.

https://github.com/kazurayam/OutOfMemoryErrorWhileUploadingVeryLargeMultipartFormData/blob/master/Scripts/POST_request_using_Apache_httpclient/Script1635600023664.groovy

This code shows how to write a HTTP client using Apache HttpClient library that uploads 1.8GB file as mime multipart/form-data. This client worked for me. However the HttpServer that I wrote was dum, and Timeout occurred while transferring a large file of 1.8GB. I failed to finish uploading the file using this pair of client/server , but I am not going to work on it any longer

@duyluong

Could you look into this?

kazurayam · November 10, 2021, 2:06am

@sahib.s.virk

As @Jass wrote, Katalon Forum is volunteer-based, users help each other. The Katalon Support team is not committed to watch this and reply to the issues here .

If you are paying for KSE and KRE, you should create a ticket at https://support.katalon.com/hc/en-us and ask their immediate action.

system · November 10, 2022, 4:55am

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Getting 'OutOfMemory' exception when Katalon try to generate execution report Katalon Studio katalon-studio	3	455	November 8, 2022
Java Heap Error while preparing report ( execution was successful ) Archive katalon-studio , web-testing	1	550	August 8, 2022
Fails to generate test report - java.lang.OutOfMemoryError: Java heap space Archive katalon-studio , web-testing	5	3026	August 8, 2022
Java Heap Space Error Katalon Studio katalon-studio	1	660	January 30, 2020
Unable to launch multiple test cases in test Suite (Katalon_Studio_Windows_64-6.3.2 onwards till 6.3.4) Bugs Report katalon-studio , bugs-report	7	1571	November 13, 2020