Katalon POSt Request with multipart/form-data and formdata fields conrtains utf-8 text not submittd the text in the same format

I have POST request with multipart/form-data type and the data fields inside this POST request has text with German characters and should be treated as utf-8 format. But unfortunately katalon has only Text or file category , so i selected Text and then i selected mime type as Text/plain for the data field which has text with German characters.

The issue is the text filed is not submitted the text in utf-8 format and hence german characters are converted to some strange symbol as per HAR file even before POST request submission. I am not sure what changes i should do to this request inorder to submit the text in correct format. Please refer attached screenshot for more details and Could you please help me here on the solution for this problem.

Due to security reasons uploaded the parts that are required in this screenshot instead of uploading entire HAR file. We can have a call if you need more details than mentioned in the screenshot

The german character ü is encoded into

The screenshoot shows “charset=ISO-8859-01”. Is it in the HTTP Request header, or Response header?

If the Response Header declares “charset=ISO-8859-1” but the message body is actually encoded in UTF-8, then the Client (Katalon GUI) will display a garbled character for ü. You should change the server app so that it sets the response header with the charset value same as the message body (“charset=utf-8”).

Thank you for the reply @kazurayam. Unfortunately, this is HTTP Request header. The Katalon itself don’t have option to send the request data in UTF-8 format. Please refer attached screenshot also, whic i tried but i am getting error message if i use any input data fomat in addition plain/text format

Note: I have used Katalon V7.9.1

You are right. Katalon GUI should enable user to set “text/plain; charset=utf8” as the Content-Type of a TEXT data part in a mulitipart/form-data. But KS GU misses that option. It is one problem.

However, there is one more problem.

c/c @duyluong

I read the source of v8.0.5, com.kms.katalon.core.webservice.common.BasicRequestor#send() and other classes.

I believe, Katalon Studio is implemented so that it it writes all String data in a HTTP Request of multipart/form-data encoded in ISO-8859-1 (= US-ASCII). Katalon Studio can not serialise the TEXT data part in UTF-8 at all. The ü will be inevitably garbled.

@swami.pasam1

if you seriously need to send POST Request with multipart/form-data with non US-ASCII characters (e.g, German character ü, Japanese Hiragana こんにちは, Chinese 简化字, Russian кириллица, Arabian شُكْرًا), I would recommend you to use other WebService test tool. I do not know which one is good.

1 Like

I would show you an evidence. Please find the attached zip, which contains a small Katalon Studio project. I made it using KS v8.1.0, but is version-independent. You can open it with any KS version.

web-service-tests.zip (111.8 KB)

Steps to reproduce

  1. open the web-service-tests project with KS
  2. find a TestObject in “Object Repository” > “POST multipart form-data”
  3. The TestObject defines a WebService Request. In “HTTP Header” tab, it has a header “Content-Type: multipart/form-data”
  4. In “HTTP Body” tab, it includes 3 part:
  • “data-in-us-ascii”, TEXT, “text/plain”, “hello,world”
  • “data-in-german”, TEXT, “text/plain;charset=utf-8”, “Grüß Gott!”
  • “data-in-japanese”, TEXT, “text/plain;charset=utf-8”, “こんにちは”
  1. Katalon Studio GUI did not allow me to type “text/plain;charset=utf-8” here. So I closed KS, open an editor, select a file <projectDir>/Object Repository/POST multipart/form-data.rs which is an XML file. I editted 2 lines:
    &quot;contentType&quot;: &quot;text/plain&quot; =>
    &quot;contentType&quot;: &quot;text/plain; charset=utf-8&quot;
  2. I reopened the project in KS, I checked the Test Object and found it has text/plain; charset=utf-8
  3. In the command line, cd to the project directory, run a command:
$ cd web-service-tests
$ groovy server.groovy

here I assume you have the Groovy interpreter is installed and runnable in the commandline.
8. server.groovy is a simple HTTP server, it listens to the URL http://localhost:80/. This server is open-sourced at https://github.com/opengl-8080/groovy-http-server developed by opengl-8080. I amended the original slightly.
9. The server can print the HTTP Header and HTTP Body of the request it received.
10. The server decodes the bytes in the HTTP Request into UNICODE string assuming that the HTTP Body is encoded in UTF-8.
11. Now you want to go back the Katalon project. Run the Test Object by clicking run_katalon_test button on the Test Object definition pane.
12. In the console, the server prints the HTTP Body as follows:

$ groovy server.groovy

port=80
base-dir=/Users/kazuakiurayama/katalon-workspace/web-service-tests/.

method = GET
uri = /
body = --sWdL5Jwq5EuDfY-bB3E9w4upALqVJrm
Content-Disposition: form-data; name="data-in-german"
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit

Gr?? Gott!
--sWdL5Jwq5EuDfY-bB3E9w4upALqVJrm
Content-Disposition: form-data; name="data-in-japanese"
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit

?????
--sWdL5Jwq5EuDfY-bB3E9w4upALqVJrm
Content-Disposition: form-data; name="data-in-us-ascii"
Content-Type: text/plain
Content-Transfer-Encoding: 8bit

hello, world
--sWdL5Jwq5EuDfY-bB3E9w4upALqVJrm--

Please note, non US-ASCII characters are garbled; Grüß Gott! is printed as Gr?? Gott!. This proves that the HTTP Request sent from Katalon Studio to the server is NOT encoded in utf-8; most probably is encoded in ISO-8859-1 = US-ASCII.

I would conclude, Katalon Studio can not send a Web service Request of multipart/form-data in utf-8, therefore it can not support international languages (German, Japanese, etc) which require utf-8.

I read the source codes of Katalon Studio v8.0.5, e.g, com.kms.katalon.core.webservice.common.BasicRequestor#send() and other classes. I found it is NOT implemented to encode multipart sections in utf-8 even if these have a Content-Type text/play; charset=utf-8.

1 Like

Thank you so much @kazurayam for your efforts to share this example, as i can’t share my api details due to security reasons. I really appreciate you. I think for now, for this service type alone, i have to use some other REST api client using java code and create a seperate utility, as i can’t use katalon direct api.

Hope upcoming katalon versions will solve this issue.

@Russ_Thomas

Could you move this topic to the Bug category?

1 Like

@duyluong

Please have a look at this.

Katalon Studio v8.1.0 bundles the Apache HttpClient library. You can check it in a <projectDir>/.classpath file. org.apache.httpcomponents.httpclient_4.5.1.jar is available for a test cases script.

I added a Test Case into the aforementioned sample project “web-service-tests”. It is a HTTP client that sends a POST request with Content-Type: multipart/form-data. The text parts are encoded in charset=utf-8.

web-service-tests.zip (136.9 KB)

// original: https://www.tutorialspoint.com/apache_httpclient/apache_httpclient_response_handlers.htm

import java.nio.charset.StandardCharsets

import org.apache.http.HttpEntity;
import org.apache.http.HttpResponse
import org.apache.http.client.ClientProtocolException;
import org.apache.http.client.ResponseHandler;
import org.apache.http.client.methods.HttpUriRequest;
import org.apache.http.client.methods.RequestBuilder;
import org.apache.http.entity.ContentType;
import org.apache.http.entity.mime.HttpMultipartMode;
import org.apache.http.entity.mime.MultipartEntityBuilder;
import org.apache.http.entity.mime.content.StringBody;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.util.EntityUtils;


CloseableHttpClient httpclient = HttpClients.createDefault()

File file = new File("./server.groovy")
StringBody part_in_us_ascii = new StringBody("hello,world", new ContentType("text/plain", StandardCharsets.UTF_8))
StringBody part_in_germany = new StringBody("Grüß Gott!", new ContentType("text/plain", StandardCharsets.UTF_8))
StringBody part_in_japanese = new StringBody("こんにちは", new ContentType("text/plain", StandardCharsets.UTF_8))

// build multipart upload request
HttpEntity data = MultipartEntityBuilder.create()
	.setMode(HttpMultipartMode.BROWSER_COMPATIBLE)
	.addPart("part-in-us-ascii", part_in_us_ascii)
	.addPart("part-in-germany", part_in_germany)
	.addPart("part-in-japanese", part_in_japanese)
	.build()

// build http request and assign multipart upload data
HttpUriRequest request = RequestBuilder
	.post("http://localhost:80")
	.setEntity(data)
	.build()

println("Executing request ${request.getRequestLine()}")

// Create a custom response handler
ResponseHandler<String> responseHandler = new MyResponseHander()

String responseBody = httpclient.execute(request, responseHandler)
System.out.println("----------------------------------------")
System.out.println(responseBody)

class MyResponseHander implements ResponseHandler<String> {
	public String handleResponse(final HttpResponse response) throws IOException {
		int status = response.getStatusLine().getStatusCode();
		if (status >= 200 && status < 300) {
			HttpEntity entity = response.getEntity();
			return entity != null ? EntityUtils.toString(entity) : null;
		} else {
			throw new ClientProtocolException("Unexpected response status: " + status);
		}
	}
}

When I tried this against the http://localhost:80 server, it printed the following output in the console.

method = POST
uri = /
body = --mNPsR4gxoIl39TteGuiMPP5cw0S7NeoSu6h8dvki
Content-Disposition: form-data; name="part-in-us-ascii"

hello,world
--mNPsR4gxoIl39TteGuiMPP5cw0S7NeoSu6h8dvki
Content-Disposition: form-data; name="part-in-germany"

Grüß Gott!
--mNPsR4gxoIl39TteGuiMPP5cw0S7NeoSu6h8dvki
Content-Disposition: form-data; name="part-in-japanese"

こんにちは
--mNPsR4gxoIl39TteGuiMPP5cw0S7NeoSu6h8dvki--


As you can see, German characters and Japanese Characters are properly encoded&decoded.


Programming note

See the following line of the “http client”.

StringBody part_in_germany = new StringBody(
        "Grüß Gott!", 
        new ContentType("text/plain", 
        StandardCharsets.UTF_8))

The request has with 3 parts. Each of parts are “text/plain”, is encoded in utf-8. This line explicitly specifies which encoding is used for each parts. If 3 parts are encoded in ISO-8859-1, utf-8, Shift_JIS, then the code should look like the following:

StringBody part_in_us_ascii = new StringBody("hello,world", ...
        StandardCharsets.ISO_ISO8859_1))
StringBody part_in_germany = new StringBody("Grüß Gott!", ...
        StandardCharsets.UTF_8))
StringBody part_in_japanese = new StringBody("こんにちは", ...
        Charset.forName("Shift-JIS")))

Apache httpclient library encodes a string of a part of a Request of “multipart/form-data”
into bytes using the charset specified for each part.

As far as I see in the current version of the source code of Katalon Studio, it does not encode the string into bytes using charsets specified per each parts. It encodes whole of a request body in UNICODE into a byte array using charset=US-ASCII. Therefore non US-ASCII characters are garbled.