Katalon Docker - How to limit the use of memory

Hi,

We are using Katalon docker image for our test. We notice recently that chrome is using a lot of virtual memory on our Jenkins machine:

PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND 
 7281 root      20   0   16.8g 105156  76900 S   9.0  0.3   0:08.76 chrome 
 1112 root      20   0 7986056   1.5g  25264 S   6.0  4.7  14:58.61 java  
 7426 root      20   0   16.5g  47976  21360 S   4.3  0.1   0:04.08 chrome
12247 sysadmin  20   0 6500160   1.7g  18696 S   2.0  5.6 184:21.07 java 
 8941 root      20   0 2273728 125028  31004 S   1.3  0.4  81:59.02 dockerd
 6070 sysadmin  20   0 7581772 167172  14488 S   1.0  0.5   0:09.53 java
 7269 root      20   0   16.1g   8912   7168 S   0.7  0.0   0:00.55 chromedriver 
 7324 root      20   0   16.4g  47072  35528 S   0.7  0.1   0:00.28 chrome

I read a similar topic here Katalon Studio cost much memory - #5 by simple where we can set option to limit memory in the katalon.ini

-XX:+AggressiveOpts
-XX:MinHeapFreeRatio=1
-XX:MaxHeapFreeRatio=2
-XX:+UseSerialGC

I wonder if the equivalent can be applied in Docker? How to set those options in Docker?

Thanks for your advice :slight_smile:

@anon46315158 any idea?

yes, i have some idea.
yet again, lack of info.
how the container is started?
there are various levels to confine resources usage under containers but i have no valuable info here.
so, unless i see something like "how the container is started’ and info about the hardware capabilities of the VM subject to this, i will just wait.
My crystal globe is broken.

Hi @anon46315158,

Please find below how we start the docker container in Jenkins pipeline:

withCredentials([string(credentialsId: 'katalon-api-key', variable: 'KATALON_API_KEY')]) {
	withDockerContainer(image: "${BUILD_IMAGE_KATALON}",
			args: "-u=root -m 4G --name ${JOB_DOCKER_NETWORK} --network=${JOB_DOCKER_NETWORK} \
		   ") {
		sh("""#!/bin/bash
		
		# set locale environment variable LC_ALL to UTF-8 encoding
		export LC_ALL=C.UTF-8
		export JAVA_TOOL_OPTIONS=-Dfile.encoding=UTF-8											
		
		# running the katalon job to run the test suite or collection with the correct profile
		katalonc.sh -projectPath="${WORKSPACE}" \
					-browserType="Chrome" \
					-retry=0 \
					-statusDelay=15 \
					-testSuiteCollectionPath="test_suite_collection_test" \
					-executionProfile="TEST" \
					--config -g_languageCode="en" \
					-apiKey="${KATALON_API_KEY}" 
	""")
	}
}

About the capabilities of the VM, I need to contact DevOps team to get the information, I don’t have access on that machine.
If there is a parameter I can use to set the max memory Chrome can use, we can adapt base on the VM capabilities.

Thanks for your help :slight_smile:
Phung

Hi,
I see you are attempting to restrict the container memory by using -m 4G (--memory) option.
From the output of top I can see you have few chrome threads, using not much physical memory (see the MEM column) but a lot of shared memory (see the VIRT column).
My guess here is, it is using a lot of swapped memory (you can inspect also your host with free -h to check what is available vs used)
So, you may want to limit also the swap usage for your container by using in addition the --memory-swap="" option.
See: https://docs.docker.com/engine/reference/run/#user-memory-constraints

Also, I noticed you are running your container as root user, not sure how good is that …
As for limiting the JVM usage … I think it is possible also by tweaking a certain file (not sure if KRE has a katalon.ini, I must check) but I doubt it will help for the current case, since it will limit only the memory for KRE himself but not for the chrome threads.
You may have to build your own docker image using the katalon one as a source and replace the file needed with your own version to achieve that.

See also here for info how to read the top output:
https://www.baeldung.com/linux/top-command#:~:text=VIRT%20

3.1. Memory Headers

The headers that are used to summarize various parameters related to memory are described below:

  • VIRT (Virtual Memory Size in KiB): Depicts the total amount of virtual memory used by the task. Virtual memory includes all code, data, and shared libraries. It also includes pages that have been swapped out and pages that have been mapped but not used
  • RES (Resident Memory Size in KiB): Stands for a subset of the virtual memory space ( VIRT ) representing the non-swapped physical memory a task is currently using
  • SHR (Shared Memory Size in KiB): Stands for a subset of resident memory ( RES ) that may be used by other processes
  • %CPU (CPU Usage): Stands for the task’s share of the elapsed CPU time since the last screen update, expressed as a percentage of total CPU time. A value greater than 100% can be reported for a multi-threaded process when top is not running in Threads Mode
  • %MEM (Memory Usage -RES): The task’s current share of available physical memory
1 Like

Side note: make sure you are also killing and removing the running container in your jenkins pipeline once the test ends.
Leftover containers tend to exhaust a lot of resources usage in time.

Thanks @anon46315158 for these precious information :slight_smile:
I am going to try to limit the swap memory and see where it goes.

About running the docker container as root, I know it’s not a good practice. But I run into an error when trying to run it with the current user (Jenkins user). I guess I need to come back on this once the memory problem is solved.

Yes, our Jenkins pipeline is removing all docker container at the end of each run :slight_smile:

Thanks
Phung

as far as i know, if you remove the -u root from the pipeline script, inside the container everything should run as a certain katalon user.
but honestly, since i don’t have a license to actually run KRE, my only way to debug such is to read the commit logs and attempt to start the docker container in a controlled environment.

Keep debugging!