Test execution issues while running multiple test suites using a docker container

Archive
,
1

Good afternoon,

For a while now we’ve been using the Docker Container from Docker Hub as we execute our tests in Jenkins. Here is an example of the script we have Jenkins execute (
#!/usr/bin/env bash

version="6.1.1"
environment="Test"
TEST_SUITES=(Test\ Suites/TF_TA\ Test\ Suite\ Collection)

testSuite=${TEST_SUITES}
echo "**********************************************************"
echo "Running test: ${testSuite}"

docker run \
	--net=host \
    --rm \
    -v "$(pwd)":/katalon/katalon/source \
    -e AUTOMATION_PASSWORD \
    -e DB_PASSWORD \
    -e AUTOMATION_LOGIN \
    katalonstudio/katalon:${version} \
    katalon-execute.sh \
    -browserType='Chrome' \
    -retry=0 -statusDelay=15 \
    -testSuiteCollectionPath="$testSuite" \
    -executionProfile="$environment"         
echo "**********************************************************"
echo "Done!"

Each test suite collection contains a number of test suites in them the lowest number is 2 the highest is 6.
I have the test suite collection configured to execute in parallel with 3 test suites executing.

For some reason when I do this (even if there are less than three test suites in the collection) I experience timeouts when trying to connect to the environment under test or the browser just crashes. We thought it was a system memory / CPU issue of our Jenkins server (we are using the same machine to host both Jenkins and the docker container that will run the Katalon Studio test execution). We recently improved that systems memory and CPU but were still having the issue (went from an XL to a 2XL in AWS parlance). When I reduce the concurrency down to 2 we have a greater success in test execution, but ideally I’d like to be able to run more.

I am curious if anyone else has run into a situation like this and resolved it. I would assume that running 3 concurrent test suites wouldn’t be that big of an impact but as our test collection grows I could see us wanting even more concurrent test suites running.

Maybe there is a configuration value that can be provided during the docker execution to allow it to use more memory/cpu but as far as we can tell it is using what it can when being called.

Thanks,
Greg