ExecutionException: Unable to execute test suite

Hello,

I am currently using Katalon Version 8.2.5, and I’m experiencing an error when using the Katalon Runtime Engine through Jenkins. I am able to execute my test collections manually through Katalon Studio, but not in Jenkins. I believe it may have to do with the execution profiles we just created in Katalon Studio.

The command I am using to execute my test suite collection:

docker run --shm-size=1048m -t --rm -v ${PWD}:${WORKSPACE} katalonstudio/katalon katalonc.sh -projectPath='${WORKSPACE}/workspace/${PROJECT_NAME_ENV}/${KATALON_REPO_NAME}.prj' -reportFolder='${WORKSPACE}/workspace/${PROJECT_NAME_ENV}/${TEST_SUITE}/' -browserType='Chrome (headless)' -noSplash -runMode='console' -retry=0 -statusDelay=15 -testSuiteCollectionPath=\"${'Test Suites/' + TEST_SUITE}\" -executionProfile='${ENVIRONMENT}' -apiKey=******** --config -webui.autoUpdateDrivers=true

The error that we are receiving is:

com.kms.katalon.execution.exception.ExecutionException: Unable to execute test suite ‘Test Suites/Co-Pilot Header Regression’
at com.kms.katalon.execution.launcher.TestSuiteCollectionConsoleLauncher.buildLauncher(TestSuiteCollectionConsoleLauncher.java:258)
at com.kms.katalon.execution.launcher.TestSuiteCollectionConsoleLauncher.buildSubLaunchers(TestSuiteCollectionConsoleLauncher.java:150)
at com.kms.katalon.execution.launcher.TestSuiteCollectionConsoleLauncher.newInstance(TestSuiteCollectionConsoleLauncher.java:88)
at com.kms.katalon.execution.console.entity.TestSuiteCollectionLauncherOptionParser.getConsoleLauncher(TestSuiteCollectionLauncherOptionParser.java:140)
at com.kms.katalon.execution.console.ConsoleExecutor.execute(ConsoleExecutor.java:74)
at com.kms.katalon.execution.console.ConsoleMain.launch(ConsoleMain.java:528)
at com.kms.katalon.console.application.ConsoleApplicationStarter.start(ConsoleApplicationStarter.java:37)
at com.kms.katalon.application.Application.runConsole(Application.java:125)
at com.kms.katalon.application.Application.start(Application.java:88)
at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:203)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:137)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:107)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:401)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:255)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:657)
at org.eclipse.equinox.launcher.Main.basicRun(Main.java:594)
at org.eclipse.equinox.launcher.Main.run(Main.java:1447)
at org.eclipse.equinox.launcher.Main.main(Main.java:1420)

These are interpolated strings which need to be wrapped in pairs of double quotes:

"${PWD}:${WORKSPACE}"

Nope!
Are not interpolated.
In linux shell terms, we may have to speak about variables expansion, which is a different animal and may have sometime subtle behavior.

The comment made by @Russ_Thomas it is still somehow valid …
More insights:

https://tldp.org/LDP/abs/html/parameter-substitution.html

1 Like

I tried the following one line in the bash shell on Mac:

$ echo docker run --shm-size=1048m -t --rm -v ${PWD}:${WORKSPACE} katalonstudio/katalon katalonc.sh -projectPath='${WORKSPACE}/workspace/${PROJECT_NAME_ENV}/${KATALON_REPO_NAME}.prj' -reportFolder='${WORKSPACE}/workspace/${PROJECT_NAME_ENV}/${TEST_SUITE}/' -browserType='Chrome (headless)' -noSplash -runMode='console' -retry=0 -statusDelay=15 -testSuiteCollectionPath=\"${'Test Suites/' + TEST_SUITE}\" -executionProfile='${ENVIRONMENT}' -apiKey=******** --config -webui.autoUpdateDrivers=true

Bash replied with the following message:

-bash: -testSuiteCollectionPath=\"${'Test Suites/' + TEST_SUITE}\": bad substitution

This must be the reason why @catherine.larchevequ got an error.

1 Like

I suppose the intention is:

"Test\ Suites/${TEST_SUITE}"

I don’t believe the problem stems from syntax errors, although the recommendations are appreciated! The command has been executed successfully in Jenkins a few hundred times now. The error I received started happening after another developer created execution profiles in Katalon Studio. I know this was the cause of the error for others but no one has posted a solution.

See:

@jsoule @valeria.perchiazzi

So … did you had this before in your command?

-executionProfile='${ENVIRONMENT}'

If the profiles were just created, seems like the developer changed also the command used in Jenkins …

Does the ${ENVIRONMENT} variable have a valid value?

Note also, from your command:

-testSuiteCollectionPath=\blah

You are running a Collection. A collection, as far as I know, is a collection of suites, each tied to a specific profile (and a given browser).
I don’t think you can override the Profile in use from the command line (I never tried and I cannot test this since I don’t have a KRE licence) in such case.
The executionProfile override from CLI should work fine when running a test suite.

LE: from:
https://docs.katalon.com/katalon-studio/docs/console-mode-execution.html#general-options

From version 7.6+, you can use this option in Test Suite Collection execution. The specified execution profile is applied to all test suites in that collection.

So, this should work also … perhaps is a regression? (provided the ENVIRONMENT variable is valid)

Oops, you are correct. I did change one thing in the command: the execution profile flag. It evaluates to develop, quality, or production depending on the environment (confirmed in the Jenkins console). We have those corresponding profiles set up in Katalon Studio. As per the docs:

-executionProfile: From version 7.6+, you can use this option in Test Suite Collection execution. The specified execution profile is applied to all test suites in that collection.

All our test suites in the execution collection should have the same execution profile applied to them. Could you elaborate more on what a regression is? @anon46315158

Well …a regression is an old bug re-introduced in a new version (or a feature known to work in previous versions stops to work in a new release).
It can happens sometime, when fixing a bug or adding a new feature, by mistake the developer can push an old code (or add new code conflicting with some features)

So, you can try some older versions of Katalon to see if is still failing …