Katalon + GitHub Actions Integration

Hello everyone!

Our company is in the process of evaluation of Katalon.

Here is what we want to do:

  1. Use Github Actions to run our pipeline actions
  2. Use the infrastructure Github provides to run the tests (this is either a docker container or a Virtual Machine)
  3. Trigger the tests when code is merged into a given branch.

Please note that we have no control over the infrastructure, and we can only setup the containers as described here: Features • GitHub Actions · GitHub.

It looks like this GitHub - katalon-studio/katalon-studio-github-action: Katalon Studio Github Action might be used for Katalon+GitHub integration.

Could someone please look into Katalon + GitHub Actions Integration and let me know what would be the best approach for it, the correct licensing package and guidelines on the POC?

Thanks and Regards,

Gregory

Shout out @ThanhTo

Hi,

While trying to use https://github.com/katalon-studio/katalon-studio-github-action (with some changes) I got the following even though I used API Key I got for the trial version:

2020-05-19T13:17:37.3897096Z Activating…
2020-05-19T13:17:37.3923581Z Start activating offline…
2020-05-19T13:17:37.3935094Z Search for valid offline licenses in folder: /root/.katalon/license
2020-05-19T13:17:37.3939682Z The number of valid offline licenses: 0
2020-05-19T13:17:37.3946269Z Offline activation failed.
2020-05-19T13:17:37.3948680Z Start activating online…
2020-05-19T13:17:38.0055032Z Katalon TestOps: Unexpected response, URL: https://analytics.katalon.com/oauth/token, Status: 400, Response: {“error”:“invalid_grant”,“error_description”:“Bad credentials”}
2020-05-19T13:17:38.0162241Z Bad credentials
2020-05-19T13:17:38.0169903Z Online activation for console mode failed.
2020-05-19T13:17:38.0170351Z Activation failed. Please make sure you are using a valid license.

Any idea what can be done? What are offline licenses? Can I get those and use for Katalon + GitHub Actions Integration? If yes, which one should I get? KSE Offline Licencse? KRE Ofline License? Thanks.

Hi,

If nobody here can help, maybe someone knows who I should contact to get help? We are kind of running out of time. If Katalon + GitHub Actions Integration cannot be done, our company will use a different tool for test automation. Thanks.

@gregory2
Please share the full execution log. Did you put KATALON_API_KEY in Github Action secrets?

Yes, I put KATALON_API_KEY in Github Action secrets.

It looks like GitHub - katalon-studio/katalon-studio-github-action: Katalon Studio Github Action is not working as it should.

When I created an action as per its Example usage, I got the following error while running this action:

When I commented out this line


I got the same error I posted above:

After I have modified the action like this:

, I can see Katalon running but with another error:

It looks like Test suite ‘"Test’ not found might be the result of
‘-testSuitePath="Test’ 'Suites/TS_RegressionTest" in xvfb (even though in command-line testSuitePath=“Test Suites/TS_RegressionTest” is correct:

Here is the full execution log of Get and run action:

2020-05-23T17:22:22.6940925Z ##[group]Run katalon-studio/katalon-studio-github-action@0.9
2020-05-23T17:22:22.6941065Z with:
2020-05-23T17:22:22.6941584Z ks_api_key: ***
2020-05-23T17:22:22.6941714Z ks_command: katalon-execute.sh -browserType=“Chrome” -retry=0 -statusDelay=15 -testSuitePath=“Test Suites/TS_RegressionTest”
2020-05-23T17:22:22.6941842Z ##[endgroup]
2020-05-23T17:22:22.6962805Z ##[command]/usr/bin/docker run --name be76db79b1d033a75d410d88032da19f87ca80_38db59 --label be76db --workdir /github/workspace --rm -e INPUT_KS_API_KEY -e INPUT_KS_COMMAND -e HOME -e GITHUB_JOB -e GITHUB_REF -e GITHUB_SHA -e GITHUB_REPOSITORY -e GITHUB_REPOSITORY_OWNER -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_ACTOR -e GITHUB_WORKFLOW -e GITHUB_HEAD_REF -e GITHUB_BASE_REF -e GITHUB_EVENT_NAME -e GITHUB_URL -e GITHUB_API_URL -e GITHUB_GRAPHQL_URL -e GITHUB_WORKSPACE -e GITHUB_ACTION -e GITHUB_EVENT_PATH -e RUNNER_OS -e RUNNER_TOOL_CACHE -e RUNNER_TEMP -e RUNNER_WORKSPACE -e ACTIONS_RUNTIME_URL -e ACTIONS_RUNTIME_TOKEN -e ACTIONS_CACHE_URL -e GITHUB_ACTIONS=true -e CI=true -v “/var/run/docker.sock”:"/var/run/docker.sock" -v “/home/runner/work/_temp/_github_home”:"/github/home" -v “/home/runner/work/_temp/_github_workflow”:"/github/workflow" -v “/home/runner/work/Katalon/Katalon”:"/github/workspace" be76db:79b1d033a75d410d88032da19f87ca80 “katalon-execute.sh -browserType=“Chrome” -retry=0 -statusDelay=15 -testSuitePath=“Test Suites/TS_RegressionTest”” ""
2020-05-23T17:22:23.2232938Z + echo ‘Please visit https://github.com/katalon-studio/docker-images for new usage. The use of katalon-execute.sh has been deprecated.’
2020-05-23T17:22:23.2233728Z + echo ‘Starting Katalon Studio’
2020-05-23T17:22:23.2233968Z + cat /katalon/version
2020-05-23T17:22:23.2234178Z ++ pwd
2020-05-23T17:22:23.2234443Z + current_dir=/github/workspace
2020-05-23T17:22:23.2234616Z + workspace_dir=/tmp/katalon_execute/workspace
2020-05-23T17:22:23.2234986Z + mkdir -p /tmp/katalon_execute/workspace
2020-05-23T17:22:23.2253763Z + chmod -R 777 /tmp/katalon_execute/workspace
2020-05-23T17:22:23.2255595Z Please visit https://github.com/katalon-studio/docker-images for new usage. The use of katalon-execute.sh has been deprecated.
2020-05-23T17:22:23.2256111Z Starting Katalon Studio
2020-05-23T17:22:23.2277740Z + source_dir=/katalon/katalon/source
2020-05-23T17:22:23.2278345Z + ‘[’ -d /katalon/katalon/source ‘]’
2020-05-23T17:22:23.2278879Z ++ pwd
2020-05-23T17:22:23.2279026Z + source_dir=/github/workspace
2020-05-23T17:22:23.2279311Z + project_dir=/tmp/katalon_execute/project
2020-05-23T17:22:23.2279704Z + mkdir -p /tmp/katalon_execute/project
2020-05-23T17:22:23.2279876Z Mozilla Firefox 75.0
2020-05-23T17:22:23.2286659Z Google Chrome 81.0.4044.129
2020-05-23T17:22:23.2295245Z Katalon Studio
2020-05-23T17:22:23.2329618Z + cp -r /github/workspace/Include /github/workspace/Katalon.prj /github/workspace/Profiles /github/workspace/README.md /github/workspace/Scripts ‘/github/workspace/Test Cases’ ‘/github/workspace/Test Suites’ /github/workspace/console.properties /github/workspace/settings /tmp/katalon_execute/project
2020-05-23T17:22:23.2330174Z ++ find /tmp/katalon_execute/project -name '
.prj’ -print -quit
2020-05-23T17:22:23.2330457Z + project_prj_file=/tmp/katalon_execute/project/Katalon.prj
2020-05-23T17:22:23.2330743Z + cd /tmp/katalon_execute/project
2020-05-23T17:22:23.2331164Z + ‘[’ -f /tmp/katalon_execute/project/build.gradle ‘]’
2020-05-23T17:22:23.2331378Z + cd /github/workspace
2020-05-23T17:22:23.2331731Z + touch /tmp/katalon_execute/project/.classpath
2020-05-23T17:22:23.2348750Z + chmod -R 777 /tmp/katalon_execute/project
2020-05-23T17:22:23.2405199Z + rm -rf /tmp/katalon_execute/project/bin
2020-05-23T17:22:23.2405700Z + rm -rf /tmp/katalon_execute/project/Libs
2020-05-23T17:22:23.2406389Z + ls -la /tmp/katalon_execute/project
2020-05-23T17:22:23.2406559Z + report_dir=/katalon/katalon/report
2020-05-23T17:22:23.2406822Z + ‘[’ -d /katalon/katalon/report ‘]’
2020-05-23T17:22:23.2407007Z + report_dir=/github/workspace/report
2020-05-23T17:22:23.2407297Z + mkdir -p /github/workspace/report
2020-05-23T17:22:23.2407679Z + args=("$KATALON_KATALON_INSTALL_DIR/katalonc" “$@”)
2020-05-23T17:22:23.2408122Z + args+=("-runMode=console")
2020-05-23T17:22:23.2408419Z + args+=("-reportFolder=$report_dir")
2020-05-23T17:22:23.2408671Z + args+=("-projectPath=$project_prj_file")
2020-05-23T17:22:23.2408863Z + cd /tmp/katalon_execute/workspace
2020-05-23T17:22:23.2410146Z + xvfb-run -a -e /dev/stdout -s ‘-screen 0 1024x768x24’ /opt/katalonstudio/katalonc katalon-execute.sh ‘-browserType=“Chrome”’ -retry=0 -statusDelay=15 ‘-testSuitePath="Test’ ‘Suites/TS_RegressionTest"’ -apiKey=
* -runMode=console -reportFolder=/github/workspace/report -projectPath=/tmp/katalon_execute/project/Katalon.prj
2020-05-23T17:22:23.2410386Z total 44
2020-05-23T17:22:23.2410540Z drwxrwxrwx 8 root root 4096 May 23 17:22 .
2020-05-23T17:22:23.2410860Z drwxr-xr-x 4 root root 4096 May 23 17:22 …
2020-05-23T17:22:23.2411687Z -rwxrwxrwx 1 root root 0 May 23 17:22 .classpath
2020-05-23T17:22:23.2411857Z drwxrwxrwx 3 root root 4096 May 23 17:22 Include
2020-05-23T17:22:23.2412189Z -rwxrwxrwx 1 root root 902 May 23 17:22 Katalon.prj
2020-05-23T17:22:23.2412305Z drwxrwxrwx 2 root root 4096 May 23 17:22 Profiles
2020-05-23T17:22:23.2412696Z -rwxrwxrwx 1 root root 9 May 23 17:22 README.md
2020-05-23T17:22:23.2412871Z drwxrwxrwx 3 root root 4096 May 23 17:22 Scripts
2020-05-23T17:22:23.2413026Z drwxrwxrwx 2 root root 4096 May 23 17:22 Test Cases
2020-05-23T17:22:23.2413179Z drwxrwxrwx 2 root root 4096 May 23 17:22 Test Suites
2020-05-23T17:22:23.2413507Z -rwxrwxrwx 1 root root 165 May 23 17:22 console.properties
2020-05-23T17:22:23.2413757Z drwxrwxrwx 4 root root 4096 May 23 17:22 settings
2020-05-23T17:22:26.7827677Z Katalon workspace folder is set to default location: /tmp/session-27f18c08
2020-05-23T17:22:26.8944455Z Starting Groovy-Eclipse compiler resolver. Specified compiler level: unspecified
2020-05-23T17:22:26.8949059Z 176 2.4.7.xx-201611170128-e46 = ACTIVE
2020-05-23T17:22:27.0584350Z
2020-05-23T17:22:27.0584641Z
2020-05-23T17:22:27.0658999Z INFO: Katalon Version: 7.4.2
2020-05-23T17:22:27.0663895Z INFO: Command-line arguments: katalon-execute.sh -browserType=“Chrome” -retry=0 -statusDelay=15 -testSuitePath=“Test Suites/TS_RegressionTest” -apiKey=****** -runMode=console -reportFolder=/github/workspace/report -projectPath=/tmp/katalon_execute/project/Katalon.prj
2020-05-23T17:22:27.0667865Z INFO: User working dir: /tmp/katalon_execute/workspace
2020-05-23T17:22:27.0672161Z INFO: Error log: /tmp/session-27f18c08/.metadata/.log
2020-05-23T17:22:27.0695170Z INFO: Katalon TestOps server URL: https://analytics.katalon.com
2020-05-23T17:22:27.0703177Z INFO: Katalon Store server URL: https://store.katalon.com
2020-05-23T17:22:27.0707597Z INFO: User home: /root
2020-05-23T17:22:27.0711357Z INFO: Java vendor: Private Build
2020-05-23T17:22:27.0714086Z INFO: Java version: 1.8.0_252
2020-05-23T17:22:27.0717969Z INFO: Local OS: Linux 64bit
2020-05-23T17:22:27.1150904Z INFO: CPU load: 1%
2020-05-23T17:22:27.1158025Z INFO: Total memory: 6927 MB
2020-05-23T17:22:27.1163296Z INFO: Free memory: 1164 MB
2020-05-23T17:22:27.1206694Z Start getting machine ID on Linux
2020-05-23T17:22:27.1348414Z End getting machine ID on Linux e6264e1cbff383cf264352375ea7d9fa
2020-05-23T17:22:27.1352187Z Start appending additional signatures
2020-05-23T17:22:27.1369675Z hardwareSerialNumber
2020-05-23T17:22:27.1372616Z osDependentUsername root
2020-05-23T17:22:27.1375629Z End appending additional signatures e6264e1cbff383cf264352375ea7d9fa__root
2020-05-23T17:22:27.1379797Z INFO: Machine ID: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
2020-05-23T17:22:27.1382916Z
2020-05-23T17:22:27.1383012Z
2020-05-23T17:22:27.1399260Z Activating…
2020-05-23T17:22:27.1423361Z Start activating offline…
2020-05-23T17:22:27.1435848Z Search for valid offline licenses in folder: /root/.katalon/license
2020-05-23T17:22:27.1439487Z The number of valid offline licenses: 0
2020-05-23T17:22:27.1442871Z Offline activation failed.
2020-05-23T17:22:27.1442990Z Start activating online…
2020-05-23T17:22:28.6961747Z Cleaning up workspace
2020-05-23T17:22:28.6965662Z Opening project file: /tmp/katalon_execute/project/Katalon.prj
2020-05-23T17:22:31.0915465Z Generating global variables…
2020-05-23T17:22:32.4587077Z Parsing custom keywords…
2020-05-23T17:22:33.1012475Z Project ‘Katalon’ opened
2020-05-23T17:22:33.1126823Z Start reloading plugins…
2020-05-23T17:22:33.4103976Z Katalon version: 7.4.2
2020-05-23T17:22:33.4104457Z Plugin info URL: https://store.katalon.com/api/products/ks?appVersion=7.4.2&appType=ENGINE&licenseType=TRIAL
2020-05-23T17:22:33.4175646Z Plugin info: {
2020-05-23T17:22:33.4175802Z “expired”: false,
2020-05-23T17:22:33.4175954Z “productId”: 59,
2020-05-23T17:22:33.4176128Z “name”: “Basic Report”,
2020-05-23T17:22:33.4176276Z “id”: 174206
2020-05-23T17:22:33.4176418Z }
2020-05-23T17:22:35.0776447Z Plugin name: Basic Report. Is online: true. Plugin location: /opt/katalonstudio/configuration/plugin/install/174206/1.0.7/katalon-studio-report-plugin.jar.
2020-05-23T17:22:35.7257217Z Plugin “Basic Report” has been installed.
2020-05-23T17:22:35.7301822Z Start check license task
2020-05-23T17:22:35.7508271Z com.kms.katalon.execution.exception.ExecutionException: com.kms.katalon.execution.exception.InvalidConsoleArgumentException: Test suite ‘"Test’ not found.
2020-05-23T17:22:35.7508563Z at com.kms.katalon.execution.console.entity.TestSuiteLauncherOptionParser.getConsoleLauncher(TestSuiteLauncherOptionParser.java:240)
2020-05-23T17:22:35.7508784Z at com.kms.katalon.execution.console.ConsoleExecutor.execute(ConsoleExecutor.java:74)
2020-05-23T17:22:35.7509124Z at com.kms.katalon.execution.console.ConsoleMain.launch(ConsoleMain.java:320)
2020-05-23T17:22:35.7509298Z at com.kms.katalon.console.application.Application.runConsole(Application.java:71)
2020-05-23T17:22:35.7509492Z at com.kms.katalon.core.application.Application.runConsole(Application.java:99)
2020-05-23T17:22:35.7509662Z at com.kms.katalon.core.application.Application.start(Application.java:74)
2020-05-23T17:22:35.7509838Z at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:196)
2020-05-23T17:22:35.7509974Z at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:134)
2020-05-23T17:22:35.7510294Z at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:104)
2020-05-23T17:22:35.7510471Z at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:388)
2020-05-23T17:22:35.7510629Z at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:243)
2020-05-23T17:22:35.7510786Z at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2020-05-23T17:22:35.7510948Z at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2020-05-23T17:22:35.7511110Z at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2020-05-23T17:22:35.7511219Z at java.lang.reflect.Method.invoke(Method.java:498)
2020-05-23T17:22:35.7511375Z at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:673)
2020-05-23T17:22:35.7511530Z at org.eclipse.equinox.launcher.Main.basicRun(Main.java:610)
2020-05-23T17:22:35.7511693Z at org.eclipse.equinox.launcher.Main.run(Main.java:1519)
2020-05-23T17:22:35.7511857Z at org.eclipse.equinox.launcher.Main.main(Main.java:1492)
2020-05-23T17:22:35.7512255Z Caused by: com.kms.katalon.execution.exception.InvalidConsoleArgumentException: Test suite ‘"Test’ not found.
2020-05-23T17:22:35.7512459Z at com.kms.katalon.execution.console.entity.TestSuiteLauncherOptionParser.throwInvalidTestSuiteIdException(TestSuiteLauncherOptionParser.java:335)
2020-05-23T17:22:35.7512608Z at com.kms.katalon.execution.console.entity.TestSuiteLauncherOptionParser.getTestSuite(TestSuiteLauncherOptionParser.java:330)
2020-05-23T17:22:35.7512783Z at com.kms.katalon.execution.console.entity.TestSuiteLauncherOptionParser.buildRunConfig(TestSuiteLauncherOptionParser.java:258)
2020-05-23T17:22:35.7512959Z at com.kms.katalon.execution.console.entity.TestSuiteLauncherOptionParser.getConsoleLauncher(TestSuiteLauncherOptionParser.java:237)
2020-05-23T17:22:35.7513202Z … 18 more
2020-05-23T17:22:35.7513297Z
2020-05-23T17:22:35.7513382Z All launchers terminated
2020-05-23T17:22:36.5915706Z + ret_code=2
2020-05-23T17:22:36.5922146Z ++ id -u
2020-05-23T17:22:36.5934707Z ++ id -g
2020-05-23T17:22:36.5944222Z + chown -R 0:0 /github/workspace/report
2020-05-23T17:22:36.5958563Z + chmod -R 777 /github/workspace/report
2020-05-23T17:22:36.5966616Z + ls /github/workspace/report
2020-05-23T17:22:36.5977320Z + cd /github/workspace
2020-05-23T17:22:36.5977843Z + exit 2

By the way, I’m wondering what kind of genius has come up with using spaces in “Test Suits” etc.? Everyone knows that using spaces in directories names it’s a recipe for disaster when it should be integrated with something else. It could’ve been named just “Suites” if Test_Suites is not good.

@gregory2
Please change -testSuitePath to:

Test Suites/TS_RegressionTest

not

Test’ ‘Suites/TS_RegressionTest

@duyluong,

In the action itself I’ve always used testSuitePath=“Test Suites/TS_RegressionTest”’, as it can be seen from the screenshots posted above and the action source posted below.

The problem is that somehow https://github.com/katalon-studio/katalon-studio-github-action is working in such a way that ’-testSuitePath=“Test’ 'Suites/TS_RegressionTest” is shown in xvfb

( even though, in command-line, testSuitePath=“Test Suites/TS_RegressionTest” is shown correctly ), as it can be seen from the screenshots and the full log posted above:

2020-05-23T17:22:23.2410146Z + xvfb-run -a -e /dev/stdout -s ‘-screen 0 1024x768x24’ /opt/katalonstudio/katalonc katalon-execute.sh ‘-browserType=“Chrome”’ -retry=0 -statusDelay=15 ‘-testSuitePath=“Test’ ‘Suites/TS_RegressionTest”’ -apiKey= * -runMode=console -reportFolder=/github/workspace/report -projectPath=/tmp/katalon_execute/project/Katalon.prj

2020-05-23T17:22:27.0663895Z INFO: Command-line arguments: katalon-execute.sh -browserType=“Chrome” -retry=0 -statusDelay=15 -testSuitePath=“Test Suites/TS_RegressionTest” -apiKey=****** -runMode=console -reportFolder=/github/workspace/report -projectPath=/tmp/katalon_execute/project/Katalon.prj

This is a basic workflow to help you get started with Actions

name: CI

on: [push]

jobs:
katalon_test_job:
runs-on: ubuntu-latest
name: Run katalon Studio Test CLI
steps:
# To use this repository’s private action, you must check out the repository
- name: Checkout
uses: actions/checkout@v2
#- uses: ./ # Uses an action in the root directory COMMENTED OUT BY ME BECAUSE OF THE ERROR g2
- name: Get and run action
#env:
#TEST_SUITE: “Test Suites/TS_RegressionTest”
uses: katalon-studio/katalon-studio-github-action@0.9
with:
# katalon_api_key: ${{ secrets.KATALON_API_KEY }}
# Katalon_command: ‘katalon-execute.sh -browserType=“Chrome” -retry=0 -statusDelay=15 -testSuitePath=“Test Suites/TS_RegressionTest”’
# : MODIFIED BY ME BECAUSE OF THE ERRORS :
ks_api_key: ${{ secrets.KATALON_API_KEY }}
ks_command: ‘katalon-execute.sh -browserType=“Chrome” -retry=0 -statusDelay=15 -testSuitePath=“Test Suites/TS_RegressionTest”’