Hello everyone!
Our company is in the process of evaluation of Katalon.
Here is what we want to do:
- Use Github Actions to run our pipeline actions
- Use the infrastructure Github provides to run the tests (this is either a docker container or a Virtual Machine)
- Trigger the tests when code is merged into a given branch.
Please note that we have no control over the infrastructure, and we can only setup the containers as described here: Features • GitHub Actions · GitHub.
It looks like this GitHub - katalon-studio/katalon-studio-github-action: Katalon Studio Github Action might be used for Katalon+GitHub integration.
Could someone please look into Katalon + GitHub Actions Integration and let me know what would be the best approach for it, the correct licensing package and guidelines on the POC?
Thanks and Regards,
Gregory
Hi,
While trying to use https://github.com/katalon-studio/katalon-studio-github-action (with some changes) I got the following even though I used API Key I got for the trial version:
2020-05-19T13:17:37.3897096Z Activating…
2020-05-19T13:17:37.3923581Z Start activating offline…
2020-05-19T13:17:37.3935094Z Search for valid offline licenses in folder: /root/.katalon/license
2020-05-19T13:17:37.3939682Z The number of valid offline licenses: 0
2020-05-19T13:17:37.3946269Z Offline activation failed.
2020-05-19T13:17:37.3948680Z Start activating online…
2020-05-19T13:17:38.0055032Z Katalon TestOps: Unexpected response, URL: https://analytics.katalon.com/oauth/token, Status: 400, Response: {“error”:“invalid_grant”,“error_description”:“Bad credentials”}
2020-05-19T13:17:38.0162241Z Bad credentials
2020-05-19T13:17:38.0169903Z Online activation for console mode failed.
2020-05-19T13:17:38.0170351Z Activation failed. Please make sure you are using a valid license.
Any idea what can be done? What are offline licenses? Can I get those and use for Katalon + GitHub Actions Integration? If yes, which one should I get? KSE Offline Licencse? KRE Ofline License? Thanks.
Hi,
If nobody here can help, maybe someone knows who I should contact to get help? We are kind of running out of time. If Katalon + GitHub Actions Integration cannot be done, our company will use a different tool for test automation. Thanks.
@gregory2
Please share the full execution log. Did you put KATALON_API_KEY in Github Action secrets?
Yes, I put KATALON_API_KEY in Github Action secrets.
It looks like GitHub - katalon-studio/katalon-studio-github-action: Katalon Studio Github Action is not working as it should.
When I created an action as per its Example usage, I got the following error while running this action:
When I commented out this line
I got the same error I posted above:
After I have modified the action like this:
, I can see Katalon running but with another error:
It looks like Test suite ‘"Test’ not found might be the result of
‘-testSuitePath="Test’ 'Suites/TS_RegressionTest" in xvfb (even though in command-line testSuitePath=“Test Suites/TS_RegressionTest” is correct:
Here is the full execution log of Get and run action:
2020-05-23T17:22:22.6940925Z ##[group]Run katalon-studio/katalon-studio-github-action@0.9
2020-05-23T17:22:22.6941065Z with:
2020-05-23T17:22:22.6941584Z ks_api_key: ***
2020-05-23T17:22:22.6941714Z ks_command: katalon-execute.sh -browserType=“Chrome” -retry=0 -statusDelay=15 -testSuitePath=“Test Suites/TS_RegressionTest”
2020-05-23T17:22:22.6941842Z ##[endgroup]
2020-05-23T17:22:22.6962805Z ##[command]/usr/bin/docker run --name be76db79b1d033a75d410d88032da19f87ca80_38db59 --label be76db --workdir /github/workspace --rm -e INPUT_KS_API_KEY -e INPUT_KS_COMMAND -e HOME -e GITHUB_JOB -e GITHUB_REF -e GITHUB_SHA -e GITHUB_REPOSITORY -e GITHUB_REPOSITORY_OWNER -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_ACTOR -e GITHUB_WORKFLOW -e GITHUB_HEAD_REF -e GITHUB_BASE_REF -e GITHUB_EVENT_NAME -e GITHUB_URL -e GITHUB_API_URL -e GITHUB_GRAPHQL_URL -e GITHUB_WORKSPACE -e GITHUB_ACTION -e GITHUB_EVENT_PATH -e RUNNER_OS -e RUNNER_TOOL_CACHE -e RUNNER_TEMP -e RUNNER_WORKSPACE -e ACTIONS_RUNTIME_URL -e ACTIONS_RUNTIME_TOKEN -e ACTIONS_CACHE_URL -e GITHUB_ACTIONS=true -e CI=true -v “/var/run/docker.sock”:"/var/run/docker.sock" -v “/home/runner/work/_temp/_github_home”:"/github/home" -v “/home/runner/work/_temp/_github_workflow”:"/github/workflow" -v “/home/runner/work/Katalon/Katalon”:"/github/workspace" be76db:79b1d033a75d410d88032da19f87ca80 “katalon-execute.sh -browserType=“Chrome” -retry=0 -statusDelay=15 -testSuitePath=“Test Suites/TS_RegressionTest”” ""
2020-05-23T17:22:23.2232938Z + echo ‘Please visit https://github.com/katalon-studio/docker-images for new usage. The use of katalon-execute.sh has been deprecated.’
2020-05-23T17:22:23.2233728Z + echo ‘Starting Katalon Studio’
2020-05-23T17:22:23.2233968Z + cat /katalon/version
2020-05-23T17:22:23.2234178Z ++ pwd
2020-05-23T17:22:23.2234443Z + current_dir=/github/workspace
2020-05-23T17:22:23.2234616Z + workspace_dir=/tmp/katalon_execute/workspace
2020-05-23T17:22:23.2234986Z + mkdir -p /tmp/katalon_execute/workspace
2020-05-23T17:22:23.2253763Z + chmod -R 777 /tmp/katalon_execute/workspace
2020-05-23T17:22:23.2255595Z Please visit https://github.com/katalon-studio/docker-images for new usage. The use of katalon-execute.sh has been deprecated.
2020-05-23T17:22:23.2256111Z Starting Katalon Studio
2020-05-23T17:22:23.2277740Z + source_dir=/katalon/katalon/source
2020-05-23T17:22:23.2278345Z + ‘[’ -d /katalon/katalon/source ‘]’
2020-05-23T17:22:23.2278879Z ++ pwd
2020-05-23T17:22:23.2279026Z + source_dir=/github/workspace
2020-05-23T17:22:23.2279311Z + project_dir=/tmp/katalon_execute/project
2020-05-23T17:22:23.2279704Z + mkdir -p /tmp/katalon_execute/project
2020-05-23T17:22:23.2279876Z Mozilla Firefox 75.0
2020-05-23T17:22:23.2286659Z Google Chrome 81.0.4044.129
2020-05-23T17:22:23.2295245Z Katalon Studio
2020-05-23T17:22:23.2329618Z + cp -r /github/workspace/Include /github/workspace/Katalon.prj /github/workspace/Profiles /github/workspace/README.md /github/workspace/Scripts ‘/github/workspace/Test Cases’ ‘/github/workspace/Test Suites’ /github/workspace/console.properties /github/workspace/settings /tmp/katalon_execute/project
2020-05-23T17:22:23.2330174Z ++ find /tmp/katalon_execute/project -name '.prj’ -print -quit
2020-05-23T17:22:23.2330457Z + project_prj_file=/tmp/katalon_execute/project/Katalon.prj
2020-05-23T17:22:23.2330743Z + cd /tmp/katalon_execute/project
2020-05-23T17:22:23.2331164Z + ‘[’ -f /tmp/katalon_execute/project/build.gradle ‘]’
2020-05-23T17:22:23.2331378Z + cd /github/workspace
2020-05-23T17:22:23.2331731Z + touch /tmp/katalon_execute/project/.classpath
2020-05-23T17:22:23.2348750Z + chmod -R 777 /tmp/katalon_execute/project
2020-05-23T17:22:23.2405199Z + rm -rf /tmp/katalon_execute/project/bin
2020-05-23T17:22:23.2405700Z + rm -rf /tmp/katalon_execute/project/Libs
2020-05-23T17:22:23.2406389Z + ls -la /tmp/katalon_execute/project
2020-05-23T17:22:23.2406559Z + report_dir=/katalon/katalon/report
2020-05-23T17:22:23.2406822Z + ‘[’ -d /katalon/katalon/report ‘]’
2020-05-23T17:22:23.2407007Z + report_dir=/github/workspace/report
2020-05-23T17:22:23.2407297Z + mkdir -p /github/workspace/report
2020-05-23T17:22:23.2407679Z + args=("$KATALON_KATALON_INSTALL_DIR/katalonc" “$@”)
2020-05-23T17:22:23.2408122Z + args+=("-runMode=console")
2020-05-23T17:22:23.2408419Z + args+=("-reportFolder=$report_dir")
2020-05-23T17:22:23.2408671Z + args+=("-projectPath=$project_prj_file")
2020-05-23T17:22:23.2408863Z + cd /tmp/katalon_execute/workspace
2020-05-23T17:22:23.2410146Z + xvfb-run -a -e /dev/stdout -s ‘-screen 0 1024x768x24’ /opt/katalonstudio/katalonc katalon-execute.sh ‘-browserType=“Chrome”’ -retry=0 -statusDelay=15 ‘-testSuitePath="Test’ ‘Suites/TS_RegressionTest"’ -apiKey=* -runMode=console -reportFolder=/github/workspace/report -projectPath=/tmp/katalon_execute/project/Katalon.prj
2020-05-23T17:22:23.2410386Z total 44
2020-05-23T17:22:23.2410540Z drwxrwxrwx 8 root root 4096 May 23 17:22 .
2020-05-23T17:22:23.2410860Z drwxr-xr-x 4 root root 4096 May 23 17:22 …
2020-05-23T17:22:23.2411687Z -rwxrwxrwx 1 root root 0 May 23 17:22 .classpath
2020-05-23T17:22:23.2411857Z drwxrwxrwx 3 root root 4096 May 23 17:22 Include
2020-05-23T17:22:23.2412189Z -rwxrwxrwx 1 root root 902 May 23 17:22 Katalon.prj
2020-05-23T17:22:23.2412305Z drwxrwxrwx 2 root root 4096 May 23 17:22 Profiles
2020-05-23T17:22:23.2412696Z -rwxrwxrwx 1 root root 9 May 23 17:22 README.md
2020-05-23T17:22:23.2412871Z drwxrwxrwx 3 root root 4096 May 23 17:22 Scripts
2020-05-23T17:22:23.2413026Z drwxrwxrwx 2 root root 4096 May 23 17:22 Test Cases
2020-05-23T17:22:23.2413179Z drwxrwxrwx 2 root root 4096 May 23 17:22 Test Suites
2020-05-23T17:22:23.2413507Z -rwxrwxrwx 1 root root 165 May 23 17:22 console.properties
2020-05-23T17:22:23.2413757Z drwxrwxrwx 4 root root 4096 May 23 17:22 settings
2020-05-23T17:22:26.7827677Z Katalon workspace folder is set to default location: /tmp/session-27f18c08
2020-05-23T17:22:26.8944455Z Starting Groovy-Eclipse compiler resolver. Specified compiler level: unspecified
2020-05-23T17:22:26.8949059Z 176 2.4.7.xx-201611170128-e46 = ACTIVE
2020-05-23T17:22:27.0584350Z
2020-05-23T17:22:27.0584641Z
2020-05-23T17:22:27.0658999Z INFO: Katalon Version: 7.4.2
2020-05-23T17:22:27.0663895Z INFO: Command-line arguments: katalon-execute.sh -browserType=“Chrome” -retry=0 -statusDelay=15 -testSuitePath=“Test Suites/TS_RegressionTest” -apiKey=****** -runMode=console -reportFolder=/github/workspace/report -projectPath=/tmp/katalon_execute/project/Katalon.prj
2020-05-23T17:22:27.0667865Z INFO: User working dir: /tmp/katalon_execute/workspace
2020-05-23T17:22:27.0672161Z INFO: Error log: /tmp/session-27f18c08/.metadata/.log
2020-05-23T17:22:27.0695170Z INFO: Katalon TestOps server URL: https://analytics.katalon.com
2020-05-23T17:22:27.0703177Z INFO: Katalon Store server URL: https://store.katalon.com
2020-05-23T17:22:27.0707597Z INFO: User home: /root
2020-05-23T17:22:27.0711357Z INFO: Java vendor: Private Build
2020-05-23T17:22:27.0714086Z INFO: Java version: 1.8.0_252
2020-05-23T17:22:27.0717969Z INFO: Local OS: Linux 64bit
2020-05-23T17:22:27.1150904Z INFO: CPU load: 1%
2020-05-23T17:22:27.1158025Z INFO: Total memory: 6927 MB
2020-05-23T17:22:27.1163296Z INFO: Free memory: 1164 MB
2020-05-23T17:22:27.1206694Z Start getting machine ID on Linux
2020-05-23T17:22:27.1348414Z End getting machine ID on Linux e6264e1cbff383cf264352375ea7d9fa
2020-05-23T17:22:27.1352187Z Start appending additional signatures
2020-05-23T17:22:27.1369675Z hardwareSerialNumber
2020-05-23T17:22:27.1372616Z osDependentUsername root
2020-05-23T17:22:27.1375629Z End appending additional signatures e6264e1cbff383cf264352375ea7d9fa__root
2020-05-23T17:22:27.1379797Z INFO: Machine ID: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
2020-05-23T17:22:27.1382916Z
2020-05-23T17:22:27.1383012Z
2020-05-23T17:22:27.1399260Z Activating…
2020-05-23T17:22:27.1423361Z Start activating offline…
2020-05-23T17:22:27.1435848Z Search for valid offline licenses in folder: /root/.katalon/license
2020-05-23T17:22:27.1439487Z The number of valid offline licenses: 0
2020-05-23T17:22:27.1442871Z Offline activation failed.
2020-05-23T17:22:27.1442990Z Start activating online…
2020-05-23T17:22:28.6961747Z Cleaning up workspace
2020-05-23T17:22:28.6965662Z Opening project file: /tmp/katalon_execute/project/Katalon.prj
2020-05-23T17:22:31.0915465Z Generating global variables…
2020-05-23T17:22:32.4587077Z Parsing custom keywords…
2020-05-23T17:22:33.1012475Z Project ‘Katalon’ opened
2020-05-23T17:22:33.1126823Z Start reloading plugins…
2020-05-23T17:22:33.4103976Z Katalon version: 7.4.2
2020-05-23T17:22:33.4104457Z Plugin info URL: https://store.katalon.com/api/products/ks?appVersion=7.4.2&appType=ENGINE&licenseType=TRIAL
2020-05-23T17:22:33.4175646Z Plugin info: {
2020-05-23T17:22:33.4175802Z “expired”: false,
2020-05-23T17:22:33.4175954Z “productId”: 59,
2020-05-23T17:22:33.4176128Z “name”: “Basic Report”,
2020-05-23T17:22:33.4176276Z “id”: 174206
2020-05-23T17:22:33.4176418Z }
2020-05-23T17:22:35.0776447Z Plugin name: Basic Report. Is online: true. Plugin location: /opt/katalonstudio/configuration/plugin/install/174206/1.0.7/katalon-studio-report-plugin.jar.
2020-05-23T17:22:35.7257217Z Plugin “Basic Report” has been installed.
2020-05-23T17:22:35.7301822Z Start check license task
2020-05-23T17:22:35.7508271Z com.kms.katalon.execution.exception.ExecutionException: com.kms.katalon.execution.exception.InvalidConsoleArgumentException: Test suite ‘"Test’ not found.
2020-05-23T17:22:35.7508563Z at com.kms.katalon.execution.console.entity.TestSuiteLauncherOptionParser.getConsoleLauncher(TestSuiteLauncherOptionParser.java:240)
2020-05-23T17:22:35.7508784Z at com.kms.katalon.execution.console.ConsoleExecutor.execute(ConsoleExecutor.java:74)
2020-05-23T17:22:35.7509124Z at com.kms.katalon.execution.console.ConsoleMain.launch(ConsoleMain.java:320)
2020-05-23T17:22:35.7509298Z at com.kms.katalon.console.application.Application.runConsole(Application.java:71)
2020-05-23T17:22:35.7509492Z at com.kms.katalon.core.application.Application.runConsole(Application.java:99)
2020-05-23T17:22:35.7509662Z at com.kms.katalon.core.application.Application.start(Application.java:74)
2020-05-23T17:22:35.7509838Z at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:196)
2020-05-23T17:22:35.7509974Z at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:134)
2020-05-23T17:22:35.7510294Z at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:104)
2020-05-23T17:22:35.7510471Z at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:388)
2020-05-23T17:22:35.7510629Z at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:243)
2020-05-23T17:22:35.7510786Z at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2020-05-23T17:22:35.7510948Z at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2020-05-23T17:22:35.7511110Z at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2020-05-23T17:22:35.7511219Z at java.lang.reflect.Method.invoke(Method.java:498)
2020-05-23T17:22:35.7511375Z at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:673)
2020-05-23T17:22:35.7511530Z at org.eclipse.equinox.launcher.Main.basicRun(Main.java:610)
2020-05-23T17:22:35.7511693Z at org.eclipse.equinox.launcher.Main.run(Main.java:1519)
2020-05-23T17:22:35.7511857Z at org.eclipse.equinox.launcher.Main.main(Main.java:1492)
2020-05-23T17:22:35.7512255Z Caused by: com.kms.katalon.execution.exception.InvalidConsoleArgumentException: Test suite ‘"Test’ not found.
2020-05-23T17:22:35.7512459Z at com.kms.katalon.execution.console.entity.TestSuiteLauncherOptionParser.throwInvalidTestSuiteIdException(TestSuiteLauncherOptionParser.java:335)
2020-05-23T17:22:35.7512608Z at com.kms.katalon.execution.console.entity.TestSuiteLauncherOptionParser.getTestSuite(TestSuiteLauncherOptionParser.java:330)
2020-05-23T17:22:35.7512783Z at com.kms.katalon.execution.console.entity.TestSuiteLauncherOptionParser.buildRunConfig(TestSuiteLauncherOptionParser.java:258)
2020-05-23T17:22:35.7512959Z at com.kms.katalon.execution.console.entity.TestSuiteLauncherOptionParser.getConsoleLauncher(TestSuiteLauncherOptionParser.java:237)
2020-05-23T17:22:35.7513202Z … 18 more
2020-05-23T17:22:35.7513297Z
2020-05-23T17:22:35.7513382Z All launchers terminated
2020-05-23T17:22:36.5915706Z + ret_code=2
2020-05-23T17:22:36.5922146Z ++ id -u
2020-05-23T17:22:36.5934707Z ++ id -g
2020-05-23T17:22:36.5944222Z + chown -R 0:0 /github/workspace/report
2020-05-23T17:22:36.5958563Z + chmod -R 777 /github/workspace/report
2020-05-23T17:22:36.5966616Z + ls /github/workspace/report
2020-05-23T17:22:36.5977320Z + cd /github/workspace
2020-05-23T17:22:36.5977843Z + exit 2
By the way, I’m wondering what kind of genius has come up with using spaces in “Test Suits” etc.? Everyone knows that using spaces in directories names it’s a recipe for disaster when it should be integrated with something else. It could’ve been named just “Suites” if Test_Suites is not good.
@gregory2
Please change -testSuitePath to:
Test Suites/TS_RegressionTest
not
Test’ ‘Suites/TS_RegressionTest
@duyluong,
In the action itself I’ve always used testSuitePath=“Test Suites/TS_RegressionTest”’, as it can be seen from the screenshots posted above and the action source posted below.
The problem is that somehow https://github.com/katalon-studio/katalon-studio-github-action is working in such a way that ’-testSuitePath=“Test’ 'Suites/TS_RegressionTest” is shown in xvfb
( even though, in command-line, testSuitePath=“Test Suites/TS_RegressionTest” is shown correctly ), as it can be seen from the screenshots and the full log posted above:
2020-05-23T17:22:23.2410146Z + xvfb-run -a -e /dev/stdout -s ‘-screen 0 1024x768x24’ /opt/katalonstudio/katalonc katalon-execute.sh ‘-browserType=“Chrome”’ -retry=0 -statusDelay=15 ‘-testSuitePath=“Test’ ‘Suites/TS_RegressionTest”’ -apiKey= * -runMode=console -reportFolder=/github/workspace/report -projectPath=/tmp/katalon_execute/project/Katalon.prj
2020-05-23T17:22:27.0663895Z INFO: Command-line arguments: katalon-execute.sh -browserType=“Chrome” -retry=0 -statusDelay=15 -testSuitePath=“Test Suites/TS_RegressionTest” -apiKey=****** -runMode=console -reportFolder=/github/workspace/report -projectPath=/tmp/katalon_execute/project/Katalon.prj
This is a basic workflow to help you get started with Actions
name: CI
on: [push]
jobs:
katalon_test_job:
runs-on: ubuntu-latest
name: Run katalon Studio Test CLI
steps:
# To use this repository’s private action, you must check out the repository
- name: Checkout
uses: actions/checkout@v2
#- uses: ./ # Uses an action in the root directory COMMENTED OUT BY ME BECAUSE OF THE ERROR g2
- name: Get and run action
#env:
#TEST_SUITE: “Test Suites/TS_RegressionTest”
uses: katalon-studio/katalon-studio-github-action@0.9
with:
# katalon_api_key: ${{ secrets.KATALON_API_KEY }}
# Katalon_command: ‘katalon-execute.sh -browserType=“Chrome” -retry=0 -statusDelay=15 -testSuitePath=“Test Suites/TS_RegressionTest”’
# : MODIFIED BY ME BECAUSE OF THE ERRORS :
ks_api_key: ${{ secrets.KATALON_API_KEY }}
ks_command: ‘katalon-execute.sh -browserType=“Chrome” -retry=0 -statusDelay=15 -testSuitePath=“Test Suites/TS_RegressionTest”’