Katalon Gitlab CI/CD; run only changed tests

Product Forums Katalon Studio
,
1

Hello everyone!

I want to know, if its possible to tell the Katalon Runtime Engine to only run changed test cases.
I have created a gitlab-ci.yml with my test suite. The test suite contains 325 test cases at the moment and every time when i change 1 test case and commit it to git, the gitlab-runner runs all 325 test cases which takes a lot of time and is not very efficient.
Can i tell the gitlab-runner to only run the changed test cases via the .yml file? Or is there a setting somewhere in Katalon which i just did not find? Or is it just impossible?

i mean, i could create another test suite and fill in the test cases that i changed and tell the runner in the .yml file to run this suite but this is also not very efficient.

So, if anyone has an idea how to solve that problem, please tell me :slight_smile:

Kind regards,

Nils

2

No such automatism is available. And I think it is impossible in the way you wrote above.

Howerver you do have an option to shorten the minutes required to execute your tests after you changed a test case code. That is Test Suite Collection.

A Test Suite Collection binds a set of Test Suites. Each bound Test Suite can be toggled ON/OFF to execute as the following screenshot shows. If you toggle OFF a Test Suite, then the testcases which are enclosed in the Test Suite will be skipped when the Test Suite Collection is invoked.

TestSuiteCollection_RunOption
TestSuiteCollection_RunOption1010Ă—596 48.3 KB

@Kokahontas has 325 test cases. Let me imagine, you categorize the 325 test cases into 31 Test Suits. Each Test Suite would contain approximately 10 test cases. And you make a Test Suite Collection named Selective Launcher. The Selective Launcher will have a list of 31 Test Suites as candidate to run.

When you changed 1 test case, you call up the definition of “Selective Launcher” and tune “Run” toggles. Possibly you will toggle ON only 1 Test Suite, which contains that updated Test Case. Then you run the Selective Launcher. It will execute only 10 Test Cases. It will finish far quicker than running all 325 test cases. Isn’t it good enough?

Test Suite Collection supports parallel execution. That may also help to shorten the minutes if your machine is equipped with good CPU/Memory resources.

Please note, you can create 2 or more Test Suite Collections, each of them contain the same set of Test Suites (= Test Cases). For example, you may want a Full Launcher as well as Selective Launcher. You can choose which one to invoke as appropriate. You may want 2 jobs in your CI server; one for the Selective Launcher, and another for the Full Launcher.

3

Well, its not impossible it just needs some knowledge :slight_smile:
If anyone is interested:

.gitlab-ci.yml:

run_only_changed_test_cases:
      stage: test
      tags:
        - katalon
      script:
        - rm /home/gitlab-runner/changedfiles.txt
        - git log -1 --pretty=%B
        - cd $directory
        - git show --stat=100000 HEAD | grep -o '.*|' | sed -r 's/\s*\|.*$//g' | sed 's/^\s*//g' | grep -E '^(Scripts)' | grep -Ev '(*Template|*template)' | xargs -I{} dirname "{}" > /home/gitlab-runner/changedfiles.txt
        - ./UseChangedFiles.sh > /home/gitlab-runner/$build/CI-CD.ts
        - xvfb-run -a -n 0 -s "-screen 0 1920x1080x24" ~/.katalon/7.2.3/Katalon_Studio_Engine_Linux_64-7.2.3/katalonc -noSplash -runMode=console -projectPath="$directory" -retry=0 -testSuitePath="$suite" -executionProfile="$profile" -browserType="$browser" -reportFolder="/var/www/html/reports/Reports/Reports-$TIMESTAMP" -g_AccountName="$USERNAME" -g_number="$phone" -apiKey="$key"
      interruptible: false
      only:
      - Core

The UseChangedFiles.sh looks like this:

#!/bin/bash¶                                                                                                                                                     
Counter="$(wc -l /home/gitlab-runner/changedfiles.txt | sed 's/\/home\/gitlab-runner\/changedfiles.txt/\ /g' )"
Header=$(echo '<?xml version="1.0" encoding="UTF-8"?>
<TestSuiteEntity>
  <description></description>
  <name>CI-CD</name>
  <tag></tag>
  <isRerun>false</isRerun>
  <mailRecipient></mailRecipient>
  <numberOfRerun>0</numberOfRerun>
  <pageLoadTimeout>10</pageLoadTimeout>
  <pageLoadTimeoutDefault>true</pageLoadTimeoutDefault>
  <rerunFailedTestCasesOnly>false</rerunFailedTestCasesOnly>
  <testSuiteGuid>1029c065-5649-492b-8b2e-0b49f609a0b0a</testSuiteGuid>'

for (( i=1; i<=$Counter; i++ ))
do

OutputFiles=$(sed -n "$i"p \/home\/gitlab-runner\/changedfiles.txt | sed s/Scripts/Test\ Cases/g)
echo -e "   <testCaseLink>
    <guid>e4403178-0525-466e-9f23-1dc4ce5c02$i</guid>
    <isReuseDriver>false</isReuseDriver>
    <isRun>true</isRun>
    <testCaseId>$OutputFiles</testCaseId>
  </testCaseLink>"

done

Its working fine and maybe it will help some people :slight_smile:

2 Likes