This post will help those understand how to integrate Katalon Studio with TestRail properly. The current guides on Katalon is not very clear in terms of explanation so I will post the proper guide I have written for my company here.
Link test cases between Katalon Studio and TestRail
To submit test results from Katalon Studio to TestRail, you need to link a test case from Katalon Studio to the one in TestRail. In order to do that, you need to save the information of the test case in TestRail to Katalon Studio’s global variable.
Setting up global variables
In the default profile of Katalon Studio, create these three variables:
G_run_testrail_tc_id : a list of executed TestRail test case IDs
G_run_testrail_tc_status : a list of executed TestRail test case status
G_testrail_run_id : a string value type which will include current TestRail test run ID.
To get the Value for the Test Run ID, go to your project on TestRail > click on Test Runs > Open the Test Run created > and see the ID next to the name i.e.: 
The value will be the numbers after the letter.
To add the above, go to Profiles > default and click Add at the top. See example below of how it should look:
Set up Katalon Studio test cases
In your Test Cases folder on Katalon, where you will have the test cases for automation, each test case will need to store a Variable related to the TestRail test case IDs. This variable should store numbers only. For instance, if you have a test case with ID C2510, the variable should be filled with “2510.” If any of Katalon Studio’s test cases relates to some TestRail test cases (to demonstrate an end-to-end test flow), the variable should be entirely filled with related TestRail test case IDs, separated by commas such as “2510, 2511, 2512.”
Although this step is manual for each test case—meaning that it takes time—it is an essential part of the process. To do this, see instructions below:
- Open the test case on Katalon Studio
- Click Variables at the bottom:
- Click Add at the top:
- Double Click Name field and add this vairable- testrail_tc_id
- Leave Value Type as String
- In Default Value, enter the number of the test case from TestRail.
Example:
Set up TestListener to collect tests and results after executing Katalon Studio test cases
Test Listeners are test steps that created based on your own criterias and will be executed when the condition is matched.
Create an AfterTestCase listener to get the testrail_tc_id and the status of the executed TestRail test case. These information are then updated to global variables for later use. To do this:
- Right Click Test Listeners folder on Katalons Test Explore > New > New Test Listener:
- Set it to a suitable name i.e.: Get_TC_STATUS
- Check the AfterTestCase and AfterTestSuite method boxes
- Click OK
- Copy and paste the following code inside @AfterTestCase
/**
* Executes after every test case ends.
*/
@AfterTestCase
def afterTestCase(TestCaseContext testCaseContext) {
try {
def tc_ids = testCaseContext.getTestCaseVariables()['testrail_tc_id'].split(",")
for (def n : (0 .. tc_ids.length - 1)) {
GlobalVariable.G_run_testrail_tc_id.add(tc_ids[n])
GlobalVariable.G_run_testrail_tc_status.add(testCaseContext.getTestCaseStatus())
}
} catch (NullPointerException ex) {
}
}
We will come back to the other method in there at the end so continue…
Create custom keywords to interact with TestRail from Katalon Studio via API
In addition to the built-in keywords, users can define custom keywords to extend the capability of Katalon Studio. Once created, custom keywords can be used in test cases just like other built-in keywords. This feature allows users to expand keywords and reuse them across projects.
The Custom Keywords will contain the API requests to TestRail to get test case from the run, update the test run and add results to the run.
- Create a New Package inside Keywords:
- Enter a suitable name i.e.: sync_testrail
- Right Click the package and select New > Keyword:
- Enter a class name i.e.: Testrail
- Click OK
Once the keyword file is created, please copy and paste the below into the file so that the code works and it has the right packages:
import groovy.json.JsonSlurper as JsonSlurper
import com.kms.katalon.core.testobject.TestObjectProperty as TestObjectProperty
import com.kms.katalon.core.testobject.ResponseObject as ResponseObject
import com.kms.katalon.core.testobject.ConditionType
import com.kms.katalon.core.testobject.RequestObject as RequestObject
import com.kms.katalon.core.testobject.impl.HttpTextBodyContent
Methods inside the Custom Keywords
Once the above is done, we will create methods inside the customer keyword we created. All the methods will need to be part of this one Keyword to make it easy. I’ve added comments in the code to make it easy to understand.
Methods below will require your credentials encoded to Base64%. To get this:
-
Enter email address for TestRail on Line 1
-
Enter your password for TestRail on Line 2
-
Click >Encode<
-
This will generate your base64 encoded credentials to paste after ‘Basic’ in the codes below. Replace everything from %xxxx% with your base64 credentials
This first method makes a GET request to TestRail to receive all test case IDs from your selected TestRail run. These IDs will be used later to update the execution results from Katalon Studio to TestRail. Copy and paste the code inside the public class:
//Makes a GET request to TestRail to get all the test case IDs from the selected TestRail run.
@Keyword
def get_tests(String id) {
def slurper = new JsonSlurper()
RequestObject ro = new RequestObject('a')
ro.setRestRequestMethod('GET')
ro.setRestUrl('https://%YOUR PROJECT%/index.php?/api/v2/get_tests/'+id)
def httpheader = new ArrayList<TestObjectProperty>()
httpheader.add(new TestObjectProperty(
'Content-Type', ConditionType.EQUALS, 'application/json'))
httpheader.add(new TestObjectProperty(
'Authorization', ConditionType.EQUALS,
'Basic %PASTE YOUR ENCODED CREDENTIALS HERE%'))
ro.setHttpHeaderProperties(httpheader)
ro.setBodyContent(new HttpTextBodyContent('', 'UTF-8', 'application/json'))
def response = WSBuiltInKeywords.sendRequest(ro)
return slurper.parseText(response.getResponseText())
}
This method makes a POST request to TestRail to update the selected test run with a new list of test case IDs. This is a common practice advised by TestRail itself. Please note that you can change the request body ‘include_all’ parameter to ‘true’ if you want to include all available test cases to the selected test run.
@Keyword
def update_run(String id, String array) {
def slurper = new JsonSlurper()
RequestObject ro = new RequestObject('Update TestRail test run')
ro.setRestRequestMethod('POST')
ro.setRestUrl('https://%YOUR PROJECT%/index.php?/api/v2/update_run/'+id)
def httpheader = new ArrayList<TestObjectProperty>()
httpheader.add(new TestObjectProperty(
'Content-Type', ConditionType.EQUALS, 'application/json'))
httpheader.add(new TestObjectProperty(
'Authorization', ConditionType.EQUALS,
'Basic %PASTE YOUR ENCODED CREDENTIALS HERE%'))
ro.setHttpHeaderProperties(httpheader)
//StringArray array = new String(tc_id.size())
def body ='{"include_all": false,"case_ids": '+ array +'}'
WebUI.comment('body='+body)
ro.setBodyContent(new HttpTextBodyContent(body, 'UTF-8', 'application/json'))
def response = WSBuiltInKeywords.sendRequest(ro)
return slurper.parseText(response.getResponseText())
}
This final method also makes a POST request to TestRail to to add test results to TestRail.
//Makes a POST to add the test results to the TestRail run.
@Keyword
def add_results(String id, String request) {
def slurper = new JsonSlurper()
RequestObject ro = new RequestObject('a')
ro.setRestRequestMethod('POST')
ro.setRestUrl('https://%YOUR PROJECT%/index.php?/api/v2/add_results_for_cases/'+id)
def httpheader = new ArrayList<TestObjectProperty>()
httpheader.add(new TestObjectProperty(
'Content-Type', ConditionType.EQUALS, 'application/json'))
httpheader.add(new TestObjectProperty(
'Authorization', ConditionType.EQUALS,
'Basic %PASTE YOUR ENCODED CREDENTIALS HERE%'))
ro.setHttpHeaderProperties(httpheader)
WebUI.comment('body='+request)
ro.setBodyContent(new HttpTextBodyContent(
request, 'UTF-8', 'application/json'))
def response = WSBuiltInKeywords.sendRequest(ro)
def response_array = slurper.parseText(response.getResponseText())
return slurper.parseText(response.getResponseText())
}
To make sure you’ve made everything correct you can play with same requests via Postman app after reading TestRail API documentation.
Create Katalon Studio test cases to handle communication between Katalon Studio and TestRail
Once all API call methods are ready, you will finally be able to start the integration. This will be done using an automated test case that we need to create inside the Test Cases Folder.
So create a test case and call it anything you want i.e.: TestRail_Update_Run_With_Results
Once you have created the test case, open it and click on Script at the bottom. Clear everything and copy and paste the code below. Also read the things to remember which are important after the below code:
import static com.kms.katalon.core.checkpoint.CheckpointFactory.findCheckpoint
import static com.kms.katalon.core.testcase.TestCaseFactory.findTestCase
import static com.kms.katalon.core.testdata.TestDataFactory.findTestData
import static com.kms.katalon.core.testobject.ObjectRepository.findTestObject
import com.kms.katalon.core.checkpoint.Checkpoint as Checkpoint
import com.kms.katalon.core.cucumber.keyword.CucumberBuiltinKeywords as CucumberKW
import com.kms.katalon.core.mobile.keyword.MobileBuiltInKeywords as Mobile
import com.kms.katalon.core.model.FailureHandling as FailureHandling
import com.kms.katalon.core.testcase.TestCase as TestCase
import com.kms.katalon.core.testdata.TestData as TestData
import com.kms.katalon.core.testobject.TestObject as TestObject
import com.kms.katalon.core.webservice.keyword.WSBuiltInKeywords as WSBuiltInKeywords
import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUI
import internal.GlobalVariable as GlobalVariable
import java.util.ArrayList as ArrayList
def response = CustomKeywords.'Testrail.get_tests'(GlobalVariable.G_testrail_run_id)
def tcs_to_add = []
for (def n : (0..response['id'].size)) {
if (((response['case_id'])[n]) != null) {
tcs_to_add.add((response['case_id'])[n])
}
}
response = CustomKeywords.'Testrail.update_run'(GlobalVariable.G_testrail_run_id, tcs_to_add.toString())
def total_tcs_to_update = []
for (def n : (0..GlobalVariable.G_run_testrail_tc_id.size)) {
if ((GlobalVariable.G_run_testrail_tc_id[n]) != null) {
total_tcs_to_update.add(GlobalVariable.G_run_testrail_tc_id[n])
}
}
for (def n : (0..tcs_to_add.size)) {
if ((tcs_to_add[n]) != null) {
total_tcs_to_update.add(tcs_to_add[n])
}
}
def PASSED = '1';
def FAILED = '5';
def status_id
String request = '{"results": ['
// Generate a request for update test results
for (def n : (0..GlobalVariable.G_run_testrail_tc_id.size)) {
if ((GlobalVariable.G_run_testrail_tc_id[n]) != null) {
if ((GlobalVariable.G_run_testrail_tc_status[n]) == 'PASSED') {
status_id = PASSED
} else {
status_id = FAILED
}
request = request.concat(((('{"case_id":' + (GlobalVariable.G_run_testrail_tc_id[n])) + ',"status_id":') + status_id) +
',"comment":"Automation test via Katalon"},')
}
}
request = request.substring(0, request.length() - 1) //removing last excessive comma from request
request = request.concat(']}')
WebUI.comment('request=' + request.toString())
response = CustomKeywords.'Testrail.add_results'(GlobalVariable.G_testrail_run_id, request)
WebUI.comment('response=' + response.toString())
Things to remember with the above script:
- With any of the code files above, you can press CTRL + A to select ALL code and then press CTRL+SHIFT+F to automagically indent it properly the groovy java way.
-
- After CustomKeywords, you must tell it where the keyword is. In our case, we called the file Testrail hence it’s TestRail.get_tests where the API methods live - You can change the Comment in the request to whatever you want Katalon to report to TestRail
AfterTestSuite Listener
The final step is adding another listener at TestSuite to call the Katalon Studio test case you created in the previous step. To do this, go back to the Test Listener created earlier and paste this after the @AfterTestCase code:
/**
* Executes after every test suite ends.
*/
@AfterTestSuite
def afterTestSuite() {
'push results to testrail'
WebUI.callTestCase(findTestCase('%replace this with path to your test case, for example: Test Cases/TestRail_Update_Run_With_Results%'), [:], FailureHandling.CONTINUE_ON_FAILURE)
}
To find the path of the test case you created with the long script, right click the test case and click on Properties. Copy the whole ID and pate it above. Below is an example of how it should actually look like if you followed the name in the guide:
example only:
How to report to TestRail once set up
After the above has all been set up, you will need to ensure that whatever test cases you have on your TestRail runs, the same automation tests are added to a Test Suite before running it. Each test case must have the testrail_tc_id variable with the correct test case ID as explained above.
To add tests to the Test Suite:
- Create a New Test Suite:
- Enter a Name and click OK
- Click Add
- Select the Test Case that you want to add which are part of the Test Run on TestRail
- DO NOT SELECT the test case which communicates with TestRail (the one with the long script. There is no harm but it will mean your results will be reported twice to TestRail.):
Once the test cases are added to the Test Suite, Hit the Run button at the top, relax and watch the magic happen as your test case are ran and the results are reported to TestRail.
Conclusion
Integrating between Katalon Studio and TestRail can help users not only reduce the complexity and cost during the testing process, but also improve the productivity and efficiency in automation testing and test case management.
Groovy is a messy language however there is so much on the web to learn about it. If you have any issues or questions, please let me know. Also, would be great to get feedback on this below to see how people got along and whether the guide was easy to follow or not.
might be worth checking on Google to see if someone has come across this. We are now moving to AzureDevOps where I hope to continue using Katalon Studio to run test cases
Testrail
