I have installed the TestRail plugin and completed the settings.
TestRail test run is getting added after execution of Test suite however no test results updated for my test cases. There were marked as Untested.
Is this because I am using a free version of Katalon Studio ?
Hi,
Have you referred to Example plugin: Build Katalon Studio TestRail Integration plugin | Katalon Docs and Upload test execution results from Katalon Studio to TestRail | Katalon Docs yet?
Yes, I have referred and followed all steps…
Is Free version of Katalon won’t support this?
@shruti1 Yes I believe this does not work because you do not have an enterprise license for Katalon. Referring to the pricing page of the TestRail plugin, it looks like it is only available with the enterprise license.
Thanks Kian.
@kreno
I have a 30 days of katalon Enterprice license, and able to add TestRail plugin. Did all set up its creating a test run however not updating the results into testRail. I have installed test rail plugin 1.1.3 and saw the same issue happened in past 2021(Refer, TestRail Integration - #58 by bionel)
@shruti1
What is your Testrail version version? It is a cloud instance or on-prem install?
If on-prem, what is the exact version?
Note that, the Katalon doc says it will only support the cloud version.
Also, I noticed, TestRail 8.0.1 was just released in June this year and it is shipped gradually to customers.
It could be an API change (they are known to break stuff whit every new version) so the plugin may have to be reviewed.
Noticed also this in the release notes:
see for version 1.1.2: Resolved issue: Execution report is not uploaded to TestRail if TestOps is enable
May be that this is not properly fixed.
Do you have TestOps integration enabled? If yes, can you try to disable it and do one more run?
If my assumption is correct and the issue is caused by TestOps integration … well, this is supposed to be fixed with this commit:
… but i don’t see any logic in this.
The only change in the code is, default status is no longer 0 (which is not a valid testrail status) but now is 2 (Blocked).
Anyway the test results are still not updated since are shown with default status at creation time ( 3 = untested) so the defect may be in the eventListener implementation actually.
However, waiting for more data from @shruti1 for now …
LE: mhm … by looking at the plugin code, more specifically here:
It looks like, in this iteration, in needs TestRailConstants.INTEGRATION_TESTCASE_ID.
This, I guess, is the tescase Id from testrail which hase to be set in the integration tab in Katalon for each and every one …
@shruti1 have you done also this step, after the suite was created in Testrail?
My test rail on-prem version:
TestRail v6.0.0.4140
TestOps integration is not enabled, however i can access TestOps in browser with my account.
Yes, I have just installed TestRail plugin and connected. After that in my test case integration tab provided test case id from TestRail.
Looks all steps I followed.
Do I need to try TestRail old version plugin like 1.1.1 ?
Will check with that as well.
How to intall old version of Test Rail plugin, I am not sure?
Also, I have not written any code for TestRail integration as I read in this forum either you have to write intergration code or you can install plugin.
Do i need to write the full code?
Acc. to katalon team, only integration with testrail cloud is supported.
I am not sure why, the API is pretty much consistent but, there may be some endpoints behaving different compared with the current cloud version.
6.0.0.4140 is quite old.
I have no idea how you can do that, other than grab the source code, revert the commits to a certain ‘milestone’ and do a maven build, followed by a manual install.
Looks like you can manually download previous version from the release page:
See also:
Anyway, most probably this will fix nothing.
Maybe, maybe not.
The only thing you can try now is, open your test suite, go to the eventLog tab, clear the log and do one more run.
If I understand the docs right, output to console from plugins should appear there, perhaps we can see something relevant to check where the upload get’s stuck, see pic:
Povided we can figure it out what is wrong, you can attempt to grab the code written by katalon team, patch it and produce your own build, so you don’t have to write everything from scratch.
The procedure on how a plugin is developed and built is documented summary here:
See also some debug instructions, not sure how this will help you:
Thanks for reply, I will try it and let you know.
@bionel
I have cleared the Event-Log and ran the test suite .
It provided results in Event-log however one exception for TestRail . please check that below
chromedriver is located at default location: C:\Katalon\Katalon_Studio_Windows_64-8.5.5\Katalon_Studio_Windows_64-8.5.5\configuration\resources\drivers\chromedriver_win32\chromedriver.exe. In case your browser is updated to a newer version, please use this command to update chromdriver: --config -webui.autoUpdateDrivers=true
Launcher status after execution process completed: Running
Start writing execution.uuid file to folder: C:\Katalon Studio\Jacobi_KatalonWebUIRegressionTests\Reports\20230711_112743\S2 TestWebUI\20230711_112743
Start generating JUnit report folder at: C:\Katalon Studio\Jacobi_KatalonWebUIRegressionTests\Reports\20230711_112743\S2 TestWebUI\20230711_112743...
JUnit report were generated at folder: C:\Katalon Studio\Jacobi_KatalonWebUIRegressionTests\Reports\20230711_112743\S2 TestWebUI\20230711_112743
For real-time monitoring and better reporting capabilities please integrate this project with Katalon TestOps (more details at https://docs.katalon.com/katalon-analytics/docs/integration-with-katalon-studio.html#enable-integration)
For real-time monitoring and better reporting capabilities please integrate this project with Katalon TestOps (more details at https://docs.katalon.com/katalon-analytics/docs/integration-with-katalon-studio.html#enable-integration)
TestRail: Start sending summary message to channel:
Summary execution result of test suite: Test Suites/S2 TestWebUI
Total test cases: 4
Total passes: 4
Total failures: 0
Total errors: 0
Total skipped: 0
TestRail: Summary message has been successfully sent
Create new test run S2 TestWebUI
Send post url: add_run/1 data: {include_all=false, case_ids=[143, 2851, 2854, 138], name=S2 TestWebUI, suite_id=2}
Receive: {"completed_on":null,"milestone_id":null,"description":null,"custom_status3_count":0,"is_completed":false,"retest_count":0,"custom_status5_count":0,"project_id":1,"id":103,"suite_id":1,"custom_status2_count":0,"include_all":false,"passed_count":0,"custom_status7_count":0,"custom_status4_count":0,"created_by":9,"url":"http:\/\/testrail.in.jaco.bi\/testrail\/index.php?\/runs\/view\/103","config_ids":[],"blocked_count":0,"created_on":1689039143,"untested_count":4,"name":"S2 TestWebUI","assignedto_id":null,"failed_count":0,"custom_status1_count":0,"custom_status6_count":0,"config":null,"plan_id":null}
Send get url get_tests/103
Receive: [{"run_id":103,"custom_expected":null,"custom_type":1,"type_id":9,"milestone_id":null,"estimate_forecast":null,"title":"Logout","custom_preconds":null,"custom_steps":null,"priority_id":3,"status_id":3,"custom_test_data":null,"refs":null,"case_id":138,"estimate":null,"assignedto_id":null,"template_id":1,"id":9253},{"run_id":103,"custom_expected":null,"custom_type":1,"type_id":9,"milestone_id":null,"estimate_forecast":null,"title":"Login with normal USER","custom_preconds":null,"custom_steps":null,"priority_id":2,"status_id":3,"custom_test_data":null,"refs":null,"case_id":143,"estimate":null,"assignedto_id":null,"template_id":1,"id":9254},{"run_id":103,"custom_expected":"verify no exception throws","custom_type":1,"type_id":9,"milestone_id":null,"estimate_forecast":null,"title":"Hamburger Menu - My Settings -> Preferences","custom_preconds":null,"custom_steps":"- change some preferences in user setting and save it","priority_id":4,"status_id":3,"custom_test_data":null,"refs":null,"case_id":2851,"estimate":null,"assignedto_id":null,"template_id":1,"id":9255},{"run_id":103,"custom_expected":null,"custom_type":1,"type_id":9,"milestone_id":null,"estimate_forecast":null,"title":"Hamburger Menu - History","custom_preconds":null,"custom_steps":null,"priority_id":4,"status_id":3,"custom_test_data":null,"refs":null,"case_id":2854,"estimate":null,"assignedto_id":null,"template_id":1,"id":9256}]
java.lang.ClassCastException: org.json.simple.JSONArray cannot be cast to org.json.simple.JSONObject
at com.katalon.plugin.testrail.TestRailConnector.getTestCaseIdInRun(TestRailConnector.java:90)
at com.katalon.plugin.testrail.TestRailEventListenerInitializer.lambda$registerListener$2(TestRailEventListenerInitializer.java:139)
at com.katalon.platform.internal.event.EventListernerDelegate.lambda$handleEvent$0(EventListernerDelegate.java:38)
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384)
at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:647)
at com.katalon.platform.internal.event.EventListernerDelegate.handleEvent(EventListernerDelegate.java:38)
at com.katalon.platform.internal.event.EventListernerDelegate$1.handleEvent(EventListernerDelegate.java:24)
at org.eclipse.e4.ui.services.internal.events.UIEventHandler.handleEvent(UIEventHandler.java:36)
at org.eclipse.equinox.internal.event.EventHandlerWrapper.handleEvent(EventHandlerWrapper.java:205)
at org.eclipse.equinox.internal.event.EventHandlerTracker.dispatchEvent(EventHandlerTracker.java:203)
at org.eclipse.equinox.internal.event.EventHandlerTracker.dispatchEvent(EventHandlerTracker.java:1)
at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:234)
at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:151)
at org.eclipse.equinox.internal.event.EventAdminImpl.dispatchEvent(EventAdminImpl.java:132)
at org.eclipse.equinox.internal.event.EventAdminImpl.sendEvent(EventAdminImpl.java:75)
at org.eclipse.equinox.internal.event.EventComponent.sendEvent(EventComponent.java:44)
at org.eclipse.e4.ui.services.internal.events.EventBroker.send(EventBroker.java:55)
at com.kms.katalon.execution.launcher.ReportableLauncher.fireTestSuiteExecutionEvent(ReportableLauncher.java:672)
at com.kms.katalon.execution.launcher.ReportableLauncher.preExecutionComplete(ReportableLauncher.java:172)
at com.kms.katalon.execution.launcher.ProcessLauncher.onWatchdogComplete(ProcessLauncher.java:194)
at com.kms.katalon.execution.launcher.LaunchWatchdog.notifyProcessTerminated(LaunchWatchdog.java:136)
at com.kms.katalon.execution.launcher.LaunchWatchdog.run(LaunchWatchdog.java:66)
at java.lang.Thread.run(Thread.java:748)
Changing Plugin version did not help, now trying with writing integration code with disabling the TestRail plugin.
Following steps from,
This gives valuable info.
The issue happens when the event listener reaches this line of code (139):
List<Long> testCaseIdInRun = connector.getTestCaseIdInRun(testRunId);
which calls the getTestCaseIdInRun method from TestrailConnector class.
It breaks at this line:
Reason:
This method send a get request to the "get_tests/" + id endpoint, which is documented here:
It is expected to receive a JSON object response, sort off:
{
"offset": 0,
"limit": 250,
"size": 236,
"_links": {
"next": null,
"prev": null,
},
"tests": [
{
"id": 1,
"title": "Test conditional formatting with basic value range",
},
{
"id": 2,
"title": "Verify line spacing on multi-page document",
},
]
}
This is a paginated endpoint, all bulk endpoints have been updated to use this structure since testrail version 6.7, see:
Anyway, you are using an older version (6.0.x).
Luckily I have also an on-prem install (is 6.6.something but the response is same)
The response is like this:
[
{
"id": 46232776,
"case_id": 3973493,
"status_id": 1,
"assignedto_id": null,
--- some more fields
}
]
Which is a Json array, therefore cannot be casted as an object.
To fix it, you will have to replace this code block completely:
String paginationNextURL = null;
String requestURL = "";
JSONArray jsonArray = new JSONArray();
do {
if (paginationNextURL != null) {
requestURL = shortenURL(paginationNextURL);
} else {
requestURL = "get_tests/" + id;
}
JSONObject response = (JSONObject) sendGet(requestURL);
JSONObject paginationLinks = (JSONObject) response.get("_links");
paginationNextURL = (String) paginationLinks.get("next");
jsonArray.addAll((JSONArray) response.get("tests"));
} while (paginationNextURL != null);
to collect properly the needed data in the jsonArray variable.
You can try something like:
requestURL = "get_tests/" + id;
JSONArray jsonArray = (JSONArray) sendGet(requestURL);
Which means you have to make your hands dirty, get the plugin source, patch it, build it, install it from local source … and cross your fingers to not break at further called endpoints.
Alternate, you can upgrade your testrail install to 6.7.x but …
You have to reach out to gurock for instruction and preferably try this on a staging instance first (has to be done incremental steps) …
… another but, above 6.6 testrail introduced a new bug (alongside others but this one is very nasty): tests attachments are no longer displayed for existing testcases using such.
This has been fixed somewhere in version 7.3.x … but, another funny stuff, above 7.0.2 testrail introduced a new database (Cassandra) to handle atachment indexing.
So, to upgrade to a decent version will be painfull.
Feel free to try.
Most probably you will be hit by a similar issue when the response to get_tests is parsed
e.g something like: How to integrate Katalon Studio to TestRail to report test case results - #11 by leszek.urbanik
… but with a bit of effort that can be fixed (seems to be the same reason as above)
The benefit of using that approach is, should work fine with katalon free, so use whatever you find appropiate.
@bionel
Thanks for your inputs. Our Test Rail gets upgraded to new version soon.
I tried with Test Rail integration code. It is executing fine, however it has not added any test run into Test Rail.
I excuted then in debug mode, I can see respose contains test case id and text.
I am not able to figure out where is the issue? Below are the screen shots:
Still the updates on Plugin I did not try.
This approach is using the update_run endpoint, which is documented here:
Note:
Updates an existing test run
So, you have to create the testrun manually in testrail and add the needed testcases from the suite, then add it in the global variable G_testrail_run_id prior to run Katalon.
Looks like, this approach does not create the run automatically, as the plugin does.
If you need also this feature you have to create one more keyword using the add_run endpoint and tune the logic of the main script accordingly (e.g if you do not provide a run id in the global variable it should create it based on a provided suite_id and either a selection of cases or with include_all)
The endpoint usage is documented here:
If you already did that (created the test run), one more thing.
I noticed you have an error in the update_run response (last pic), looks like a testcase it is missing from the run (C143) in testrail so the api call fails.
Check from where you get that value (perhaps the Global variable you set?)
Yes, you are correct @bionel
I got the results updated when i created new test run in test rail, added respective test cases to it and provided id in variable ‘G_testrail_run_id’ in katalon.
The TestRail plugin should work in this way however not working, not sure it is because we have on-premises software. Anyway it is not stopping automation.
Thanks for your help 
I have provided the explanation … it is not exactly due to on-prem, but due to the version.
Anyway, I think the approach with custom keywords is more flexible, e.g you can override the global variable at runtime when using KRE, and you can easily addapt the code to provide more flexibility
The plugin is lacking some features and it is a bit more difficult to extend it (but not impossible)
