Hi @tri.tle thanks for your feedback, here are a few examples why we don’t use the Katalon TestRail integration.
- We are TestRail Server Customers and according to your TestRail integration documentation
The TestRail Integration plugin only supports integrating Katalon Studio with TestRail Cloud
- It is only possible to link Katalon TestCases with TestRail TestCases and Katalon TestSuites with TestRail Runs at a high level. However, unfortunately not all parameters of the TestRail API (https://www.gurock.com/testrail/docs/api/reference/results/) can be passed via the Katalon Integration to TestRail, which is why the result, if it can be pushed at all, is only very superficial in terms of the level of detail. e.g.:
a) In our TestRail configuration the Result field “Version” is a mandatory field. Katalon can run the test successfully, but cannot push the result because you cannot submit the version. This should be possible by, for example, reading the currentVersion at runtime from the frontend of the application under test and passing it to the plugin for push.
b) Currently only the overall status of the test result can be passed. This is determined automatically. If a Katalon test is successful, the TestRail test is also successful or if a TestStep in Katalon fails, the overall status in TestRail is also set to “Failed”. You can’t see in TestRail which Step failed.
c) It should be possible to define in Katalon which Katalon test step/test step is to be assigned to which TestRail test step. On the one hand it should be possible to set the test result automatically, on the other hand it should be possible to overwrite it with an own status if desired. It should be possible to set the default states (Passed/Blocked/Failed/Retest) as well as the custom states (e.g. “Passed with Errors”) both on step level and on overall level. As a workaround, it would help if you could at least push a comment or screenshots on the overall level.
d) The highlight would be if you could also interact with a Defect plugin. That means that e.g. in case of a “Failed” status a Jira-Issue is created automatically and the ID of the Jira-Issue is pushed to TestRail together with the TestRail-Result. However, this would only be “nice to have”.
Even though the CLI tool is far from perfect, its feature set could be a good starting point for what the Katalon TestRail plugin should support out of the box. For example, a code first approach, with which TestRail cases could be automatically created and linked, would also be helpful.
It should be possible to jump from Katalon to a TestRail Case or vice versa.
If there are several Katalon tests for a TestCase, Results are alienated. If the first katalon test fails but a subsequent casekatalon test for the same test is passed, the overall result is “passed”. For the overall result it should be possible to consider the results of all Katalon tests.