TestRail Integration

Feedback & Reviews Documentation
21

Hello @Vitallica

If you have a Katalon Studio Enterprise license, please follow this document to download the plugin, put it in your project plugin’s folder and refresh the project.

Jass

22

Thank you, Jass

23

Hi there,
I have Katalon Runtime Engine license, and as it is said, TestRail Integration plugin is included into it. But in Katalon store in my account I see that trial license to the TestRail Integration plugin is expired. How can I activate it again?

24

Hello @irina.chervyakova

Katalon Runtime Engine only gives an ability to execute tests in console mode or CLI and integrate with CI/CD systems while plugins are inclusive in Katalon Studio Enterprise. Please subscribe to a Katalon Studio Enterprise to continue using the TestRail Integration plugin.

Jass

25

Hello Jass,

thank you for explanations. I have been guided by the following:

image
image911×396 36.8 KB

If Katalon Runtime Engine subscription does not allow using the plugin, it would be better not to mention it in the pricing description.

Regards,
Irina

26

Hi Irina

I should have explained our plugins’ pricing policy more clearly. I’m sorry for any inconvenience this may have caused.

After an Enterprise plugin’s trial period, you need a Katalon Studio Enterprise license to configure it in Katalon Studio. In your case, if you haven’t configured the integration with TestRail during the plugin’s trial, you need a Katalon Studio Enterprise license to perform/modify the configuration steps.

However, if you have completed the configuration, you can continue using the plugin with either Katalon Studio Enterprise or Katalon Runtime Engine license.

Feel free to reply should you still have any concerns.

Jass

27

Can we add more than one test case number in integration text box?

28

Hello @jitender.virk

Could you please answer the following information regarding your feature request?

  1. Given you’re allowed to map multiple TestRail test cases with a single Katalon test case
    When executing that Katalon single test case
    Then all linked TestRail test cases share the same test result. ( all passed or all failed)

    Is this scenario what you expect?

  2. Given you’re allowed to map multiple TestRail test cases with a single Katalon test case
    When querying Katalon Test Cases linked to a TestRail test run in Dynamic Test Suite
    Then return all Katalon test cases associated with the TestRail Test Case IDs in that test run

    Is this scenario what you expect?

Thanks in advance

Jass

29

Is there any way to send to the Testrail not only a test run status (Passed, Failed, etc) but an original error message (a part of message probably) if a test failed?

Thanks.

30

I was able to get this to work this evening, after some troubleshooting. The document is not very clear here and should be revised to make things clearer. Here are my lessons learned and solutions:

Setup:

  1. In Test Rail -> Enable the API
  2. In Katalon -> Install the Test Rail Plugin then:
  • Open Katalon Studio > from Katalon Studio menu bar > Project > Settings.
  • In the Project Settings window > Plugins > TestRail > Check Using TestRail to enable the integration.
  • Enter your TR creds and test the connection

Map Test Rail Cases with Katalon:

Test Rail

  • Open your test rail project and go To Test Cases
  • There will be an ID assocaited with each test case. IE: C3573, etc
  • Denote only the Numeric Values. You will not enter any letters in to Katalon. If you do, the ID will not be found.

Katalon Studio

  • Locate the test case that you wish to link to a TR Case.
  • Open the test case and then click on Integration at the bottom
  • Enter the NUMERIC only value from the test case identified earlier. IE: 3573
  • Save your changes

Create a new Test Run result in Test Rail from a Katalon Run

Katalon Studio:

  • Create a new Test Suite in Katalon
  • For the name of the test suite enter: S{TestCaseID} Test Case Description: IE: S3573 Super User Login Scenarios. NOTE: The "3573 appended to the S here is the same test case ID that I linked up above.
  • Add the test case within the test suite that you wish to run
  • Execute the test suite
  • When the test suite is completed, go back to Test Rail. You will see a Newly created test run for this listed.

Over write the test results in the newly created test run in Test rail:

After you follow the steps above, a newly created test run will be created by Katalon -> Test Rail. If you want to add to this newly created test run in Test Rail without created a new one:

  • Create a new Test Suite in Katalon
  • This one will have the name of R{TestRunID} Same Description
  • To obtain the test run ID, open the test run created in Test Rail by Katalon. At the top a URL will be listed similar to this: https:/testrailurl.io/index.php?/runs/view/598&group_by=cases:section_id&group_order=asc
  • In this example: “598” following “/runs/view” is the test run ID
  • Your new Test Suite in Katalon should be titled: R598 Super User Login Scenarios
  • Add in the test case that is linked to the test run
  • Execute the test suite.
  • Results will be appended to the test run in Test Rail instead of a newly created test run.

NOTE: One thing that really help was using the Event log in Katalon during my troubleshooting process.

1 Like
31

Update to my prior post:

NOTE: If you want to add more test cases to this suite you can, and they will be part of the test result:

  • Follow steps above to link additional Katalon tests with TR cases vis the TR integration in Katalon
  • Within the S3575 suite, add in your additional test cases in Katalon that are linked together
  • Kick off the test suite
  • Test Run will be created in test rail with multiple test cases and results.
  • You can then rename the test run whatever you want in Test Rail.
33

Hello @Kevin_Coffrin

We truly appreciate your contribution. Our team will update the document asap.

Happy testing!

Jass

35

Is there any chance that developers will provide a possibility to send not only a test run status, but an error/failed log (or just part of it) into Result Comment in TestRail using TestRail plugin?

36

Since Test Rail no longer have Test Suite, does this mean that 1 Test Case in Test Rail can only be linked to 1 Test Suite in Katalon??

37

I was able to generate the Test Run in Test Rail after running the Test Suite in Katalon however, the test case status or test run status are not sending to Test Rail after every test suite execution on Katalon.

1 Like
38

I connected everything and it’s working fine, but I can’t see the results of the test cases in the Test Rail Dashboard. Any suggestions?

39

We were using the integration for a long time and didn’t change our setup, but suddenly the results are not transferred to TestRail anymore after executing a TestSuite in Katalon.
I use an “S” in the beginning of my TestSuites name, so that a new TestRun will be created in TestRail after executing the TestSuite in Katalon. This works and all my desired testcases are incldued to this TestRun but without any result.

@Jass
Can you please fix the TestRail integration?
This is a big drawback for us, if the automatic transfer of the results is not working anymore.

1 Like
40

Hi @m.ott

There is a known issue with TestRail integration as the TestRail APIs have changed. Our development team is looking into the issue and will inform you of new update.

Jass

2 Likes