How to integrate Katalon Studio to TestRail to report test case results

Product Forums Katalon Studio
,
21

@ leszek.urbanik
@thebenluker

I’m getting the same NullPointerException as “leszek.urbanik”. My default profile has a value set for global variable G_testrail_run_id. The test script has id in single quotes. Is there a solution?

image
image986×350 14.5 KB

image
image1400×905 46.2 KB

22

No, not really. We just bought the plugin…

23

How do you get all of this to work in the plugin? I installed the plugin, setup the connection, and tested it, but it is not clear how I am supposed to actually use the plugin. There were no instructions on the plugin page.

1 Like
24

hi bro @Jayesh_Mohane for me it’s not working. Now i am trying that your tutorial. because i am trying that plugin it;s not working.
May you help?

25

What is the issue that you are having? I no longer use Katalon :frowning:

26

On TestRail not updated the status

but i get this

TestRail: Start sending summary message to channel:

Summary execution result of test suite: Test Suites/As a Brand/Advanced Regression Testing/Sign in Module

Total test cases: 1

Total passes: 1

Total failures: 0

Total errors: 0

Total skipped: 0

TestRail: Summary message has been successfully sent

Slack: Start sending summary message to channel: ui_automation

Slack: Summary message has been successfully sent

1 Like
27

2019-10-16 14:33:28.260 INFO c.k.katalon.core.main.TestSuiteExecutor - START Test Suites/As a Brand/Advanced Regression Testing/Sign in Module
2019-10-16 14:33:28.339 INFO c.k.katalon.core.main.TestSuiteExecutor - hostName =
2019-10-16 14:33:28.340 INFO c.k.katalon.core.main.TestSuiteExecutor - os = Mac OS X 64bit
2019-10-16 14:33:28.341 INFO c.k.katalon.core.main.TestSuiteExecutor - hostAddress =
2019-10-16 14:33:28.368 INFO c.k.katalon.core.main.TestSuiteExecutor - katalonVersion = 6.3.3.11
2019-10-16 14:33:28.697 INFO c.k.katalon.core.main.TestCaseExecutor - --------------------
2019-10-16 14:33:28.698 INFO c.k.katalon.core.main.TestCaseExecutor - START Test Cases/As a Brand/01. Sign in Module/1.2 Check sign in/TC01_Sign in with valid account
2019-10-16 14:33:28.912 INFO c.k.katalon.core.main.TestCaseExecutor - (Default) testrail_tc_id = 3
2019-10-16 14:33:29.157 DEBUG t.TC01_Sign in with valid account - 1: openBrowser("")
2019-10-16 14:33:29.367 INFO c.k.k.core.webui.driver.DriverFactory - Starting ‘Chrome’ driver
Oct 16, 2019 2:33:29 PM org.openqa.selenium.remote.DesiredCapabilities chrome
INFO: Using new ChromeOptions() is preferred to DesiredCapabilities.chrome()
2019-10-16 14:33:29.404 INFO c.k.k.core.webui.driver.DriverFactory - Action delay is set to 0 seconds
Starting ChromeDriver 76.0.3809.68 (420c9498db8ce8fcd190a954d51297672c1515d5-refs/branch-heads/3809@{#864}) on port 29471
Only local connections are allowed.
Please protect ports used by ChromeDriver and related test frameworks to prevent access by malicious code.
[1571211224.736][WARNING]: This version of ChromeDriver has not been tested with Chrome version 77.
Oct 16, 2019 2:33:45 PM org.openqa.selenium.remote.ProtocolHandshake createSession
INFO: Detected dialect: W3C
2019-10-16 14:33:45.127 INFO c.k.k.core.webui.driver.DriverFactory - sessionId = 40dacde7e034f6f34504d92c2dfe05a7
2019-10-16 14:33:45.160 INFO c.k.k.core.webui.driver.DriverFactory - browser = Chrome 77.0.3865.120
2019-10-16 14:33:45.161 INFO c.k.k.core.webui.driver.DriverFactory - platform = Mac OS X
2019-10-16 14:33:45.161 INFO c.k.k.core.webui.driver.DriverFactory - seleniumVersion = 3.141.59
2019-10-16 14:33:45.162 INFO c.k.k.core.webui.driver.DriverFactory - proxyInformation = ProxyInformation{proxyOption=USE_SYSTEM, proxyServerType=HTTP, password=, proxyServerAddress=, proxyServerPort=0}
2019-10-16 14:33:45.164 DEBUG t.TC01_Sign in with valid account - 2: navigateToUrl(G_SiteURL, STOP_ON_FAILURE)
2019-10-16 14:33:49.203 DEBUG t.TC01_Sign in with valid account - 3: setText(findTestObject(“Object Repository/Login Page/ts-hq-textfield-email”), emailbrand, STOP_ON_FAILURE)
2019-10-16 14:33:49.561 DEBUG t.TC01_Sign in with valid account - 4: setText(findTestObject(“Object Repository/Login Page/ts-hq-textfield-pwd”), password, STOP_ON_FAILURE)
2019-10-16 14:33:49.731 DEBUG t.TC01_Sign in with valid account - 5: click(findTestObject(“Object Repository/Login Page/ts-hq-button-signin”), STOP_ON_FAILURE)
2019-10-16 14:33:49.895 DEBUG t.TC01_Sign in with valid account - 6: delay(wait_to_medium, STOP_ON_FAILURE)
2019-10-16 14:33:54.964 DEBUG t.TC01_Sign in with valid account - 7: verifyTextPresent(“ITEMS”, false, STOP_ON_FAILURE)
2019-10-16 14:33:57.924 DEBUG t.TC01_Sign in with valid account - 8: verifyTextPresent(“Lite description about Item”, false, STOP_ON_FAILURE)
2019-10-16 14:33:58.159 ERROR c.k.k.core.context.internal.TestHooker - :x: org.codehaus.groovy.runtime.InvokerInvocationException: groovy.lang.MissingMethodException: No signature of method: java.lang.String.add() is applicable for argument types: (java.lang.String) values: [3]
Possible solutions: any(), any(groovy.lang.Closure), wait(), find(), wait(long), find(java.lang.CharSequence)
2019-10-16 14:33:58.162 INFO c.k.katalon.core.main.TestCaseExecutor - END Test Cases/As a Brand/01. Sign in Module/1.2 Check sign in/TC01_Sign in with valid account
2019-10-16 14:33:58.286 INFO c.k.katalon.core.main.TestCaseExecutor - --------------------
2019-10-16 14:33:58.286 INFO c.k.katalon.core.main.TestCaseExecutor - CALL Test Cases/TestRail_Update_Run_With_Results
2019-10-16 14:33:58.819 DEBUG t.TestRail_Update_Run_With_Results - 1: response = com.testrail.syncTestRail.get_tests(G_testrail_run_id)
2019-10-16 14:33:58.906 ERROR k.k.c.m.CustomKeywordDelegatingMetaClass - :x: No such property: WSBuiltInKeywords for class: com.testrail.syncTestRail
2019-10-16 14:33:58.909 ERROR c.k.katalon.core.main.TestCaseExecutor - :x: Test Cases/TestRail_Update_Run_With_Results FAILED.
Reason:
groovy.lang.MissingPropertyException: No such property: WSBuiltInKeywords for class: com.testrail.syncTestRail
at com.testrail.syncTestRail.get_tests(syncTestRail.groovy:49)
at com.testrail.syncTestRail.invokeMethod(syncTestRail.groovy)
at com.kms.katalon.core.main.CustomKeywordDelegatingMetaClass.invokeStaticMethod(CustomKeywordDelegatingMetaClass.java:50)
at TestRail_Update_Run_With_Results.run(TestRail_Update_Run_With_Results:17)
at com.kms.katalon.core.main.ScriptEngine.run(ScriptEngine.java:194)
at com.kms.katalon.core.main.ScriptEngine.runScriptAsRawText(ScriptEngine.java:119)
at com.kms.katalon.core.main.TestCaseExecutor.runScript(TestCaseExecutor.java:337)
at com.kms.katalon.core.main.TestCaseExecutor.doExecute(TestCaseExecutor.java:328)
at com.kms.katalon.core.main.TestCaseExecutor.processExecutionPhase(TestCaseExecutor.java:307)
at com.kms.katalon.core.main.TestCaseExecutor.accessMainPhase(TestCaseExecutor.java:299)
at com.kms.katalon.core.main.TestCaseExecutor.execute(TestCaseExecutor.java:233)
at com.kms.katalon.core.main.TestCaseMain.runTestCase(TestCaseMain.java:114)
at com.kms.katalon.core.keyword.builtin.CallTestCaseKeyword$_callTestCase_closure1.doCall(CallTestCaseKeyword.groovy:59)
at com.kms.katalon.core.keyword.builtin.CallTestCaseKeyword$_callTestCase_closure1.call(CallTestCaseKeyword.groovy)
at com.kms.katalon.core.keyword.internal.KeywordMain.runKeyword(KeywordMain.groovy:68)
at com.kms.katalon.core.keyword.builtin.CallTestCaseKeyword.callTestCase(CallTestCaseKeyword.groovy:81)
at com.kms.katalon.core.keyword.builtin.CallTestCaseKeyword.execute(CallTestCaseKeyword.groovy:44)
at com.kms.katalon.core.keyword.internal.KeywordExecutor.executeKeywordForPlatform(KeywordExecutor.groovy:56)
at com.kms.katalon.core.keyword.BuiltinKeywords.callTestCase(BuiltinKeywords.groovy:334)
at TestSuiteTestRailID.afterTestSuite(TestSuiteTestRailID.groovy:32)
at TestSuiteTestRailID.invokeMethod(TestSuiteTestRailID.groovy)
at com.kms.katalon.core.context.internal.TestHooker.invokeMethod(TestHooker.java:111)
at com.kms.katalon.core.context.internal.TestHooker.lambda$3(TestHooker.java:95)
at com.kms.katalon.core.context.internal.TestHooker.invokeContextMethods(TestHooker.java:94)
at com.kms.katalon.core.context.internal.TestListenerCollector.lambda$3(TestListenerCollector.java:88)
at com.kms.katalon.core.context.internal.TestListenerCollector.handleListenerEvent(TestListenerCollector.java:85)
at com.kms.katalon.core.context.internal.ExecutionEventManager.publicEvent(ExecutionEventManager.java:36)
at com.kms.katalon.core.main.TestSuiteExecutor.execute(TestSuiteExecutor.java:99)
at com.kms.katalon.core.main.TestCaseMain.startTestSuite(TestCaseMain.java:157)
at com.kms.katalon.core.main.TestCaseMain$startTestSuite$0.call(Unknown Source)
at TempTestSuite1571211205127.run(TempTestSuite1571211205127.groovy:35)

2019-10-16 14:33:58.915 INFO c.k.katalon.core.main.TestCaseExecutor - END CALL Test Cases/TestRail_Update_Run_With_Results
2019-10-16 14:33:58.915 INFO c.k.katalon.core.main.TestCaseExecutor - --------------------
2019-10-16 14:33:58.942 ERROR c.k.k.core.keyword.internal.KeywordMain - :x: Unable to call Test Case ‘Test Cases/TestRail_Update_Run_With_Results’ (Root cause: com.kms.katalon.core.exception.StepErrorException: Call Test Case ‘Test Cases/TestRail_Update_Run_With_Results’ failed because of error(s)
at com.kms.katalon.core.keyword.builtin.CallTestCaseKeyword$_callTestCase_closure1.doCall(CallTestCaseKeyword.groovy:66)
at com.kms.katalon.core.keyword.builtin.CallTestCaseKeyword$_callTestCase_closure1.call(CallTestCaseKeyword.groovy)
at com.kms.katalon.core.keyword.internal.KeywordMain.runKeyword(KeywordMain.groovy:68)
at com.kms.katalon.core.keyword.builtin.CallTestCaseKeyword.callTestCase(CallTestCaseKeyword.groovy:81)
at com.kms.katalon.core.keyword.builtin.CallTestCaseKeyword.execute(CallTestCaseKeyword.groovy:44)
at com.kms.katalon.core.keyword.internal.KeywordExecutor.executeKeywordForPlatform(KeywordExecutor.groovy:56)
at com.kms.katalon.core.keyword.BuiltinKeywords.callTestCase(BuiltinKeywords.groovy:334)
at TestSuiteTestRailID.afterTestSuite(TestSuiteTestRailID.groovy:32)
at TestSuiteTestRailID.invokeMethod(TestSuiteTestRailID.groovy)
at com.kms.katalon.core.context.internal.TestHooker.invokeMethod(TestHooker.java:111)
at com.kms.katalon.core.context.internal.TestHooker.lambda$3(TestHooker.java:95)
at com.kms.katalon.core.context.internal.TestHooker.invokeContextMethods(TestHooker.java:94)
at com.kms.katalon.core.context.internal.TestListenerCollector.lambda$3(TestListenerCollector.java:88)
at com.kms.katalon.core.context.internal.TestListenerCollector.handleListenerEvent(TestListenerCollector.java:85)
at com.kms.katalon.core.context.internal.ExecutionEventManager.publicEvent(ExecutionEventManager.java:36)
at com.kms.katalon.core.main.TestSuiteExecutor.execute(TestSuiteExecutor.java:99)
at com.kms.katalon.core.main.TestCaseMain.startTestSuite(TestCaseMain.java:157)
at com.kms.katalon.core.main.TestCaseMain$startTestSuite$0.call(Unknown Source)
at TempTestSuite1571211205127.run(TempTestSuite1571211205127.groovy:35)
Caused by: groovy.lang.MissingPropertyException: No such property: WSBuiltInKeywords for class: com.testrail.syncTestRail
at com.testrail.syncTestRail.get_tests(syncTestRail.groovy:49)
at com.testrail.syncTestRail.invokeMethod(syncTestRail.groovy)
at com.kms.katalon.core.main.CustomKeywordDelegatingMetaClass.invokeStaticMethod(CustomKeywordDelegatingMetaClass.java:50)
at TestRail_Update_Run_With_Results.run(TestRail_Update_Run_With_Results:17)
at com.kms.katalon.core.main.ScriptEngine.run(ScriptEngine.java:194)
at com.kms.katalon.core.main.ScriptEngine.runScriptAsRawText(ScriptEngine.java:119)
at com.kms.katalon.core.main.TestCaseExecutor.runScript(TestCaseExecutor.java:337)
at com.kms.katalon.core.main.TestCaseExecutor.doExecute(TestCaseExecutor.java:328)
at com.kms.katalon.core.main.TestCaseExecutor.processExecutionPhase(TestCaseExecutor.java:307)
at com.kms.katalon.core.main.TestCaseExecutor.accessMainPhase(TestCaseExecutor.java:299)
at com.kms.katalon.core.main.TestCaseExecutor.execute(TestCaseExecutor.java:233)
at com.kms.katalon.core.main.TestCaseMain.runTestCase(TestCaseMain.java:114)
at com.kms.katalon.core.keyword.builtin.CallTestCaseKeyword$_callTestCase_closure1.doCall(CallTestCaseKeyword.groovy:59)
… 18 more
)
2019-10-16 14:33:58.956 ERROR c.k.k.core.context.internal.TestHooker - :x: com.kms.katalon.core.exception.StepErrorException: Call Test Case ‘Test Cases/TestRail_Update_Run_With_Results’ failed because of error(s)
2019-10-16 14:33:59.210 INFO c.k.katalon.core.main.TestSuiteExecutor - --------------------
2019-10-16 14:33:59.211 INFO c.k.katalon.core.main.TestSuiteExecutor - END Test Suites/As a Brand/Advanced Regression Testing/Sign in Module
2019-10-16 14:33:59.212 INFO c.k.katalon.core.main.TestSuiteExecutor - ====================

28

10-18-2019 11:10:45 AM G_run_testrail_tc_id.add(tc_ids[n])

Elapsed time: 0.062s

org.codehaus.groovy.runtime.InvokerInvocationException: groovy.lang.MissingMethodException: No signature of method: java.lang.String.add() is applicable for argument types: (java.lang.String) values: []

Possible solutions: any(), any(groovy.lang.Closure), wait(), find(), wait(long), take(int)

Anyone can help?
@devalex88

29

Hi @januari

Can you express clearly what you’re trying to do, the expected behavior and what seems to be wrong. And please try to separate between plain text and code (you can wrap the code with triple single quote so that they display like

code

Instead of

code

30

Hi @ThanhTo, I hope you can directing me, Thank you

51
512094×818 246 KB
18
182166×848 265 KB
25
252170×836 301 KB
33
332180×848 219 KB

The point is I already following step by step like documentation, and I got errors like this image. I did not reduce or add the step expect URL my TestRail and Authorization.

defaultFile

15
151236×210 17.4 KB
32
321482×462 76.6 KB
41
411612×676 160 KB
56
561588×890 186 KB
08
081454×830 180 KB
15
151956×452 52.1 KB

32

Hi, I have been facing same problems as stated above. But there has been no solution yet. Can someone help please?

=============== ROOT CAUSE =====================
Caused by: java.lang.NullPointerException: Cannot get property ‘size’ on null object

07-29-2020 03:26:50 PM callTestCase(findTestCase(“Test Cases/SupplierPortal/TestRail_Update_Run_With_Results”), [:], CONTINUE_ON_FAILURE)

Elapsed time: 3.271s

Unable to call Test Case ‘Test Cases/SupplierPortal/TestRail_Update_Run_With_Results’ (Root cause: com.kms.katalon.core.exception.StepErrorException: Call Test Case ‘Test Cases/SupplierPortal/TestRail_Update_Run_With_Results’ failed because of error(s)
at com.kms.katalon.core.keyword.builtin.CallTestCaseKeyword$_callTestCase_closure1.doCall(CallTestCaseKeyword.groovy:66)
at com.kms.katalon.core.keyword.builtin.CallTestCaseKeyword$_callTestCase_closure1.call(CallTestCaseKeyword.groovy)
at com.kms.katalon.core.keyword.internal.KeywordMain.runKeyword(KeywordMain.groovy:68)

and so on…

33

Hi, I was wondering if this problem has already been resolved. because I have the same. I installed the plugin and followed all the steps from the documentaie exactly. Testrail does not process the summary of the output from Katalon.

34

At first TestRail plugin has to be installed in Katalon Studio. …
Select one Test case -> integration tab -> enter ID of test case in TestRail -> Save.
After running a Test Suite in Katalon, based on the Test Suite name, a corresponding Test Run will be created or updated in TestRail.

35

Hi Lewis, Thank you for your quick response. I have performed the following steps: 1. Plugin installed (connection succeeded), 2. Open a project in TestRail and test cases created, 3. In Katalon the number of the test cases linked, 4 in TestRail created a test run with specific test cases, 5. In Katalon test performed (name starts with S1). 6. In TestRail a run is created with S1 in the name. However the results are nog processed in TestRail.

36

I see so many of them facing this issue ‘Caused by: java.lang.NullPointerException: Cannot get property ‘size’ on null object’ and there is not solution yet.
I face the similar problem. Is there anyone who can help answer this?

37

Hi @Jayesh_Mohane ,
I’m getting 302 error code as response. Throwing exception at returning the test case ids in get_tests keyword.

020-10-29 14:42:43.545 ERROR k.k.c.m.CustomKeywordDelegatingMetaClass - :x: Text must not be null or empty
2020-10-29 14:42:43.549 ERROR c.k.katalon.core.main.TestCaseExecutor - :x: Test Cases/New Test Case FAILED.
Reason:
java.lang.IllegalArgumentException: Text must not be null or empty
at libCustomKeyword.testrailIntegration.get_tests(testrailIntegration.groovy:50)
at libCustomKeyword.testrailIntegration.invokeMethod(testrailIntegration.groovy)
at com.kms.katalon.core.main.CustomKeywordDelegatingMetaClass.invokeStaticMethod(CustomKeywordDelegatingMetaClass.java:50)
at New Test Case.run(New Test Case:18)
at com.kms.katalon.core.main.ScriptEngine.run(ScriptEngine.java:194)
at com.kms.katalon.core.main.ScriptEngine.runScriptAsRawText(ScriptEngine.java:119)
at com.kms.katalon.core.main.TestCaseExecutor.runScript(TestCaseExecutor.java:339)
at com.kms.katalon.core.main.TestCaseExecutor.doExecute(TestCaseExecutor.java:330)
at com.kms.katalon.core.main.TestCaseExecutor.processExecutionPhase(TestCaseExecutor.java:309)
at com.kms.katalon.core.main.TestCaseExecutor.accessMainPhase(TestCaseExecutor.java:301)
at com.kms.katalon.core.main.TestCaseExecutor.execute(TestCaseExecutor.java:235)
at com.kms.katalon.core.main.TestCaseMain.runTestCase(TestCaseMain.java:114)
at com.kms.katalon.core.keyword.builtin.CallTestCaseKeyword$_callTestCase_closure1.doCall(CallTestCaseKeyword.groovy:59)
at com.kms.katalon.core.keyword.builtin.CallTestCaseKeyword$_callTestCase_closure1.call(CallTestCaseKeyword.groovy)
at com.kms.katalon.core.keyword.internal.KeywordMain.runKeyword(KeywordMain.groovy:68)
at com.kms.katalon.core.keyword.builtin.CallTestCaseKeyword.callTestCase(CallTestCaseKeyword.groovy:81)
at com.kms.katalon.core.keyword.builtin.CallTestCaseKeyword.execute(CallTestCaseKeyword.groovy:44)
at com.kms.katalon.core.keyword.internal.KeywordExecutor.executeKeywordForPlatform(KeywordExecutor.groovy:73)
at com.kms.katalon.core.keyword.BuiltinKeywords.callTestCase(BuiltinKeywords.groovy:350)
at BaseTest.afterTestSuite(BaseTest.groovy:126)
at BaseTest.invokeMethod(BaseTest.groovy)
at com.kms.katalon.core.context.internal.TestHooker.invokeMethod(TestHooker.java:111)
at com.kms.katalon.core.context.internal.TestHooker.lambda$3(TestHooker.java:95)
at com.kms.katalon.core.context.internal.TestHooker.invokeContextMethods(TestHooker.java:94)
at com.kms.katalon.core.context.internal.TestListenerCollector.lambda$3(TestListenerCollector.java:88)
at com.kms.katalon.core.context.internal.TestListenerCollector.handleListenerEvent(TestListenerCollector.java:85)
at com.kms.katalon.core.context.internal.ExecutionEventManager.publicEvent(ExecutionEventManager.java:36)
at com.kms.katalon.core.main.TestSuiteExecutor.execute(TestSuiteExecutor.java:98)
at com.kms.katalon.core.main.TestCaseMain.startTestSuite(TestCaseMain.java:157)
at com.kms.katalon.core.main.TestCaseMain$startTestSuite$0.call(Unknown Source)
at TempTestSuite1603962725450.run(TempTestSuite1603962725450.groovy:39)

Do you have any idea how it can be resloved?

38

The answer to the null pointer exception is that session Authentication should be enabled in the Test rails using administrative settings. When I tried it manually I am getting this error.

image
image1478×97 2.85 KB

39

Hi h.Kramer,

I am facing the same issue. Were you able to get a solution?

Thanks,
Parth

40

Hi Parth.Jadeja, We no longer use TestRail. There were too many conflicts with Katalon. We have switched to devop azure test. And we don’t regret that.

41

Hi h.Kramer, Thanks for your response. I was able to debug the issue and use the plugin successfully. Just interested to know what kind of conflicts were you facing with Katalon & Testrail? It will be great if you could elaborate as we are embarking on this journey. If you wish to share your experience through an email please send an email to parthjadeja@gmail.com

Thanks,
Parth