How to integrate Katalon Studio to TestRail to report test case results

Thank you so much @Jayesh_Mohane. Your tutorial helped us understand more about TestRail and we have built a TestRail integration plugin based on these ideas.

2 Likes

HI @devalex88, that is great to hear and thank you for developing the plugin to make integration with TestRail easier.

1 Like

Thank you @Jayesh_Mohane for such a great tutorial.
But I occured to face a problem while accomplishing it:

response = get_tests(G_testrail_run_id) FAILED.
Reason:
java.lang.StringIndexOutOfBoundsException: String index out of range: -1

how do you think what could be wrong?

Hi @lubomir.pe,

No worries. Can you please send the whole error and the part of the code. I believe this will be in the long script test case and it needs to be the following:
image

1 Like

Additionally, I would strongly recommend that you download the TestRail Plugin that has been developed which is a lot easier than this script. IT can be downloaded from here:
https://store.katalon.com/product/13/TestRail-Integration

I have been using it and it is simple and fast to use.

Thanks,
Jayesh

2 Likes

@Jayesh_Mohane thank you, it totally worked for me
Did you manage to run testsuits with TestRail integration in Jenkins? I am asking because I also have issues with it, but I believe this is not the right topic, just curious.
If someone has this flow (Katalon-Testrail-Jenkins), I would highly appreciate any help
lubomir.peretyatko@gmail.com

Glad that has worked for you. Unfortunately, this is not something I have come across and have done :frowning: might be worth checking on Google to see if someone has come across this. We are now moving to AzureDevOps where I hope to continue using Katalon Studio to run test cases

1 Like

Hello all,

did anyone encountered such issue:

04-16-2019 01:10:46 PM for (def n : (0…response[id].size))

Elapsed time: 0.023s

Get_TC_STATUS.afterTestSuite:49

Get_TC_STATUS.invokeMethod:0

for (def n : (0…response[id].size)) FAILED.
Reason:
java.lang.NullPointerException: Cannot get property ‘size’ on null object
at TestRail_Update_Run_With_Results.run(TestRail_Update_Run_With_Results:22)
at com.kms.katalon.core.main.ScriptEngine.run(ScriptEngine.java:194)
at com.kms.katalon.core.main.ScriptEngine.runScriptAsRawText(ScriptEngine.java:119)
at com.kms.katalon.core.main.TestCaseExecutor.runScript(TestCaseExecutor.java:331)
at com.kms.katalon.core.main.TestCaseExecutor.doExecute(TestCaseExecutor.java:322)
at com.kms.katalon.core.main.TestCaseExecutor.processExecutionPhase(TestCaseExecutor.java:301)
at com.kms.katalon.core.main.TestCaseExecutor.accessMainPhase(TestCaseExecutor.java:293)
at com.kms.katalon.core.main.TestCaseExecutor.execute(TestCaseExecutor.java:227)
at com.kms.katalon.core.main.TestCaseMain.runTestCase(TestCaseMain.java:114)
at com.kms.katalon.core.keyword.builtin.CallTestCaseKeyword$_callTestCase_closure1.doCall(CallTestCaseKeyword.groovy:59)
at com.kms.katalon.core.keyword.builtin.CallTestCaseKeyword$_callTestCase_closure1.call(CallTestCaseKeyword.groovy)
at com.kms.katalon.core.keyword.internal.KeywordMain.runKeyword(KeywordMain.groovy:66)
at com.kms.katalon.core.keyword.builtin.CallTestCaseKeyword.callTestCase(CallTestCaseKeyword.groovy:81)
at com.kms.katalon.core.keyword.builtin.CallTestCaseKeyword.execute(CallTestCaseKeyword.groovy:44)
at com.kms.katalon.core.keyword.internal.KeywordExecutor.executeKeywordForPlatform(KeywordExecutor.groovy:56)
at com.kms.katalon.core.keyword.BuiltinKeywords.callTestCase(BuiltinKeywords.groovy:334)
at Get_TC_STATUS.afterTestSuite(Get_TC_STATUS.groovy:49)
at Get_TC_STATUS.invokeMethod(Get_TC_STATUS.groovy)
at com.kms.katalon.core.context.internal.TestHooker.invokeMethod(TestHooker.java:132)
at com.kms.katalon.core.context.internal.TestHooker.lambda$5(TestHooker.java:116)
at com.kms.katalon.core.context.internal.TestHooker.invokeContextMethods(TestHooker.java:115)
at com.kms.katalon.core.context.internal.TestListenerCollector.lambda$3(TestListenerCollector.java:60)
at com.kms.katalon.core.context.internal.TestListenerCollector.handleListenerEvent(TestListenerCollector.java:57)
at com.kms.katalon.core.context.internal.ExecutionEventManager.publicEvent(ExecutionEventManager.java:36)
at com.kms.katalon.core.main.TestSuiteExecutor.execute(TestSuiteExecutor.java:89)
at com.kms.katalon.core.main.TestCaseMain.startTestSuite(TestCaseMain.java:157)
at com.kms.katalon.core.main.TestCaseMain$startTestSuite$0.call(Unknown Source)
at TempTestSuite1555413021757.run(TempTestSuite1555413021757.groovy:35)

?

I’m struggling with it for some time and I’m afraid I’m out of ideas what possibly I can be doing wrong in here…

I tried TR Integration Plugin and it works well so it must be something with those scripts.

Thanks,
Leszek

Hi there,

I am getting something similar. The tests execute fine but fail to update TR run with results:

2019-04-16 08:47:58.711 INFO c.k.katalon.core.main.TestCaseExecutor - CALL Test Cases/TestRail_Update_Run_With_Results
2019-04-16 08:47:58.811 DEBUG t.TestRail_Update_Run_With_Results - 1: response = Testrail.get_tests(G_testrail_run_id)
2019-04-16 08:47:58.862 ERROR k.k.c.m.CustomKeywordDelegatingMetaClass - :x: Testrail
2019-04-16 08:47:58.873 ERROR c.k.katalon.core.main.TestCaseExecutor - :x: response = Testrail.get_tests(G_testrail_run_id) FAILED.
Reason:
java.lang.ClassNotFoundException: Testrail
at com.kms.katalon.core.main.CustomKeywordDelegatingMetaClass.getCustomKeywordClassAndSetMetaClass(CustomKeywordDelegatingMetaClass.java:98)
at com.kms.katalon.core.main.CustomKeywordDelegatingMetaClass.invokeStaticMethod(CustomKeywordDelegatingMetaClass.java:44)
at TestRail_Update_Run_With_Results.run(TestRail_Update_Run_With_Results:18)
at com.kms.katalon.core.main.ScriptEngine.run(ScriptEngine.java:194)
at com.kms.katalon.core.main.ScriptEngine.runScriptAsRawText(ScriptEngine.java:119)
at com.kms.katalon.core.main.TestCaseExecutor.runScript(TestCaseExecutor.java:342)
at com.kms.katalon.core.main.TestCaseExecutor.doExecute(TestCaseExecutor.java:333)
at com.kms.katalon.core.main.TestCaseExecutor.processExecutionPhase(TestCaseExecutor.java:312)
at com.kms.katalon.core.main.TestCaseExecutor.accessMainPhase(TestCaseExecutor.java:304)
at com.kms.katalon.core.main.TestCaseExecutor.execute(TestCaseExecutor.java:238)
at com.kms.katalon.core.main.TestCaseMain.runTestCase(TestCaseMain.java:114)
at com.kms.katalon.core.keyword.builtin.CallTestCaseKeyword$_callTestCase_closure1.doCall(CallTestCaseKeyword.groovy:59)
at com.kms.katalon.core.keyword.builtin.CallTestCaseKeyword$_callTestCase_closure1.call(CallTestCaseKeyword.groovy)
at com.kms.katalon.core.keyword.internal.KeywordMain.runKeyword(KeywordMain.groovy:66)
at com.kms.katalon.core.keyword.builtin.CallTestCaseKeyword.callTestCase(CallTestCaseKeyword.groovy:81)
at com.kms.katalon.core.keyword.builtin.CallTestCaseKeyword.execute(CallTestCaseKeyword.groovy:44)
at com.kms.katalon.core.keyword.internal.KeywordExecutor.executeKeywordForPlatform(KeywordExecutor.groovy:56)
at com.kms.katalon.core.keyword.BuiltinKeywords.callTestCase(BuiltinKeywords.groovy:334)
at GET_TC_STATUS.afterTestSuite(GET_TC_STATUS.groovy:45)
at GET_TC_STATUS.invokeMethod(GET_TC_STATUS.groovy)
at com.kms.katalon.core.context.internal.TestHooker.invokeMethod(TestHooker.java:109)
at com.kms.katalon.core.context.internal.TestHooker.lambda$3(TestHooker.java:93)
at com.kms.katalon.core.context.internal.TestHooker.invokeContextMethods(TestHooker.java:92)
at com.kms.katalon.core.context.internal.TestListenerCollector.lambda$3(TestListenerCollector.java:88)
at com.kms.katalon.core.context.internal.TestListenerCollector.handleListenerEvent(TestListenerCollector.java:85)
at com.kms.katalon.core.context.internal.ExecutionEventManager.publicEvent(ExecutionEventManager.java:36)
at com.kms.katalon.core.main.TestSuiteExecutor.execute(TestSuiteExecutor.java:89)
at com.kms.katalon.core.main.TestCaseMain.startTestSuite(TestCaseMain.java:157)
at com.kms.katalon.core.main.TestCaseMain$startTestSuite$0.call(Unknown Source)
at TempTestSuite1555426068992.run(TempTestSuite1555426068992.groovy:35)

2019-04-16 08:47:58.876 ERROR c.k.katalon.core.main.TestCaseExecutor - :x: Test Cases/TestRail_Update_Run_With_Results FAILED.
Reason:
java.lang.ClassNotFoundException: Testrail
at com.kms.katalon.core.main.CustomKeywordDelegatingMetaClass.getCustomKeywordClassAndSetMetaClass(CustomKeywordDelegatingMetaClass.java:98)
at com.kms.katalon.core.main.CustomKeywordDelegatingMetaClass.invokeStaticMethod(CustomKeywordDelegatingMetaClass.java:44)
at TestRail_Update_Run_With_Results.run(TestRail_Update_Run_With_Results:18)
at com.kms.katalon.core.main.ScriptEngine.run(ScriptEngine.java:194)
at com.kms.katalon.core.main.ScriptEngine.runScriptAsRawText(ScriptEngine.java:119)
at com.kms.katalon.core.main.TestCaseExecutor.runScript(TestCaseExecutor.java:342)
at com.kms.katalon.core.main.TestCaseExecutor.doExecute(TestCaseExecutor.java:333)
at com.kms.katalon.core.main.TestCaseExecutor.processExecutionPhase(TestCaseExecutor.java:312)
at com.kms.katalon.core.main.TestCaseExecutor.accessMainPhase(TestCaseExecutor.java:304)
at com.kms.katalon.core.main.TestCaseExecutor.execute(TestCaseExecutor.java:238)
at com.kms.katalon.core.main.TestCaseMain.runTestCase(TestCaseMain.java:114)
at com.kms.katalon.core.keyword.builtin.CallTestCaseKeyword$_callTestCase_closure1.doCall(CallTestCaseKeyword.groovy:59)
at com.kms.katalon.core.keyword.builtin.CallTestCaseKeyword$_callTestCase_closure1.call(CallTestCaseKeyword.groovy)
at com.kms.katalon.core.keyword.internal.KeywordMain.runKeyword(KeywordMain.groovy:66)
at com.kms.katalon.core.keyword.builtin.CallTestCaseKeyword.callTestCase(CallTestCaseKeyword.groovy:81)
at com.kms.katalon.core.keyword.builtin.CallTestCaseKeyword.execute(CallTestCaseKeyword.groovy:44)
at com.kms.katalon.core.keyword.internal.KeywordExecutor.executeKeywordForPlatform(KeywordExecutor.groovy:56)
at com.kms.katalon.core.keyword.BuiltinKeywords.callTestCase(BuiltinKeywords.groovy:334)
at GET_TC_STATUS.afterTestSuite(GET_TC_STATUS.groovy:45)
at GET_TC_STATUS.invokeMethod(GET_TC_STATUS.groovy)
at com.kms.katalon.core.context.internal.TestHooker.invokeMethod(TestHooker.java:109)
at com.kms.katalon.core.context.internal.TestHooker.lambda$3(TestHooker.java:93)
at com.kms.katalon.core.context.internal.TestHooker.invokeContextMethods(TestHooker.java:92)
at com.kms.katalon.core.context.internal.TestListenerCollector.lambda$3(TestListenerCollector.java:88)
at com.kms.katalon.core.context.internal.TestListenerCollector.handleListenerEvent(TestListenerCollector.java:85)
at com.kms.katalon.core.context.internal.ExecutionEventManager.publicEvent(ExecutionEventManager.java:36)
at com.kms.katalon.core.main.TestSuiteExecutor.execute(TestSuiteExecutor.java:89)
at com.kms.katalon.core.main.TestCaseMain.startTestSuite(TestCaseMain.java:157)
at com.kms.katalon.core.main.TestCaseMain$startTestSuite$0.call(Unknown Source)
at TempTestSuite1555426068992.run(TempTestSuite1555426068992.groovy:35)

2019-04-16 08:47:58.877 INFO c.k.katalon.core.main.TestCaseExecutor - END CALL Test Cases/TestRail_Update_Run_With_Results
2019-04-16 08:47:58.877 INFO c.k.katalon.core.main.TestCaseExecutor - --------------------
2019-04-16 08:47:58.881 ERROR c.k.k.core.keyword.internal.KeywordMain - :x: Unable to call Test Case ‘Test Cases/TestRail_Update_Run_With_Results’ (Root cause: com.kms.katalon.core.exception.StepErrorException: Call Test Case ‘Test Cases/TestRail_Update_Run_With_Results’ failed because of error(s)
at com.kms.katalon.core.keyword.builtin.CallTestCaseKeyword$_callTestCase_closure1.doCall(CallTestCaseKeyword.groovy:66)
at com.kms.katalon.core.keyword.builtin.CallTestCaseKeyword$_callTestCase_closure1.call(CallTestCaseKeyword.groovy)
at com.kms.katalon.core.keyword.internal.KeywordMain.runKeyword(KeywordMain.groovy:66)
at com.kms.katalon.core.keyword.builtin.CallTestCaseKeyword.callTestCase(CallTestCaseKeyword.groovy:81)
at com.kms.katalon.core.keyword.builtin.CallTestCaseKeyword.execute(CallTestCaseKeyword.groovy:44)
at com.kms.katalon.core.keyword.internal.KeywordExecutor.executeKeywordForPlatform(KeywordExecutor.groovy:56)
at com.kms.katalon.core.keyword.BuiltinKeywords.callTestCase(BuiltinKeywords.groovy:334)
at GET_TC_STATUS.afterTestSuite(GET_TC_STATUS.groovy:45)
at GET_TC_STATUS.invokeMethod(GET_TC_STATUS.groovy)
at com.kms.katalon.core.context.internal.TestHooker.invokeMethod(TestHooker.java:109)
at com.kms.katalon.core.context.internal.TestHooker.lambda$3(TestHooker.java:93)
at com.kms.katalon.core.context.internal.TestHooker.invokeContextMethods(TestHooker.java:92)
at com.kms.katalon.core.context.internal.TestListenerCollector.lambda$3(TestListenerCollector.java:88)
at com.kms.katalon.core.context.internal.TestListenerCollector.handleListenerEvent(TestListenerCollector.java:85)
at com.kms.katalon.core.context.internal.ExecutionEventManager.publicEvent(ExecutionEventManager.java:36)
at com.kms.katalon.core.main.TestSuiteExecutor.execute(TestSuiteExecutor.java:89)
at com.kms.katalon.core.main.TestCaseMain.startTestSuite(TestCaseMain.java:157)
at com.kms.katalon.core.main.TestCaseMain$startTestSuite$0.call(Unknown Source)
at TempTestSuite1555426068992.run(TempTestSuite1555426068992.groovy:35)
Caused by: java.lang.ClassNotFoundException: Testrail
at com.kms.katalon.core.main.CustomKeywordDelegatingMetaClass.getCustomKeywordClassAndSetMetaClass(CustomKeywordDelegatingMetaClass.java:98)
at com.kms.katalon.core.main.CustomKeywordDelegatingMetaClass.invokeStaticMethod(CustomKeywordDelegatingMetaClass.java:44)
at TestRail_Update_Run_With_Results.run(TestRail_Update_Run_With_Results:18)
at com.kms.katalon.core.main.ScriptEngine.run(ScriptEngine.java:194)
at com.kms.katalon.core.main.ScriptEngine.runScriptAsRawText(ScriptEngine.java:119)
at com.kms.katalon.core.main.TestCaseExecutor.runScript(TestCaseExecutor.java:342)
at com.kms.katalon.core.main.TestCaseExecutor.doExecute(TestCaseExecutor.java:333)
at com.kms.katalon.core.main.TestCaseExecutor.processExecutionPhase(TestCaseExecutor.java:312)
at com.kms.katalon.core.main.TestCaseExecutor.accessMainPhase(TestCaseExecutor.java:304)
at com.kms.katalon.core.main.TestCaseExecutor.execute(TestCaseExecutor.java:238)
at com.kms.katalon.core.main.TestCaseMain.runTestCase(TestCaseMain.java:114)
at com.kms.katalon.core.keyword.builtin.CallTestCaseKeyword$_callTestCase_closure1.doCall(CallTestCaseKeyword.groovy:59)
… 18 more
)

I was following this guide:

I urgently need some help to resolve this.

thank you

1 Like

Hi @leszek.urbanik,

Sorry for the late reply - I think I know where the problem is. Can you please check your long script Test Case where the long code is…
Looking at where it is failing, I can see that you are missing ’ ’ in [id]. This is where it is failing:

image

It should be the following in your script:
_for (def n : (0…response[id].size)) {_

Let me know if that works for you.

Thank you

Please see this response.

Hello @Jayesh_Mohane

no problem. Thanks for the answer.

Unfortunately it’s the response which is missing ’ ’ signs. They are present in the script:

Any other ideas? :slight_smile:

Hi @leszek.urbanik,

Seems to be working fine for me. Can you please confirm if you have G_testrail_run_id set in the default profile and given it the value of whatever the Test Plan ID is from TestRail?

Yes, it’s there.

Hi @Jayesh_Mohane, Thanks for the tutorial. It has helped a lot.

But I had followed exactly the same steps mentioned by you, but I am facing an issue which is like -

Test Cases/TestRail_Update_Run_With_Results FAILED.

Reason:

java.lang.StringIndexOutOfBoundsException: String index out of range: -1

at com.kms.katalon.core.main.CustomKeywordDelegatingMetaClass.invokeStaticMethod(CustomKeywordDelegatingMetaClass.java:43)

at TestRail_Update_Run_With_Results.run(TestRail_Update_Run_With_Results:18)

at com.kms.katalon.core.main.ScriptEngine.run(ScriptEngine.java:194)

at com.kms.katalon.core.main.ScriptEngine.runScriptAsRawText(ScriptEngine.java:119)

at com.kms.katalon.core.main.TestCaseExecutor.runScript(TestCaseExecutor.java:336)

at com.kms.katalon.core.main.TestCaseExecutor.doExecute(TestCaseExecutor.java:327)

at com.kms.katalon.core.main.TestCaseExecutor.processExecutionPhase(TestCaseExecutor.java:306)

at com.kms.katalon.core.main.TestCaseExecutor.accessMainPhase(TestCaseExecutor.java:298)

at com.kms.katalon.core.main.TestCaseExecutor.execute(TestCaseExecutor.java:232)

at com.kms.katalon.core.main.TestCaseMain.runTestCase(TestCaseMain.java:114)

at com.kms.katalon.core.keyword.builtin.CallTestCaseKeyword$_callTestCase_closure1.doCall(CallTestCaseKeyword.groovy:59)

at com.kms.katalon.core.keyword.builtin.CallTestCaseKeyword$_callTestCase_closure1.call(CallTestCaseKeyword.groovy)

at com.kms.katalon.core.keyword.internal.KeywordMain.runKeyword(KeywordMain.groovy:66)

at com.kms.katalon.core.keyword.builtin.CallTestCaseKeyword.callTestCase(CallTestCaseKeyword.groovy:81)

at com.kms.katalon.core.keyword.builtin.CallTestCaseKeyword.execute(CallTestCaseKeyword.groovy:44)

at com.kms.katalon.core.keyword.internal.KeywordExecutor.executeKeywordForPlatform(KeywordExecutor.groovy:56)

at com.kms.katalon.core.keyword.BuiltinKeywords.callTestCase(BuiltinKeywords.groovy:334)

at Get_TC_STATUS.afterTestSuite(Get_TC_STATUS.groovy:44)

at Get_TC_STATUS.invokeMethod(Get_TC_STATUS.groovy)

at com.kms.katalon.core.context.internal.TestHooker.invokeMethod(TestHooker.java:109)

at com.kms.katalon.core.context.internal.TestHooker.lambda$3(TestHooker.java:93)

at com.kms.katalon.core.context.internal.TestHooker.invokeContextMethods(TestHooker.java:92)

at com.kms.katalon.core.context.internal.TestListenerCollector.lambda$3(TestListenerCollector.java:88)

at com.kms.katalon.core.context.internal.TestListenerCollector.handleListenerEvent(TestListenerCollector.java:85)

at com.kms.katalon.core.context.internal.ExecutionEventManager.publicEvent(ExecutionEventManager.java:36)

at com.kms.katalon.core.main.TestSuiteExecutor.execute(TestSuiteExecutor.java:89)

at com.kms.katalon.core.main.TestCaseMain.startTestSuite(TestCaseMain.java:157)

at com.kms.katalon.core.main.TestCaseMain$startTestSuite$0.call(Unknown Source)

at TempTestSuite1558947967342.run(TempTestSuite1558947967342.groovy:35)

I am not able to understand exactly what is the issues here… can you kindly help me what is the issue here?

Thanks in advance!

Running integration with katalon 6.2 and got below issue

Test Cases/TestRail_Update_Run_With_Results FAILED.
Reason:
org.codehaus.groovy.runtime.metaclass.MissingMethodExceptionNoStack: No signature of method: Testrail.get_tests() is applicable for argument types: (java.lang.String) values: [1]
Possible solutions: getClass(), getAt(java.lang.String)

@Jayesh_Mohane please help

thank you

Did you ever solve this? Got the exact same problem over here. Thanks

@ leszek.urbanik
@thebenluker

I’m getting the same NullPointerException as “leszek.urbanik”. My default profile has a value set for global variable G_testrail_run_id. The test script has id in single quotes. Is there a solution?

No, not really. We just bought the plugin…

How do you get all of this to work in the plugin? I installed the plugin, setup the connection, and tested it, but it is not clear how I am supposed to actually use the plugin. There were no instructions on the plugin page.

1 Like