I am working with JIRA where I need to update the status of test cases post exeuction. I want to keep a global variable to change its value based on whether the test case passed or failed. Later I am calling an API to change the status based on what value this variable holds.
I want to avoid writing lines of code after every call to markPassed, markFailed or markFailedAndStop.
I was wondering if we can add any custom lines in these inbuilt methods?
You can make your own keyword that implement KeywordUtil.Markpassed
Here is the source code of com.kms.katalon.core.util.KeywordUtil
I was wondering if we can add any custom lines in these inbuilt methods?
Use Groovy’s metaprogramming feature. You can replace the implementation of markPassed method of com.kms.katalon.core.util.KeywordUtil class at runtime.
Learn the detail by reading this doc:
http://groovy-lang.org/metaprogramming.html#_methods
I already have a big regression suite and if I create a new method, it will be a big refactor activity. That’s the reason why I am looking to alter the behavior of inbuilt method.
In Katalon, where should I write my my new class with modified method?
What will be the format of invokeMethod() in that class?
No, you do not have to create a new class. Your tests will keep on using the com.kms.katalon.core.util.KeywordUtil class. No changes required for the callers = your existing tests.
You need to add a Groovy script which replaces the implementation of the markPassed method of com.kms.katalon.core.util.KeywordUtil class before the markPassed method is called by your tests. How to implement it? — in @beforeTestSuite-annotated method (or in @beforeTestCase-annotated method) of a TestListener class. You can write something like this:
KeywordUtil.metaClass.static.markPassed = { String message ->
// insert whatever lines of processing you want
// the following 2 lines are copy&pasted from the original implementation
logger.logPassed(message);
ErrorCollector.getCollector().setKeywordPassed(true);
// insert whatever lines of processing you want
}
Similarly You can replace the implementation of other methods markFailed and markFailedAndStop as well.
My sample code:
I changed and it worked. Thank you. Just one more question. When I am calling the markFailed after adding the implementation given by you, it is failing the test case and coloring it red in the html report. However, it is not failing the step (the step is still shown in green). do you know why. I am adding a screenshot
This is the markFailed method that I added in @BeforeTestCase:
KeywordUtil.metaClass.static.markFailed = { String message →
LocalDateTime localDateTime = LocalDateTime.now()
GlobalVariable.samplevariable = “abc”
println “Global variable sample variable changed”
ErrorCollector.getCollector().addError(new StepFailedException(message))
}
Can you please suggest how can I mark the step in red too?
Please check if the step WAS shown in red when you execute your test WITHOUT new @BeforeTestCase-annotated method. I guess that the step was shown in green as well.
Can you please suggest how can I mark the step in red too?
Since the TEST STEP markFailed("Failing intentionally") ran successful without error, I think it is appropriate to show it in green, not in red.
The test case was successfully marked FAILED as you wanted. Isn’t it enough?
Hi,
No, When I remove the markFailed implementation in the test listener, it shows the step in red as failed. I am adding both the screenshots.
Please note that if I implement the markFailed method in the listener, it is not even printing the failure message argument that I am passing while calling markFailed from the test case which is “Failing intentionally”.
This sounds like a scope issue. I don’t think test case code runs in the same scope as the code running in a listener - the listener is hooked up and runs before the test case ever exists.
But @kazurayam knows more about the startup code than I do – I’ll defer to his wisdom.
In short, I have no idea how to make TEST STEP shown in red in the Report while replacing the com.kms.katalon.core.util.KeywordUtil#markFailed method implementation using Groovy’s metaprogramming feature.
I tried comparing 2 cases: with or without replacing KeywordUtil#markFailed method, looked at how JUnit_Report.xml files are compiled by Katalon Studio.
in JUnit_Report.xml I saw the following output:
<testcase name="Test Cases/TC2" classname="Test Cases/TC2" status="FAILED">
<failure .../>
<system-out>...
2019-08-22 06:09:12 - [TEST_STEP][FAILED] - markFailed("called markFailed"): called markFailed
2019-08-22 06:09:12 - [MESSAGE][FAILED] - called markFailed</system-out>
<system-err>2019-08-22 06:09:12 - [TEST_CASE][FAILED] - Test Cases/TC2: Test Cases/TC2 FAILED.
Reason:
com.kms.katalon.core.exception.StepFailedException: called markFailed
at com.kms.katalon.core.util.KeywordUtil.markFailed(KeywordUtil.java:19)
at com.kms.katalon.core.util.KeywordUtil$markFailed.call(Unknown Source)
at TC2.run(TC2:5)
...</system-err>
</testcase>
in JUnit_Report.xml I saw the following output:
<testcase name="Test Cases/TC2" classname="Test Cases/TC2" status="FAILED">
<failure ... />
<system-out>...
2019-08-22 06:06:54 - [TEST_STEP][PASSED] - markFailed("called markFailed"): null</system-out>
<system-err>2019-08-22 06:06:54 - [TEST_CASE][FAILED] - Test Cases/TC2: Test Cases/TC2 FAILED.
Reason:
com.kms.katalon.core.exception.StepFailedException: called markFailed
at TL1$_beforeTestSuite_closure2.doCall(TL1.groovy:39)
at TC2.run(TC2:5)
...</system-err>
</testcase>
In the above 2 fragments, you can see a few slight differences.
Especially in the latter case, a null appears. It tells me, something wicked this way comes.
Why these 2 fragments are generated differently? — It solely depends on how the internal code of Katalon Studio (which generates JUnit_Report.xml and HTML report) is designed. I do not know its detail. I do not know if the source of code is publicly disclosed or not.
I hope Katalon Team to review the report-generating program — why null appears here?
Thank you very much for the detailed explanation. Is it possible to call the original method from overloaded method of markFailed with the message (parameter) passed from the function call?
I do not know. I have never wanted to do it. All I can say is that I need to read through The Apache Groovy programming language - Runtime and compile-time metaprogramming and try to find the answer.
@Rakesh_Kumar instead of overriding the method via metaprogramming you can try a different approach.
just define a custom keyword. do the needed logging into it and call (unmodified) markFailed at the end.
no longer use the listener
Hi @Ibus. Yes I thought of this idea. However, I already have a complete framework with a big suite collection. I will need to change in all the functions one by one (more than 1000).
Is is possible to define the operation from the custom class that you are referring into the overrided method and call the original method at the end of it?
Which of course is why you like the metaprogramming solution - you cover all your existing code in one shot.
I’m pretty sure my theory above holds. Your listener exists in a context beyond your test case context - it has come and gone by the time your TC is running so the override no longer exists (it’s a shame none of the devs are responding to shed some light here).
So… don’t use a listener.
You need to find the least painful way to add a wrapper over every test case. The wrapper can then instantiate the overridden markFailed method (or, like @Ibus said, do your own thing). With 1000+ TCs, that’s still a lot of work, but at least then you will have a way to do your own before/after code AND be running in TC context/scope.
Aside: I’ve made the same mistake (assuming the listener is in the same scope/context as the TC). A particular TC was throwing an exception but my “global” catch wasn’t seeing it. It took me a while to realize, the throw was happening in the context of the listener, not my global code. That hurt me badly for a good few hours while I was hunting it down. Hence why I’m pretty certain that’s your problem, too.