Poor error diagnostics when Cucumber Feature is problematic

I am studying BDD in Katalon Studio version 5.10.1 on Windows 7.

I made a Github repository here: with git tag dev0.3

When I run ‘Include/Features/Calculator.feature’, it fails with the following message:

2019-01-10 11:43:37.134 e[34mINFO e[0;39m e[36mc.k.k.core.main.WSVerificationExecutor   -e[0;39m e[39m--------------------e[0;39m
2019-01-10 11:43:37.150 e[34mINFO e[0;39m e[36mc.k.k.core.main.WSVerificationExecutor   -e[0;39m e[39mSTART Verificatione[0;39m
2019-01-10 11:43:37.586 e[34mINFO e[0;39m e[36mc.k.k.c.c.k.CucumberBuiltinKeywords      -e[0;39m e[39mStarting run keyword runFeatureFile: 'Include/features/Calculator.feature' and extract report to folder: 'C:\Users\qcq0264\AppData\Local\Temp\Katalon\Include\features\Calculator.feature\20190110_114333/cucumber_report/1547088217571'...e[0;39m
2019-01-10 11:43:38.431 e[34mINFO e[0;39m e[36mc.k.k.c.c.keyword.CucumberReporter       -e[0;39m e[39m--------------------e[0;39m
2019-01-10 11:43:38.431 e[34mINFO e[0;39m e[36mc.k.k.c.c.keyword.CucumberReporter       -e[0;39m e[39mSTART SCENARIO Adde[0;39m
2019-01-10 11:43:38.465 e[1;31mERRORe[0;39m e[36mc.k.k.core.keyword.internal.KeywordMain  -e[0;39m e[31m❌ Keyword runFeatureFile was failed (Root cause: java.lang.UnsupportedOperationException)e[0;39m
2019-01-10 11:43:38.497 e[1;31mERRORe[0;39m e[36mc.k.k.core.main.WSVerificationExecutor   -e[0;39m e[31m❌ Verification FAILED.e[0;39m
e[31mReason:e[0;39m
e[31mcom.kms.katalon.core.exception.StepFailedException: Keyword runFeatureFile was failed (Root cause: java.lang.UnsupportedOperationException)e[0;39m
e[31m	at com.kms.katalon.core.keyword.internal.KeywordMain.stepFailed(KeywordMain.groovy:36)e[0;39m
e[31m	at com.kms.katalon.core.keyword.internal.KeywordMain.stepFailed(KeywordMain.groovy)e[0;39m
e[31m	at com.kms.katalon.core.keyword.internal.KeywordMain.runKeyword(KeywordMain.groovy:56)e[0;39m
e[31m	at com.kms.katalon.core.keyword.internal.KeywordMain$runKeyword.call(Unknown Source)e[0;39m
e[31m	at com.kms.katalon.core.cucumber.keyword.CucumberBuiltinKeywords.runFeatureFile(CucumberBuiltinKeywords.groovy:46)e[0;39m
e[31m	at com.kms.katalon.core.cucumber.keyword.CucumberBuiltinKeywords$runFeatureFile$0.callStatic(Unknown Source)e[0;39m
e[31m	at com.kms.katalon.core.cucumber.keyword.CucumberBuiltinKeywords.runFeatureFile(CucumberBuiltinKeywords.groovy:101)e[0;39m
e[31m	at com.kms.katalon.core.cucumber.keyword.CucumberBuiltinKeywords$runFeatureFile.call(Unknown Source)e[0;39m
e[31m	at WSVerification1547088217378.run(WSVerification1547088217378:2)e[0;39m
e[31m	at com.kms.katalon.core.main.ScriptEngine.run(ScriptEngine.java:194)e[0;39m
e[31m	at com.kms.katalon.core.main.ScriptEngine.runScriptAsRawText(ScriptEngine.java:119)e[0;39m
e[31m	at com.kms.katalon.core.main.WSVerificationExecutor.runScript(WSVerificationExecutor.java:146)e[0;39m
e[31m	at com.kms.katalon.core.main.WSVerificationExecutor.doExecute(WSVerificationExecutor.java:140)e[0;39m
e[31m	at com.kms.katalon.core.main.WSVerificationExecutor.processExecutionPhase(WSVerificationExecutor.java:123)e[0;39m
e[31m	at com.kms.katalon.core.main.WSVerificationExecutor.accessMainPhase(WSVerificationExecutor.java:115)e[0;39m
e[31m	at com.kms.katalon.core.main.WSVerificationExecutor.execute(WSVerificationExecutor.java:103)e[0;39m
e[31m	at com.kms.katalon.core.main.TestCaseMain.runFeatureFile(TestCaseMain.java:136)e[0;39m
e[31m	at com.kms.katalon.core.main.TestCaseMain$runFeatureFile$0.call(Unknown Source)e[0;39m
e[31m	at TempTempCase1547088213959.run(TempTempCase1547088213959.groovy:21)e[0;39m
2019-01-10 11:43:38.497 e[34mINFO e[0;39m e[36mc.k.k.core.main.WSVerificationExecutor   -e[0;39m e[39mEND Verificatione[0;39m

There would be something wrong in my Feature files and Step Definitions. I am responsible for it.

But I would argue that the error message above is NOT informative at all. It does not tell me the reason why UnsupportedOperationException was raised.

I looked at the 'C:\Users\qcq0264\AppData\Local\Temp\Katalon\Include\features\Calculator.feature\20190110_114333/cucumber_report/1547088217571' filder, which appers in the above message. But I could not find any useful information.

Through some hours of BDD feature in Katalon Studio, I have got a feeling that the error diagnostics by Katalon Studio for errors reported Cucumber requires much more development. I got frustrated by poor messages.

2 Likes

Hi @kazurayam,

We will log it as ticket to display more helpful messages.

Thanks for your investigation.

I looked at the line:

I looked at the source code of CucumberBuiltinKeywords:

I think the runFeatureFile method should print the stack trace of UnsupportedOperationException.

1 Like

Similar to this issue:

in that a stack trace information is consealed by Katalon Studio Keyword runner

1 Like

I added a Test Case which runs Calculator.feature: Test Cases/FeatureRunners/RunCalculatorFeater

This test case overrides the runKeyword method of the com.kms.katalon.core.keyword.internal.KeywordMain class using Grovvy’s ExpandoMetaClass.

When I run the test case I got the following Stack trace printed.

java.lang.UnsupportedOperationException
	at cucumber.runner.HookTestStep.getStepText(HookTestStep.java:68)
	at com.kms.katalon.core.cucumber.keyword.CucumberReporter.getStepText(CucumberReporter.java:119)
	at com.kms.katalon.core.cucumber.keyword.CucumberReporter.access$3(CucumberReporter.java:114)
	at com.kms.katalon.core.cucumber.keyword.CucumberReporter$5.receive(CucumberReporter.java:74)
	at com.kms.katalon.core.cucumber.keyword.CucumberReporter$5.receive(CucumberReporter.java:1)
	at cucumber.runner.EventBus.send(EventBus.java:28)
	at cucumber.runner.TestStep.run(TestStep.java:38)
	at cucumber.runner.TestCase.run(TestCase.java:41)
	at cucumber.runner.Runner.runPickle(Runner.java:44)
	at cucumber.runtime.Runtime.runFeature(Runtime.java:120)
	at cucumber.runtime.Runtime.run(Runtime.java:106)
	at cucumber.api.cli.Main.run(Main.java:35)
	at cucumber.api.cli.Main$run.call(Unknown Source)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:133)
	at com.kms.katalon.core.cucumber.keyword.CucumberBuiltinKeywords$_runFeatureFile_closure1.doCall(CucumberBuiltinKeywords.groovy:76)
	at com.kms.katalon.core.cucumber.keyword.CucumberBuiltinKeywords$_runFeatureFile_closure1.doCall(CucumberBuiltinKeywords.groovy)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1024)
	at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:117)
	at Script1547093701636$_run_closure1.doCall(Script1547093701636.groovy:8)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1024)
	at groovy.lang.Closure.call(Closure.java:414)
	at org.codehaus.groovy.runtime.metaclass.ClosureStaticMetaMethod.invoke(ClosureStaticMetaMethod.java:62)
	at org.codehaus.groovy.runtime.callsite.StaticMetaMethodSite$StaticMetaMethodSiteNoUnwrapNoCoerce.invoke(StaticMetaMethodSite.java:151)
	at org.codehaus.groovy.runtime.callsite.StaticMetaMethodSite.call(StaticMetaMethodSite.java:91)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:141)
	at com.kms.katalon.core.cucumber.keyword.CucumberBuiltinKeywords.runFeatureFile(CucumberBuiltinKeywords.groovy:46)
	at com.kms.katalon.core.cucumber.keyword.CucumberBuiltinKeywords$runFeatureFile$0.callStatic(Unknown Source)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallStatic(CallSiteArray.java:56)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callStatic(AbstractCallSite.java:194)2019-01-10 13:22:46.285 e[1;31mERRORe[0;39m e[36mc.k.k.core.keyword.internal.KeywordMain  -e[0;39m e[31m❌ Keyword runFeatureFile was failed (Root cause: java.lang.UnsupportedOperationException)e[0;39m

This stack trace guides me that I should look at cucumber.runner.HookTestStep.getStepText(HookTestStep.java:68).

Well, I have to dive even deeper.

1 Like

I found the source of cucumber.runner.HookTestStep on GitHub

Wow, in the HEAD of the MASTER branch, HookTestStep has only 18 lines. But I saw a string (HookTestStep.java:68).

Is the cucumber-jvm bundled in Katalon Studio 5.10.1 old?

1 Like

I found Kataln Studio 5.10.1 bundles io.cucumber.core.3.0.2.jar (May, 2018)

On the other hand, the latest version of cucumber-jvm is 4.2.0 (Oct, 2018)

We seem to have a significant gap of versions.

1 Like

Hi @kazurayam, indeed we are still using the old library, therefore the deprecated method getStepText threw the UnsupportedOperationException, we will update the library in upcoming release, thanks for the investigation !

1 Like

Thanh To,

Thank you for your response,

Do you mean you will change com.kms.katalon.core.cucumber.keyword.CucumberReporter.getStepText() so that it would not call cucumber.runner.HookTestStep.getStepText() which is already deprecated?

How long would it take for you to fix this problem and release the new Katalon Studio version? days, weeks, months? Depending on it, I would reconsider the priorities of my current tasks.

1 Like

Hi @ThanhTo, we are currently considering using the Cucumber implementation in Katalon Studio for some testing requirements in our company. Can you now say when the integration of the new version of Cucumber will come? Or is this already the case with the current version 6.0.6?

1 Like

Hi @kazurayam,

if you, like me, wanted to use the @Before and @After hooks of Cucumber (which seems not to be possible in Katalon Studio at the moment), the Cucumber Runner should help you, which you can simply call from a normal test case, see in this video tutorial.

This gives you access to all the usual possibilities built into Katalon Studio for setup and teardown actions,
e.g. starting the web browser before executing several Cucumber tests and close it afterwards. Normally you should not even have to use TestListeners for this. All activities required can also be noted directly before and after calling the Cucumber Runner. The test case code would be quite simple as:

WebUI.openBrowser('')
WebUI.maximizeWindow()
WebUI.deleteAllCookies()
CucumberKW.runWithCucumberRunner(CucumberRunner.class)
WebUI.closeBrowser()

While your Runner class can look simply like this:

import org.junit.runner.RunWith
import cucumber.api.CucumberOptions
import cucumber.api.junit.Cucumber

@RunWith(Cucumber.class)
@CucumberOptions(
    features="Include/features",
    glue="",
    tags="@MySomkeTests",
    plugin=["pretty", "html:ReportFolder"]
)
public class CucumberRunner {
}

Hope that helps you.

3 Likes