This is a companion discussion topic for the original entry at https://docs.katalon.com/katalon-studio/docs/cucumber-kw-run-feature-file-tag.html
This is a companion discussion topic for the original entry at https://docs.katalon.com/katalon-studio/docs/cucumber-kw-run-feature-file-tag.html
Hello
Actually this is working when we want to include @tags when running feature. but how if we want to exclude @tags ?
is it possible to add an option to run all @tags except specific tags for exampleâŚ
Thanks
Hi @chaker,
Please define your custom Cucumber runner with the tags you want to exclude. For how to apply it in Katalon Studio, you can refer to https://docs.katalon.com/katalon-studio/docs/cucumber-kw-run-cucumber-runner.html. For the syntax on tag exclusion, please refer to https://cucumber.io/docs/cucumber/api/
thank you very much. I really appreciate
Please how do we implement tags in RunFeatureFileWithTags when we have many tags to add ? is it this way for example :
âŚRunFeatureFileWithTagsâ(âInclude/features/test.featureâ, â@Connexion, ~@Identifier, ~@Adressâ)
and how we can launch cucumber tests with tags in command line ? i mean is it possible to add tags directly in the command line without having each time adding/changing tags in the runner class.
Hi @chaker,
For the case when there is no tag exclusion, you can provide your tags as an array of string, e.g. [tag1, tag2] as String[]. In case you want to achieve tag exclusion, you have to write a custom keyword to do that. You can refer to the following answer for how to do it. Run Cucumber tests using dynamic tags
thanks @huynguyen but i need to add tag exclusion, this custom keyword does not work for me with exclusion tags. I am thinking about a way to make the tags as global variables in cucumber runner class (cucumber.options) but I could not find a work aroundâŚ
Please any help ?
Thanks for your time
Hi @chaker,
I mean you have to write a custom keyword to achieve that. The code in the link provided is just a reference for your own implementation.
I found a work around. thanks to you @huynguyen and this post @GrumpyMeow
Here what I did, I create a customkeyword similar to what you suggested but based on âTag expressionsâ
This is how âtag expressionsâ work : (source : Cucumber Reference - Cucumber Documentation)
@tag1 : Scenarios tagged with @tag1
@tag1 and not @tag2 : Scenarios tagged with @tag1 that arenât also tagged with @tag12
@tag1 and @tag2 : Scenarios tagged with both @tag1 and @tag2
@tag1 or @tag2 : Scenarios tagged with either @tag1 or @tag2
we can use parenthesis :
(@tag1 or @tag2) and (not @tag3)
here my test case :
CustomKeywords.âcucumber.CustomRun.runFeatureFileWithTagsâ(âInclude/features/test.featureâ, â@tag1 and not @tag2 and not @tag3â)
Quick question by the way, I am tring to implement an extent cucumber report like this :
as the actual katalon cucumber report does not have statistics and graphic stuff. Please can you help on how can we acheive this.
Thanks again
Here my customkeyword :
package cucumber
import static com.kms.katalon.core.checkpoint.CheckpointFactory.findCheckpoint
import static com.kms.katalon.core.testcase.TestCaseFactory.findTestCase
import static com.kms.katalon.core.testdata.TestDataFactory.findTestData
import static com.kms.katalon.core.testobject.ObjectRepository.findTestObjectimport com.kms.katalon.core.annotation.Keyword
import com.kms.katalon.core.checkpoint.Checkpoint
import com.kms.katalon.core.mobile.keyword.MobileBuiltInKeywords as Mobile
import com.kms.katalon.core.model.FailureHandling
import com.kms.katalon.core.testcase.TestCase
import com.kms.katalon.core.testdata.TestData
import com.kms.katalon.core.testobject.TestObject
import com.kms.katalon.core.webservice.keyword.WSBuiltInKeywords
import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords
import internal.GlobalVariableimport java.text.MessageFormat
import org.apache.commons.lang3.StringUtils
import org.junit.runner.Computer
import org.junit.runner.JUnitCore
import org.junit.runner.Result
import org.junit.runner.notification.Failureimport com.kms.katalon.core.configuration.RunConfiguration
import com.kms.katalon.core.cucumber.keyword.CucumberReporter
import com.kms.katalon.core.cucumber.keyword.CucumberRunnerResult
import com.kms.katalon.core.cucumber.keyword.internal.CucumberRunnerResultImpl
import com.kms.katalon.core.keyword.BuiltinKeywords
import com.kms.katalon.core.keyword.internal.KeywordMain
import com.kms.katalon.core.logging.KeywordLogger
import com.kms.katalon.core.model.RunningMode
import cucumber.api.cli.Main
import groovy.transform.CompileStatic
import com.kms.katalon.core.util.internal.PathUtil//public class CustomRun {
public class CustomRun extends BuiltinKeywords {private static final KeywordLogger logger = KeywordLogger.getInstance(CustomRun.class)
/**
Runs the given Feature file with
featureId
by invoking{@link cucumber.api.cli.Main#run(String, ClassLoader)}.
The generated reports will be extracted in the current report folder with the following path:
<report_folder>/cucumber_report/<current_time_stamp>
@param relativeFilePath
relativeFilePath of Feature file
@param flowControl
an instance {@link FailureHandling} that controls the running flow
@return
an instance of {@link CucumberRunnerResult} that includes status of keyword and report folder location.
@since 5.7
@see CucumberRunnerResult
*/
@Keyword
public static CucumberRunnerResult runFeatureFileWithTags(String relativeFilePath, FailureHandling flowControl, String tagExpression) {
return KeywordMain.runKeyword({
if (StringUtils.isEmpty(relativeFilePath)) {
throw new IllegalArgumentException(âfeatureRelativeFilePath param must not be null or emptyâ)
}
String reportDir = RunConfiguration.getReportFolder() + â/Cucumber_Report/â + System.currentTimeMillis()
String projectDir = RunConfiguration.getProjectDir()
RunningMode runningMode = RunConfiguration.getRunningMode()logger.logInfo( MessageFormat.format("Starting run keyword runFeatureFileWithTags: '{0}' and extract report to folder: '{1}'...", reportDir)) String[] argv = [ "-g", "", projectDir + "/" + relativeFilePath, "--strict", "--plugin", "pretty", "--plugin", "html:Reports/Cucumber_Report/cucumber.html", "--plugin", "json:Reports/Cucumber_Report/cucumber.json", "--plugin", "junit:Reports/Cucumber_Report/cucumber.xml", "--plugin", CucumberReporter.class.getName(), "--tags", tagExpression //"@Rechercher and not @Adresse and not @Identifier" ] if (runningMode == RunningMode.CONSOLE) { argv = argv + ["--monochrome"] } boolean runSuccess = Main.run(argv, CustomRun.class.getClassLoader()) == 0 CucumberRunnerResultImpl cucumberResult = new CucumberRunnerResultImpl( runSuccess ? 'passed' : 'failed', reportDir) if (runSuccess) { logger.logPassed(MessageFormat.format("Feature file: ''{0}'' was passed", relativeFilePath)) } else { KeywordMain.stepFailed(MessageFormat.format("Feature file ''{0}'' was failed", relativeFilePath), flowControl) } return cucumberResult
}, flowControl, âKeyword runFeatureFileWithTags was failedâ)
}
/**
- Runs the given Feature file with
featureId
by invoking- {@link cucumber.api.cli.Main#run(String, ClassLoader)}
- @param relativeFilePath
- relativeFilePath of Feature file
- @return
- an instance of {@link CucumberRunnerResult} that includes status of keyword and report folder location.
- @since 5.7
*/
@Keyword
public static boolean runFeatureFileWithTags(String relativeFilePath, String tagExpression) {
return runFeatureFileWithTags(relativeFilePath, RunConfiguration.getDefaultFailureHandling(), tagExpression)
}/**
Runs the given Feature folder and its nested sub-folder with
folderRelativePath
by invoking {@link cucumber.api.cli.Main#run(String, ClassLoader)}.
The generated reports will be extracted in the current report folder with the following path:
<report_folder>/cucumber_report/<current_time_stamp>
@param folderRelativePath
folder relative path that starts from the current project location
@param flowControl
an instance {@link FailureHandling} that controls the running flow
@return
an instance of {@link CucumberRunnerResult} that includes status of keyword and report folder location.
@since 5.7
*/
@Keyword
public static boolean runFeatureFolderWithTags(String folderRelativePath, FailureHandling flowControl, String tagExpression) {
return KeywordMain.runKeyword({
if (StringUtils.isEmpty(folderRelativePath)) {
throw new IllegalArgumentException(âfolderRelativePath param must not be nullâ)
}
String reportDir = RunConfiguration.getReportFolder() + â/cucumber_report/â + System.currentTimeMillis()
String projectDir = RunConfiguration.getProjectDir()
RunningMode runningMode = RunConfiguration.getRunningMode()logger.logInfo( MessageFormat.format("Starting run keyword>runFeatureFolderWithTags: ''{0}'' and extract report to folder: ''{1}''...", reportDir)) String[] argv = [ "-g", "", projectDir + "/" + folderRelativePath, "--strict", "--plugin", "pretty", "--plugin", "html:Reports/Cucumber_Report/cucumber.html", "--plugin", "json:Reports/Cucumber_Report/cucumber.json", "--plugin", "junit:Reports/Cucumber_Report/cucumber.xml", "--tags", tagExpression ] if (runningMode == RunningMode.CONSOLE) { argv = argv + ["--monochrome"] } boolean runSuccess = Main.run(argv, CustomRun.class.getClassLoader()) == 0 CucumberRunnerResultImpl cucumberResult = new CucumberRunnerResultImpl( runSuccess ? 'passed' : 'failed', reportDir) if (runSuccess) { logger.logPassed(MessageFormat.format("All feature files in ''{0}'' were passed", folderRelativePath)) } else { KeywordMain.stepFailed(MessageFormat.format("Run feature folder ''{0}'' failed", folderRelativePath)) } return cucumberResult
}, flowControl, âKeyword runFeatureFolderWithTags was failedâ)
}
/**
- Runs the given Feature folder and its nested sub-folder with
folderRelativePath
- by invoking {@link cucumber.api.cli.Main#run(String, ClassLoader)}
- @param folderRelativePath
- folder relative path that starts from current project location
- @return
- an instance of {@link CucumberRunnerResult} that includes status of keyword and report folder location.
- @since 5.7
*/
@Keyword
public static boolean runFeatureFolderWithTags(String folderRelativePath, String tagExpression) {
return runFeatureFolderWithTags(folderRelativePath, RunConfiguration.getDefaultFailureHandling(), tagExpression)
}/**
- Runs the given
cucumberRunnerClass
that is annotated with {@link Cucumber} runner by invoke JUnit- runner.
- @param cucumberRunnerClass
- a class that is annotated with {@link Cucumber} runner.
- Example of
cucumberRunnerClass
:
- Example #1: Run all Feature files in Include/features Folder
- import org.junit.runner.RunWith;
- import cucumber.api.CucumberOptions;
- import cucumber.api.junit.Cucumber;
- @RunWith(Cucumber.class)
- @CucumberOptions(features = âInclude/featuresâ, glue = ââ)
- public class MyCucumberRunner {}
- Example #2: Run all Feature files in a specified file/folder
- import org.junit.runner.RunWith;
- import cucumber.api.CucumberOptions;
- import cucumber.api.junit.Cucumber;
- @RunWith(Cucumber.class)
- @CucumberOptions(features = âYour_Folder_Or_File_Pathâ, glue = ââ)
- public class MyCucumberRunner {}
- Example #3: Run all Feature files in a specified file/folder, generate JUnit Cucumber report with XML pretty
- format,
- and copy to a specified folder
- import org.junit.runner.RunWith;
- import cucumber.api.CucumberOptions;
- import cucumber.api.junit.Cucumber;
- @RunWith(Cucumber.class)
- @CucumberOptions(features=âYour_Folder_Pathâ, glue=ââ, plugin = [âprettyâ,
"junit:Folder_Name/cucumber.xml"])
- public class MyCucumberRunner {
- }
- Example #4: Run all Feature files in a specified file/folder, generate multi Cucumber reports with XML, JSON,
- HTML pretty format,
- and copy to a specified folder
- import org.junit.runner.RunWith;
- import cucumber.api.CucumberOptions;
- import cucumber.api.junit.Cucumber;
- @RunWith(Cucumber.class)
- @CucumberOptions(features=âYour_Folder_Pathâ, glue=ââ, plugin = [âprettyâ,
"junit:Folder_Name/cucumber.xml",
"html:Folder_Name",
"json:Folder_Name/cucumber.json"])
- public class MyCucumberRunner {
- }
- @param flowControl
- an instance {@link FailureHandling} that controls the running flow
- @return
- an instance of {@link CucumberRunnerResult} that includes status of keyword and JUnit Runner result.
- @since 5.7
*/
@Keyword
public static CucumberRunnerResult runWithCucumberRunner(Class cucumberRunnerClass, FailureHandling flowControl) {
return KeywordMain.runKeyword({
JUnitCore core = new JUnitCore()
Computer computer = new Computer()
Result result = core.run(computer, cucumberRunnerClass)
boolean runSuccess = result.wasSuccessful()
CucumberRunnerResultImpl cucumberResult = new CucumberRunnerResultImpl(
runSuccess ? âpassedâ : âfailedâ, ââ, result)
if (runSuccess) {
logger.logPassed(MessageFormat.format(âRun with ââ{0}ââ was passedâ, cucumberRunnerClass.getName()))
} else {
List failuresDescriptions =
for (Failure failure : result.getFailures()) {
failuresDescriptions.add(failure.getMessage())
}
KeywordMain.stepFailed(
MessageFormat.format(âThese following reason:\n {0}â, failuresDescriptions))
}
return cucumberResult
}, flowControl, âKeyword runWithCucumberRunner was failedâ)
}/**
- Runs the given
cucumberRunnerClass
that is annotated with {@link Cucumber} runner by invoke JUnit- runner.
- @param cucumberRunnerClass
- a class that is annotated with {@link Cucumber} runner.
- Example of
cucumberRunnerClass
:
- Example #1: Run all Feature files in Include/features Folder
- import org.junit.runner.RunWith;
- import cucumber.api.CucumberOptions;
- import cucumber.api.junit.Cucumber;
- @RunWith(Cucumber.class)
- @CucumberOptions(features = âInclude/featuresâ, glue = ââ)
- public class MyCucumberRunner {}
- Example #2: Run all Feature files in a specified file/folder
- import org.junit.runner.RunWith;
- import cucumber.api.CucumberOptions;
- import cucumber.api.junit.Cucumber;
- @RunWith(Cucumber.class)
- @CucumberOptions(features = âYour_Folder_Or_File_Pathâ, glue = ââ)
- public class MyCucumberRunner {}
- Example #3: Run all Feature files in a specified file/folder, generate JUnit Cucumber report with XML pretty
- format,
- and copy to a specified folder
- import org.junit.runner.RunWith;
- import cucumber.api.CucumberOptions;
- import cucumber.api.junit.Cucumber;
- @RunWith(Cucumber.class)
- @CucumberOptions(features=âYour_Folder_Pathâ, glue=ââ, plugin = [âprettyâ,
"junit:Folder_Name/cucumber.xml"])
- public class MyCucumberRunner {
- }
- Example #4: Run all Feature files in a specified file/folder, generate multi Cucumber reports with XML, JSON,
- HTML pretty format,
- and copy to a specified folder
- import org.junit.runner.RunWith;
- import cucumber.api.CucumberOptions;
- import cucumber.api.junit.Cucumber;
- @RunWith(Cucumber.class)
- @CucumberOptions(features=âYour_Folder_Pathâ, glue=ââ, plugin = [âprettyâ,
"junit:Folder_Name/cucumber.xml",
"html:Folder_Name",
"json:Folder_Name/cucumber.json"])
- public class MyCucumberRunner {
- }
- an instance of {@link CucumberRunnerResult} that includes status of keyword and JUnit Runner result.
- @since 5.7
*/
@Keyword
public static CucumberRunnerResult runWithCucumberRunner(Class cucumberRunnerClass) {
return runWithCucumberRunner(cucumberRunnerClass, RunConfiguration.getDefaultFailureHandling())
}}
Hello, I donât know why always failed when I run runFeatureFileWithTags. Can you help me?
Test Cases/Login FAILED.
Reason:
groovy.lang.MissingMethodException: No signature of method: static com.kms.katalon.core.cucumber.keyword.CucumberBuiltinKeywords.runFeatureFileWithTags() is applicable for argument types: (java.lang.String, java.lang.String, java.lang.String) values: [Include/features/Login.feature, @login, @failedlogin]
at Login.run(Login:16)
at com.kms.katalon.core.main.ScriptEngine.run(ScriptEngine.java:194)
at com.kms.katalon.core.main.ScriptEngine.runScriptAsRawText(ScriptEngine.java:119)
at com.kms.katalon.core.main.TestCaseExecutor.runScript(TestCaseExecutor.java:337)
at com.kms.katalon.core.main.TestCaseExecutor.doExecute(TestCaseExecutor.java:328)
at com.kms.katalon.core.main.TestCaseExecutor.processExecutionPhase(TestCaseExecutor.java:307)
at com.kms.katalon.core.main.TestCaseExecutor.accessMainPhase(TestCaseExecutor.java:299)
at com.kms.katalon.core.main.TestCaseExecutor.execute(TestCaseExecutor.java:233)
at com.kms.katalon.core.main.TestCaseMain.runTestCase(TestCaseMain.java:114)
at com.kms.katalon.core.main.TestCaseMain.runTestCase(TestCaseMain.java:105)
at com.kms.katalon.core.main.TestCaseMain$runTestCase$0.call(Unknown Source)
at TempTestCase1593568374091.run(TempTestCase1593568374091.groovy:21)
@m.septianggadirja Please can you provide the details of the test case you run (featurefilewithtag) and which version of katalon you use.
import static com.kms.katalon.core.checkpoint.CheckpointFactory.findCheckpoint
import static com.kms.katalon.core.testcase.TestCaseFactory.findTestCase
import static com.kms.katalon.core.testdata.TestDataFactory.findTestData
import static com.kms.katalon.core.testobject.ObjectRepository.findTestObject
import com.kms.katalon.core.checkpoint.Checkpoint as Checkpoint
import com.kms.katalon.core.cucumber.keyword.CucumberBuiltinKeywords as CucumberKW
import com.kms.katalon.core.mobile.keyword.MobileBuiltInKeywords as Mobile
import com.kms.katalon.core.model.FailureHandling as FailureHandling
import com.kms.katalon.core.testcase.TestCase as TestCase
import com.kms.katalon.core.testdata.TestData as TestData
import com.kms.katalon.core.testobject.TestObject as TestObject
import com.kms.katalon.core.webservice.keyword.WSBuiltInKeywords as WS
import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUI
import internal.GlobalVariable as GlobalVariable
CucumberKW.runFeatureFileWithTags(âInclude/features/Login.featureâ, â@loginâ, â@failedloginâ)
and use Katalon 6.3.3-2ae328334
Hi @m.septianggadirja
If you used the ârunFeatureFileWithTagsâ Keyword I posted above
then you should change your test case as :
CucumberKW.runFeatureFileWithTags(âInclude/features/Login.featureâ, â@login or @failedloginâ)
instead of :
CucumberKW.runFeatureFileWithTags(âInclude/features/Login.featureâ, â@loginâ, â@failedloginâ)
As I explained in my previous post here how tags work (using OR / AND) :
@tag1 and not @tag2 : Scenarios tagged with @tag1 that arenât also tagged with @tag12
@tag1 and @tag2 : Scenarios tagged with both @tag1 and @tag2
@tag1 or @tag2 : Scenarios tagged with either @tag1 or @tag2
we can use even parenthesis : (@tag1 or @tag2) and (not @tag3)
Now if you have not added a specific keyword for ârunFeatureFileWithTagsâ then it will not work as this feature ârunFeatureFileWithTagsâ is only available in Katalon 7 versions (not sure which version exactly)
Glad to hear it. I am happy for you
Sorry for the late response
For your case, please declare the syntax like this:
CucumberKW.runFeatureFileWithTags('Include/features/test.feature', "@Connexion", "not @Identifier","not @Adress")
or
CucumberKW.runFeatureFileWithTags('Include/features/test.feature', "@Connexion and (not @Identifier) and (not @Adress)")
References: https://cucumber.io/docs/cucumber/api#list-configuration-options
@chaker - Can you please provide more details on RunConfiguration.getDefaultFailureHandling() method? Actually I am developing a mini framework in Katalon as per the client need and I came across this method - getDefaultFailureHandling in RunConfiguration. So if I define this at global level, will there be no need of separately adding âFailureHandling.CONTINUE_ON_FAILUREâ, âFailureHandling.STOP_ON_FAILUREâ at Test case / suite level? Currently, I am adding these two methods at the end of each test steps for each operation (like mobile.tap. mobile,waitforElementPresent etc.)
For future readers benefit, if you want to use the OOTB keyword to execute multiple tags you cannot use either the syntax described in this document or the default way provided in Katalon Studio. Examples of what does and does not work can be found below.
Note: This is all tested in Katalon 8.1.0 on Windows 10.
Working
Note the tags are comma delimited in one string.
CucumberKW.runFeatureFileWithTags('Include/features/path/to/file/testScripts.feature', '@tag1, @tag2')
Result: The scenarios related to all tags are executed as expected.
Not working - syntax documented on this page
Note the tags are comma delimited as different strings.
CucumberKW.runFeatureFileWithTags('Include/features/path/to/file/testScripts.feature', '@tag1', '@tag2')
Result: The test step passes, but no scenarios are executed.
Not working - the default way the keyword is created using Katalon Studioâs manual steps screen
Note the tags are provided in an array of strings.
CucumberKW.runFeatureFileWithTags('Include/features/path/to/file/testScripts.feature', ((['@test', '@closeTheBrowser']) as String))
Result: The test step fails with the following error message:
Caused by: cucumber.runtime.TagExpressionOld$BadTagException: Bad tag: â[@tag1â