[Cucumber] Run Feature File with Tags


This is a companion discussion topic for the original entry at https://docs.katalon.com/katalon-studio/docs/cucumber-kw-run-feature-file-tag.html

Hello
Actually this is working when we want to include @tags when running feature. but how if we want to exclude @tags ?
is it possible to add an option to run all @tags except specific tags for example…

Thanks

Hi @chaker,
Please define your custom Cucumber runner with the tags you want to exclude. For how to apply it in Katalon Studio, you can refer to https://docs.katalon.com/katalon-studio/docs/cucumber-kw-run-cucumber-runner.html. For the syntax on tag exclusion, please refer to https://cucumber.io/docs/cucumber/api/

thank you very much. I really appreciate

Please how do we implement tags in RunFeatureFileWithTags when we have many tags to add ? is it this way for example :
…RunFeatureFileWithTags’(‘Include/features/test.feature’, “@Connexion, ~@Identifier, ~@Adress”)

and how we can launch cucumber tests with tags in command line ? i mean is it possible to add tags directly in the command line without having each time adding/changing tags in the runner class.

Hi @chaker,
For the case when there is no tag exclusion, you can provide your tags as an array of string, e.g. [tag1, tag2] as String[]. In case you want to achieve tag exclusion, you have to write a custom keyword to do that. You can refer to the following answer for how to do it. Run Cucumber tests using dynamic tags

1 Like

thanks @huynguyen but i need to add tag exclusion, this custom keyword does not work for me with exclusion tags. I am thinking about a way to make the tags as global variables in cucumber runner class (cucumber.options) but I could not find a work around…
Please any help ?
Thanks for your time

Hi @chaker,
I mean you have to write a custom keyword to achieve that. The code in the link provided is just a reference for your own implementation.

1 Like

I found a work around. thanks to you @huynguyen and this post @GrumpyMeow

Here what I did, I create a customkeyword similar to what you suggested but based on “Tag expressions”

This is how “tag expressions” work : (source : https://cucumber.io/docs/cucumber/api/)

@tag1 : Scenarios tagged with @tag1
@tag1 and not @tag2 : Scenarios tagged with @tag1 that aren’t also tagged with @tag12
@tag1 and @tag2 : Scenarios tagged with both @tag1 and @tag2
@tag1 or @tag2 : Scenarios tagged with either @tag1 or @tag2
we can use parenthesis :
(@tag1 or @tag2) and (not @tag3)

here my test case :

CustomKeywords.‘cucumber.CustomRun.runFeatureFileWithTags’(‘Include/features/test.feature’, “@tag1 and not @tag2 and not @tag3”)

Quick question by the way, I am tring to implement an extent cucumber report like this :



as the actual katalon cucumber report does not have statistics and graphic stuff. Please can you help on how can we acheive this.
Thanks again

Here my customkeyword :

package cucumber

import static com.kms.katalon.core.checkpoint.CheckpointFactory.findCheckpoint
import static com.kms.katalon.core.testcase.TestCaseFactory.findTestCase
import static com.kms.katalon.core.testdata.TestDataFactory.findTestData
import static com.kms.katalon.core.testobject.ObjectRepository.findTestObject

import com.kms.katalon.core.annotation.Keyword
import com.kms.katalon.core.checkpoint.Checkpoint
import com.kms.katalon.core.mobile.keyword.MobileBuiltInKeywords as Mobile
import com.kms.katalon.core.model.FailureHandling
import com.kms.katalon.core.testcase.TestCase
import com.kms.katalon.core.testdata.TestData
import com.kms.katalon.core.testobject.TestObject
import com.kms.katalon.core.webservice.keyword.WSBuiltInKeywords
import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords
import internal.GlobalVariable

import java.text.MessageFormat

import org.apache.commons.lang3.StringUtils
import org.junit.runner.Computer
import org.junit.runner.JUnitCore
import org.junit.runner.Result
import org.junit.runner.notification.Failure

import com.kms.katalon.core.configuration.RunConfiguration
import com.kms.katalon.core.cucumber.keyword.CucumberReporter
import com.kms.katalon.core.cucumber.keyword.CucumberRunnerResult
import com.kms.katalon.core.cucumber.keyword.internal.CucumberRunnerResultImpl
import com.kms.katalon.core.keyword.BuiltinKeywords
import com.kms.katalon.core.keyword.internal.KeywordMain
import com.kms.katalon.core.logging.KeywordLogger
import com.kms.katalon.core.model.RunningMode
import cucumber.api.cli.Main
import groovy.transform.CompileStatic
import com.kms.katalon.core.util.internal.PathUtil

//public class CustomRun {
public class CustomRun extends BuiltinKeywords {

private static final KeywordLogger logger = KeywordLogger.getInstance(CustomRun.class)

/**

  • Runs the given Feature file with featureId by invoking

  • {@link cucumber.api.cli.Main#run(String[], ClassLoader)}.

  • The generated reports will be extracted in the current report folder with the following path: <report_folder>/cucumber_report/<current_time_stamp>

  • @param relativeFilePath

  • relativeFilePath of Feature file

  • @param flowControl

  • an instance {@link FailureHandling} that controls the running flow

  • @return

  • an instance of {@link CucumberRunnerResult} that includes status of keyword and report folder location.

  • @since 5.7

  • @see CucumberRunnerResult
    */
    @Keyword
    public static CucumberRunnerResult runFeatureFileWithTags(String relativeFilePath, FailureHandling flowControl, String tagExpression) {
    return KeywordMain.runKeyword({
    if (StringUtils.isEmpty(relativeFilePath)) {
    throw new IllegalArgumentException(“featureRelativeFilePath param must not be null or empty”)
    }
    String reportDir = RunConfiguration.getReportFolder() + “/Cucumber_Report/” + System.currentTimeMillis()
    String projectDir = RunConfiguration.getProjectDir()
    RunningMode runningMode = RunConfiguration.getRunningMode()

     logger.logInfo(
     		MessageFormat.format("Starting run keyword runFeatureFileWithTags: '{0}' and extract report to folder: '{1}'...", reportDir))
     String[] argv = [
     	"-g",
     	"",
     	projectDir + "/" + relativeFilePath,
     	"--strict",
     	"--plugin",
     	"pretty",
     	"--plugin",
     	"html:Reports/Cucumber_Report/cucumber.html",
     	"--plugin",
     	"json:Reports/Cucumber_Report/cucumber.json",
     	"--plugin",
     	"junit:Reports/Cucumber_Report/cucumber.xml",
     	"--plugin",
     	CucumberReporter.class.getName(),
     	"--tags",
     	tagExpression
     	//"@Rechercher and not @Adresse and not @Identifier"
     ]
     if (runningMode == RunningMode.CONSOLE) {
     	argv = argv + ["--monochrome"]
     }
     boolean runSuccess = Main.run(argv, CustomRun.class.getClassLoader()) == 0
     CucumberRunnerResultImpl cucumberResult = new CucumberRunnerResultImpl(
     		runSuccess ? 'passed' : 'failed', reportDir)
     if (runSuccess) {
     	logger.logPassed(MessageFormat.format("Feature file: ''{0}'' was passed", relativeFilePath))
     } else {
     	KeywordMain.stepFailed(MessageFormat.format("Feature file ''{0}'' was failed", relativeFilePath), flowControl)
     }
     return cucumberResult
    

    }, flowControl, “Keyword runFeatureFileWithTags was failed”)
    }

/**

  • Runs the given Feature file with featureId by invoking
  • {@link cucumber.api.cli.Main#run(String[], ClassLoader)}
  • @param relativeFilePath
  • relativeFilePath of Feature file
  • @return
  • an instance of {@link CucumberRunnerResult} that includes status of keyword and report folder location.
  • @since 5.7
    */
    @Keyword
    public static boolean runFeatureFileWithTags(String relativeFilePath, String tagExpression) {
    return runFeatureFileWithTags(relativeFilePath, RunConfiguration.getDefaultFailureHandling(), tagExpression)
    }

/**

  • Runs the given Feature folder and its nested sub-folder with folderRelativePath

  • by invoking {@link cucumber.api.cli.Main#run(String[], ClassLoader)}.

  • The generated reports will be extracted in the current report folder with the following path: <report_folder>/cucumber_report/<current_time_stamp>

  • @param folderRelativePath

  • folder relative path that starts from the current project location

  • @param flowControl

  • an instance {@link FailureHandling} that controls the running flow

  • @return

  • an instance of {@link CucumberRunnerResult} that includes status of keyword and report folder location.

  • @since 5.7
    */
    @Keyword
    public static boolean runFeatureFolderWithTags(String folderRelativePath, FailureHandling flowControl, String tagExpression) {
    return KeywordMain.runKeyword({
    if (StringUtils.isEmpty(folderRelativePath)) {
    throw new IllegalArgumentException(“folderRelativePath param must not be null”)
    }
    String reportDir = RunConfiguration.getReportFolder() + “/cucumber_report/” + System.currentTimeMillis()
    String projectDir = RunConfiguration.getProjectDir()
    RunningMode runningMode = RunConfiguration.getRunningMode()

     logger.logInfo(
     		MessageFormat.format("Starting run keyword>runFeatureFolderWithTags: ''{0}'' and extract report to folder: ''{1}''...",
     		reportDir))
     String[] argv = [
     	"-g",
     	"",
     	projectDir + "/" + folderRelativePath,
     	"--strict",
     	"--plugin",
     	"pretty",
     	"--plugin",
     	"html:Reports/Cucumber_Report/cucumber.html",
     	"--plugin",
     	"json:Reports/Cucumber_Report/cucumber.json",
     	"--plugin",
     	"junit:Reports/Cucumber_Report/cucumber.xml",
     	"--tags",
     	tagExpression
     ]
     if (runningMode == RunningMode.CONSOLE) {
     	argv = argv + ["--monochrome"]
     }
    
     boolean runSuccess = Main.run(argv, CustomRun.class.getClassLoader()) == 0
     CucumberRunnerResultImpl cucumberResult = new CucumberRunnerResultImpl(
     		runSuccess ? 'passed' : 'failed', reportDir)
     if (runSuccess) {
     	logger.logPassed(MessageFormat.format("All feature files in ''{0}'' were passed", folderRelativePath))
     } else {
     	KeywordMain.stepFailed(MessageFormat.format("Run feature folder ''{0}'' failed", folderRelativePath))
     }
     return cucumberResult
    

    }, flowControl, “Keyword runFeatureFolderWithTags was failed”)
    }

/**

  • Runs the given Feature folder and its nested sub-folder with folderRelativePath
  • by invoking {@link cucumber.api.cli.Main#run(String[], ClassLoader)}
  • @param folderRelativePath
  • folder relative path that starts from current project location
  • @return
  • an instance of {@link CucumberRunnerResult} that includes status of keyword and report folder location.
  • @since 5.7
    */
    @Keyword
    public static boolean runFeatureFolderWithTags(String folderRelativePath, String tagExpression) {
    return runFeatureFolderWithTags(folderRelativePath, RunConfiguration.getDefaultFailureHandling(), tagExpression)
    }

/**

  • Runs the given cucumberRunnerClass that is annotated with {@link Cucumber} runner by invoke JUnit
  • runner.
  • @param cucumberRunnerClass
  • a class that is annotated with {@link Cucumber} runner.
  • Example of cucumberRunnerClass:
  • Example #1: Run all Feature files in Include/features Folder
  • 
    
  • import org.junit.runner.RunWith;
  • import cucumber.api.CucumberOptions;
  • import cucumber.api.junit.Cucumber;
  • @RunWith(Cucumber.class)
  • @CucumberOptions(features = “Include/features”, glue = “”)
  • public class MyCucumberRunner {}
  • Example #2: Run all Feature files in a specified file/folder
  • 
    
  • import org.junit.runner.RunWith;
  • import cucumber.api.CucumberOptions;
  • import cucumber.api.junit.Cucumber;
  • @RunWith(Cucumber.class)
  • @CucumberOptions(features = “Your_Folder_Or_File_Path”, glue = “”)
  • public class MyCucumberRunner {}
  • Example #3: Run all Feature files in a specified file/folder, generate JUnit Cucumber report with XML pretty
  • format,
  • and copy to a specified folder
  • 
    
  • import org.junit.runner.RunWith;
  • import cucumber.api.CucumberOptions;
  • import cucumber.api.junit.Cucumber;
  • @RunWith(Cucumber.class)
  • @CucumberOptions(features=“Your_Folder_Path”, glue="", plugin = [“pretty”,
  •                  "junit:Folder_Name/cucumber.xml"])
    
  • public class MyCucumberRunner {
  • }
  • Example #4: Run all Feature files in a specified file/folder, generate multi Cucumber reports with XML, JSON,
  • HTML pretty format,
  • and copy to a specified folder
  • 
    
  • import org.junit.runner.RunWith;
  • import cucumber.api.CucumberOptions;
  • import cucumber.api.junit.Cucumber;
  • @RunWith(Cucumber.class)
  • @CucumberOptions(features=“Your_Folder_Path”, glue="", plugin = [“pretty”,
  •                  "junit:Folder_Name/cucumber.xml",
    
  •                  "html:Folder_Name",
    
  •                  "json:Folder_Name/cucumber.json"])
    
  • public class MyCucumberRunner {
  • }
  • @param flowControl
  • an instance {@link FailureHandling} that controls the running flow
  • @return
  • an instance of {@link CucumberRunnerResult} that includes status of keyword and JUnit Runner result.
  • @since 5.7
    */
    @Keyword
    public static CucumberRunnerResult runWithCucumberRunner(Class cucumberRunnerClass, FailureHandling flowControl) {
    return KeywordMain.runKeyword({
    JUnitCore core = new JUnitCore()
    Computer computer = new Computer()
    Result result = core.run(computer, cucumberRunnerClass)
    boolean runSuccess = result.wasSuccessful()
    CucumberRunnerResultImpl cucumberResult = new CucumberRunnerResultImpl(
    runSuccess ? ‘passed’ : ‘failed’, ‘’, result)
    if (runSuccess) {
    logger.logPassed(MessageFormat.format(“Run with ‘’{0}’’ was passed”, cucumberRunnerClass.getName()))
    } else {
    List failuresDescriptions = []
    for (Failure failure : result.getFailures()) {
    failuresDescriptions.add(failure.getMessage())
    }
    KeywordMain.stepFailed(
    MessageFormat.format(“These following reason:\n {0}”, failuresDescriptions))
    }
    return cucumberResult
    }, flowControl, “Keyword runWithCucumberRunner was failed”)
    }

/**

  • Runs the given cucumberRunnerClass that is annotated with {@link Cucumber} runner by invoke JUnit
  • runner.
  • @param cucumberRunnerClass
  • a class that is annotated with {@link Cucumber} runner.
  • Example of cucumberRunnerClass:
  • Example #1: Run all Feature files in Include/features Folder
  • 
    
  • import org.junit.runner.RunWith;
  • import cucumber.api.CucumberOptions;
  • import cucumber.api.junit.Cucumber;
  • @RunWith(Cucumber.class)
  • @CucumberOptions(features = “Include/features”, glue = “”)
  • public class MyCucumberRunner {}
  • Example #2: Run all Feature files in a specified file/folder
  • 
    
  • import org.junit.runner.RunWith;
  • import cucumber.api.CucumberOptions;
  • import cucumber.api.junit.Cucumber;
  • @RunWith(Cucumber.class)
  • @CucumberOptions(features = “Your_Folder_Or_File_Path”, glue = “”)
  • public class MyCucumberRunner {}
  • Example #3: Run all Feature files in a specified file/folder, generate JUnit Cucumber report with XML pretty
  • format,
  • and copy to a specified folder
  • 
    
  • import org.junit.runner.RunWith;
  • import cucumber.api.CucumberOptions;
  • import cucumber.api.junit.Cucumber;
  • @RunWith(Cucumber.class)
  • @CucumberOptions(features=“Your_Folder_Path”, glue="", plugin = [“pretty”,
  •                  "junit:Folder_Name/cucumber.xml"])
    
  • public class MyCucumberRunner {
  • }
  • Example #4: Run all Feature files in a specified file/folder, generate multi Cucumber reports with XML, JSON,
  • HTML pretty format,
  • and copy to a specified folder
  • 
    
  • import org.junit.runner.RunWith;
  • import cucumber.api.CucumberOptions;
  • import cucumber.api.junit.Cucumber;
  • @RunWith(Cucumber.class)
  • @CucumberOptions(features=“Your_Folder_Path”, glue="", plugin = [“pretty”,
  •                  "junit:Folder_Name/cucumber.xml",
    
  •                  "html:Folder_Name",
    
  •                  "json:Folder_Name/cucumber.json"])
    
  • public class MyCucumberRunner {
  • }
  • an instance of {@link CucumberRunnerResult} that includes status of keyword and JUnit Runner result.
  • @since 5.7
    */
    @Keyword
    public static CucumberRunnerResult runWithCucumberRunner(Class cucumberRunnerClass) {
    return runWithCucumberRunner(cucumberRunnerClass, RunConfiguration.getDefaultFailureHandling())
    }

}

Hello, I don’t know why always failed when I run runFeatureFileWithTags. Can you help me?

Test Cases/Login FAILED.
Reason:
groovy.lang.MissingMethodException: No signature of method: static com.kms.katalon.core.cucumber.keyword.CucumberBuiltinKeywords.runFeatureFileWithTags() is applicable for argument types: (java.lang.String, java.lang.String, java.lang.String) values: [Include/features/Login.feature, @login, @failedlogin]
at Login.run(Login:16)
at com.kms.katalon.core.main.ScriptEngine.run(ScriptEngine.java:194)
at com.kms.katalon.core.main.ScriptEngine.runScriptAsRawText(ScriptEngine.java:119)
at com.kms.katalon.core.main.TestCaseExecutor.runScript(TestCaseExecutor.java:337)
at com.kms.katalon.core.main.TestCaseExecutor.doExecute(TestCaseExecutor.java:328)
at com.kms.katalon.core.main.TestCaseExecutor.processExecutionPhase(TestCaseExecutor.java:307)
at com.kms.katalon.core.main.TestCaseExecutor.accessMainPhase(TestCaseExecutor.java:299)
at com.kms.katalon.core.main.TestCaseExecutor.execute(TestCaseExecutor.java:233)
at com.kms.katalon.core.main.TestCaseMain.runTestCase(TestCaseMain.java:114)
at com.kms.katalon.core.main.TestCaseMain.runTestCase(TestCaseMain.java:105)
at com.kms.katalon.core.main.TestCaseMain$runTestCase$0.call(Unknown Source)
at TempTestCase1593568374091.run(TempTestCase1593568374091.groovy:21)

@m.septianggadirja Please can you provide the details of the test case you run (featurefilewithtag) and which version of katalon you use.

import static com.kms.katalon.core.checkpoint.CheckpointFactory.findCheckpoint
import static com.kms.katalon.core.testcase.TestCaseFactory.findTestCase
import static com.kms.katalon.core.testdata.TestDataFactory.findTestData
import static com.kms.katalon.core.testobject.ObjectRepository.findTestObject
import com.kms.katalon.core.checkpoint.Checkpoint as Checkpoint
import com.kms.katalon.core.cucumber.keyword.CucumberBuiltinKeywords as CucumberKW
import com.kms.katalon.core.mobile.keyword.MobileBuiltInKeywords as Mobile
import com.kms.katalon.core.model.FailureHandling as FailureHandling
import com.kms.katalon.core.testcase.TestCase as TestCase
import com.kms.katalon.core.testdata.TestData as TestData
import com.kms.katalon.core.testobject.TestObject as TestObject
import com.kms.katalon.core.webservice.keyword.WSBuiltInKeywords as WS
import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUI
import internal.GlobalVariable as GlobalVariable

CucumberKW.runFeatureFileWithTags(‘Include/features/Login.feature’, “@login”, “@failedlogin”)

and use Katalon 6.3.3-2ae328334

@chaker because katalon version?

Hi @m.septianggadirja
If you used the “runFeatureFileWithTags” Keyword I posted above
then you should change your test case as :
CucumberKW.runFeatureFileWithTags(‘Include/features/Login.feature’, “@login or @failedlogin”)

instead of :
CucumberKW.runFeatureFileWithTags(‘Include/features/Login.feature’, “@login”, “@failedlogin”)

As I explained in my previous post here how tags work (using OR / AND) :
@tag1 and not @tag2 : Scenarios tagged with @tag1 that aren’t also tagged with @tag12
@tag1 and @tag2 : Scenarios tagged with both @tag1 and @tag2
@tag1 or @tag2 : Scenarios tagged with either @tag1 or @tag2
we can use even parenthesis : (@tag1 or @tag2) and (not @tag3)

Now if you have not added a specific keyword for “runFeatureFileWithTags” then it will not work as this feature “runFeatureFileWithTags” is only available in Katalon 7 versions (not sure which version exactly)

Thanks @chaker , resolved after update to version 7.5

Glad to hear it. I am happy for you :slight_smile:

Sorry for the late response

For your case, please declare the syntax like this:

CucumberKW.runFeatureFileWithTags('Include/features/test.feature', "@Connexion", "not @Identifier","not @Adress")

or

CucumberKW.runFeatureFileWithTags('Include/features/test.feature', "@Connexion and (not @Identifier) and (not @Adress)")

References: https://cucumber.io/docs/cucumber/api#list-configuration-options

1 Like

@chaker - Can you please provide more details on RunConfiguration.getDefaultFailureHandling() method? Actually I am developing a mini framework in Katalon as per the client need and I came across this method - getDefaultFailureHandling in RunConfiguration. So if I define this at global level, will there be no need of separately adding “FailureHandling.CONTINUE_ON_FAILURE”, “FailureHandling.STOP_ON_FAILURE” at Test case / suite level? Currently, I am adding these two methods at the end of each test steps for each operation (like mobile.tap. mobile,waitforElementPresent etc.)

For future readers benefit, if you want to use the OOTB keyword to execute multiple tags you cannot use either the syntax described in this document or the default way provided in Katalon Studio. Examples of what does and does not work can be found below.

Note: This is all tested in Katalon 8.1.0 on Windows 10.

Working
Note the tags are comma delimited in one string.

CucumberKW.runFeatureFileWithTags('Include/features/path/to/file/testScripts.feature', '@tag1, @tag2')

Result: The scenarios related to all tags are executed as expected.

Not working - syntax documented on this page
Note the tags are comma delimited as different strings.

CucumberKW.runFeatureFileWithTags('Include/features/path/to/file/testScripts.feature', '@tag1', '@tag2')

Result: The test step passes, but no scenarios are executed.

Not working - the default way the keyword is created using Katalon Studio’s manual steps screen
Note the tags are provided in an array of strings.

CucumberKW.runFeatureFileWithTags('Include/features/path/to/file/testScripts.feature', ((['@test', '@closeTheBrowser']) as String))

Result: The test step fails with the following error message:
Caused by: cucumber.runtime.TagExpressionOld$BadTagException: Bad tag: “[@tag1