[Cucumber] Run Feature File with Tags


This is a companion discussion topic for the original entry at https://docs.katalon.com/katalon-studio/docs/cucumber-kw-run-feature-file-tag.html

Hello
Actually this is working when we want to include @tags when running feature. but how if we want to exclude @tags ?
is it possible to add an option to run all @tags except specific tags for example…

Thanks

Hi @chaker,
Please define your custom Cucumber runner with the tags you want to exclude. For how to apply it in Katalon Studio, you can refer to https://docs.katalon.com/katalon-studio/docs/cucumber-kw-run-cucumber-runner.html. For the syntax on tag exclusion, please refer to https://cucumber.io/docs/cucumber/api/

thank you very much. I really appreciate

Please how do we implement tags in RunFeatureFileWithTags when we have many tags to add ? is it this way for example :
…RunFeatureFileWithTags’(‘Include/features/test.feature’, “@Connexion, ~@Identifier, ~@Adress”)

and how we can launch cucumber tests with tags in command line ? i mean is it possible to add tags directly in the command line without having each time adding/changing tags in the runner class.

Hi @chaker,
For the case when there is no tag exclusion, you can provide your tags as an array of string, e.g. [tag1, tag2] as String[]. In case you want to achieve tag exclusion, you have to write a custom keyword to do that. You can refer to the following answer for how to do it. Run Cucumber tests using dynamic tags

1 Like

thanks @huynguyen but i need to add tag exclusion, this custom keyword does not work for me with exclusion tags. I am thinking about a way to make the tags as global variables in cucumber runner class (cucumber.options) but I could not find a work around…
Please any help ?
Thanks for your time

Hi @chaker,
I mean you have to write a custom keyword to achieve that. The code in the link provided is just a reference for your own implementation.

1 Like

I found a work around. thanks to you @huynguyen and this post @GrumpyMeow

Here what I did, I create a customkeyword similar to what you suggested but based on “Tag expressions”

This is how “tag expressions” work : (source : https://cucumber.io/docs/cucumber/api/)

@tag1 : Scenarios tagged with @tag1
@tag1 and not @tag2 : Scenarios tagged with @tag1 that aren’t also tagged with @tag12
@tag1 and @tag2 : Scenarios tagged with both @tag1 and @tag2
@tag1 or @tag2 : Scenarios tagged with either @tag1 or @tag2
we can use parenthesis :
(@tag1 or @tag2) and (not @tag3)

here my test case :

CustomKeywords.‘cucumber.CustomRun.runFeatureFileWithTags’(‘Include/features/test.feature’, “@tag1 and not @tag2 and not @tag3”)

Quick question by the way, I am tring to implement an extent cucumber report like this :



as the actual katalon cucumber report does not have statistics and graphic stuff. Please can you help on how can we acheive this.
Thanks again

Here my customkeyword :

package cucumber

import static com.kms.katalon.core.checkpoint.CheckpointFactory.findCheckpoint
import static com.kms.katalon.core.testcase.TestCaseFactory.findTestCase
import static com.kms.katalon.core.testdata.TestDataFactory.findTestData
import static com.kms.katalon.core.testobject.ObjectRepository.findTestObject

import com.kms.katalon.core.annotation.Keyword
import com.kms.katalon.core.checkpoint.Checkpoint
import com.kms.katalon.core.mobile.keyword.MobileBuiltInKeywords as Mobile
import com.kms.katalon.core.model.FailureHandling
import com.kms.katalon.core.testcase.TestCase
import com.kms.katalon.core.testdata.TestData
import com.kms.katalon.core.testobject.TestObject
import com.kms.katalon.core.webservice.keyword.WSBuiltInKeywords
import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords
import internal.GlobalVariable

import java.text.MessageFormat

import org.apache.commons.lang3.StringUtils
import org.junit.runner.Computer
import org.junit.runner.JUnitCore
import org.junit.runner.Result
import org.junit.runner.notification.Failure

import com.kms.katalon.core.configuration.RunConfiguration
import com.kms.katalon.core.cucumber.keyword.CucumberReporter
import com.kms.katalon.core.cucumber.keyword.CucumberRunnerResult
import com.kms.katalon.core.cucumber.keyword.internal.CucumberRunnerResultImpl
import com.kms.katalon.core.keyword.BuiltinKeywords
import com.kms.katalon.core.keyword.internal.KeywordMain
import com.kms.katalon.core.logging.KeywordLogger
import com.kms.katalon.core.model.RunningMode
import cucumber.api.cli.Main
import groovy.transform.CompileStatic
import com.kms.katalon.core.util.internal.PathUtil

//public class CustomRun {
public class CustomRun extends BuiltinKeywords {

private static final KeywordLogger logger = KeywordLogger.getInstance(CustomRun.class)

/**

  • Runs the given Feature file with featureId by invoking

  • {@link cucumber.api.cli.Main#run(String[], ClassLoader)}.

  • The generated reports will be extracted in the current report folder with the following path: <report_folder>/cucumber_report/<current_time_stamp>

  • @param relativeFilePath

  • relativeFilePath of Feature file

  • @param flowControl

  • an instance {@link FailureHandling} that controls the running flow

  • @return

  • an instance of {@link CucumberRunnerResult} that includes status of keyword and report folder location.

  • @since 5.7

  • @see CucumberRunnerResult
    */
    @Keyword
    public static CucumberRunnerResult runFeatureFileWithTags(String relativeFilePath, FailureHandling flowControl, String tagExpression) {
    return KeywordMain.runKeyword({
    if (StringUtils.isEmpty(relativeFilePath)) {
    throw new IllegalArgumentException(“featureRelativeFilePath param must not be null or empty”)
    }
    String reportDir = RunConfiguration.getReportFolder() + “/Cucumber_Report/” + System.currentTimeMillis()
    String projectDir = RunConfiguration.getProjectDir()
    RunningMode runningMode = RunConfiguration.getRunningMode()

     logger.logInfo(
     		MessageFormat.format("Starting run keyword runFeatureFileWithTags: '{0}' and extract report to folder: '{1}'...", reportDir))
     String[] argv = [
     	"-g",
     	"",
     	projectDir + "/" + relativeFilePath,
     	"--strict",
     	"--plugin",
     	"pretty",
     	"--plugin",
     	"html:Reports/Cucumber_Report/cucumber.html",
     	"--plugin",
     	"json:Reports/Cucumber_Report/cucumber.json",
     	"--plugin",
     	"junit:Reports/Cucumber_Report/cucumber.xml",
     	"--plugin",
     	CucumberReporter.class.getName(),
     	"--tags",
     	tagExpression
     	//"@Rechercher and not @Adresse and not @Identifier"
     ]
     if (runningMode == RunningMode.CONSOLE) {
     	argv = argv + ["--monochrome"]
     }
     boolean runSuccess = Main.run(argv, CustomRun.class.getClassLoader()) == 0
     CucumberRunnerResultImpl cucumberResult = new CucumberRunnerResultImpl(
     		runSuccess ? 'passed' : 'failed', reportDir)
     if (runSuccess) {
     	logger.logPassed(MessageFormat.format("Feature file: ''{0}'' was passed", relativeFilePath))
     } else {
     	KeywordMain.stepFailed(MessageFormat.format("Feature file ''{0}'' was failed", relativeFilePath), flowControl)
     }
     return cucumberResult
    

    }, flowControl, “Keyword runFeatureFileWithTags was failed”)
    }

/**

  • Runs the given Feature file with featureId by invoking
  • {@link cucumber.api.cli.Main#run(String[], ClassLoader)}
  • @param relativeFilePath
  • relativeFilePath of Feature file
  • @return
  • an instance of {@link CucumberRunnerResult} that includes status of keyword and report folder location.
  • @since 5.7
    */
    @Keyword
    public static boolean runFeatureFileWithTags(String relativeFilePath, String tagExpression) {
    return runFeatureFileWithTags(relativeFilePath, RunConfiguration.getDefaultFailureHandling(), tagExpression)
    }

/**

  • Runs the given Feature folder and its nested sub-folder with folderRelativePath

  • by invoking {@link cucumber.api.cli.Main#run(String[], ClassLoader)}.

  • The generated reports will be extracted in the current report folder with the following path: <report_folder>/cucumber_report/<current_time_stamp>

  • @param folderRelativePath

  • folder relative path that starts from the current project location

  • @param flowControl

  • an instance {@link FailureHandling} that controls the running flow

  • @return

  • an instance of {@link CucumberRunnerResult} that includes status of keyword and report folder location.

  • @since 5.7
    */
    @Keyword
    public static boolean runFeatureFolderWithTags(String folderRelativePath, FailureHandling flowControl, String tagExpression) {
    return KeywordMain.runKeyword({
    if (StringUtils.isEmpty(folderRelativePath)) {
    throw new IllegalArgumentException(“folderRelativePath param must not be null”)
    }
    String reportDir = RunConfiguration.getReportFolder() + “/cucumber_report/” + System.currentTimeMillis()
    String projectDir = RunConfiguration.getProjectDir()
    RunningMode runningMode = RunConfiguration.getRunningMode()

     logger.logInfo(
     		MessageFormat.format("Starting run keyword>runFeatureFolderWithTags: ''{0}'' and extract report to folder: ''{1}''...",
     		reportDir))
     String[] argv = [
     	"-g",
     	"",
     	projectDir + "/" + folderRelativePath,
     	"--strict",
     	"--plugin",
     	"pretty",
     	"--plugin",
     	"html:Reports/Cucumber_Report/cucumber.html",
     	"--plugin",
     	"json:Reports/Cucumber_Report/cucumber.json",
     	"--plugin",
     	"junit:Reports/Cucumber_Report/cucumber.xml",
     	"--tags",
     	tagExpression
     ]
     if (runningMode == RunningMode.CONSOLE) {
     	argv = argv + ["--monochrome"]
     }
    
     boolean runSuccess = Main.run(argv, CustomRun.class.getClassLoader()) == 0
     CucumberRunnerResultImpl cucumberResult = new CucumberRunnerResultImpl(
     		runSuccess ? 'passed' : 'failed', reportDir)
     if (runSuccess) {
     	logger.logPassed(MessageFormat.format("All feature files in ''{0}'' were passed", folderRelativePath))
     } else {
     	KeywordMain.stepFailed(MessageFormat.format("Run feature folder ''{0}'' failed", folderRelativePath))
     }
     return cucumberResult
    

    }, flowControl, “Keyword runFeatureFolderWithTags was failed”)
    }

/**

  • Runs the given Feature folder and its nested sub-folder with folderRelativePath
  • by invoking {@link cucumber.api.cli.Main#run(String[], ClassLoader)}
  • @param folderRelativePath
  • folder relative path that starts from current project location
  • @return
  • an instance of {@link CucumberRunnerResult} that includes status of keyword and report folder location.
  • @since 5.7
    */
    @Keyword
    public static boolean runFeatureFolderWithTags(String folderRelativePath, String tagExpression) {
    return runFeatureFolderWithTags(folderRelativePath, RunConfiguration.getDefaultFailureHandling(), tagExpression)
    }

/**

  • Runs the given cucumberRunnerClass that is annotated with {@link Cucumber} runner by invoke JUnit
  • runner.
  • @param cucumberRunnerClass
  • a class that is annotated with {@link Cucumber} runner.
  • Example of cucumberRunnerClass:
  • Example #1: Run all Feature files in Include/features Folder
  • 
    
  • import org.junit.runner.RunWith;
  • import cucumber.api.CucumberOptions;
  • import cucumber.api.junit.Cucumber;
  • @RunWith(Cucumber.class)
  • @CucumberOptions(features = “Include/features”, glue = “”)
  • public class MyCucumberRunner {}
  • Example #2: Run all Feature files in a specified file/folder
  • 
    
  • import org.junit.runner.RunWith;
  • import cucumber.api.CucumberOptions;
  • import cucumber.api.junit.Cucumber;
  • @RunWith(Cucumber.class)
  • @CucumberOptions(features = “Your_Folder_Or_File_Path”, glue = “”)
  • public class MyCucumberRunner {}
  • Example #3: Run all Feature files in a specified file/folder, generate JUnit Cucumber report with XML pretty
  • format,
  • and copy to a specified folder
  • 
    
  • import org.junit.runner.RunWith;
  • import cucumber.api.CucumberOptions;
  • import cucumber.api.junit.Cucumber;
  • @RunWith(Cucumber.class)
  • @CucumberOptions(features=“Your_Folder_Path”, glue="", plugin = [“pretty”,
  •                  "junit:Folder_Name/cucumber.xml"])
    
  • public class MyCucumberRunner {
  • }
  • Example #4: Run all Feature files in a specified file/folder, generate multi Cucumber reports with XML, JSON,
  • HTML pretty format,
  • and copy to a specified folder
  • 
    
  • import org.junit.runner.RunWith;
  • import cucumber.api.CucumberOptions;
  • import cucumber.api.junit.Cucumber;
  • @RunWith(Cucumber.class)
  • @CucumberOptions(features=“Your_Folder_Path”, glue="", plugin = [“pretty”,
  •                  "junit:Folder_Name/cucumber.xml",
    
  •                  "html:Folder_Name",
    
  •                  "json:Folder_Name/cucumber.json"])
    
  • public class MyCucumberRunner {
  • }
  • @param flowControl
  • an instance {@link FailureHandling} that controls the running flow
  • @return
  • an instance of {@link CucumberRunnerResult} that includes status of keyword and JUnit Runner result.
  • @since 5.7
    */
    @Keyword
    public static CucumberRunnerResult runWithCucumberRunner(Class cucumberRunnerClass, FailureHandling flowControl) {
    return KeywordMain.runKeyword({
    JUnitCore core = new JUnitCore()
    Computer computer = new Computer()
    Result result = core.run(computer, cucumberRunnerClass)
    boolean runSuccess = result.wasSuccessful()
    CucumberRunnerResultImpl cucumberResult = new CucumberRunnerResultImpl(
    runSuccess ? ‘passed’ : ‘failed’, ‘’, result)
    if (runSuccess) {
    logger.logPassed(MessageFormat.format(“Run with ‘’{0}’’ was passed”, cucumberRunnerClass.getName()))
    } else {
    List failuresDescriptions = []
    for (Failure failure : result.getFailures()) {
    failuresDescriptions.add(failure.getMessage())
    }
    KeywordMain.stepFailed(
    MessageFormat.format(“These following reason:\n {0}”, failuresDescriptions))
    }
    return cucumberResult
    }, flowControl, “Keyword runWithCucumberRunner was failed”)
    }

/**

  • Runs the given cucumberRunnerClass that is annotated with {@link Cucumber} runner by invoke JUnit
  • runner.
  • @param cucumberRunnerClass
  • a class that is annotated with {@link Cucumber} runner.
  • Example of cucumberRunnerClass:
  • Example #1: Run all Feature files in Include/features Folder
  • 
    
  • import org.junit.runner.RunWith;
  • import cucumber.api.CucumberOptions;
  • import cucumber.api.junit.Cucumber;
  • @RunWith(Cucumber.class)
  • @CucumberOptions(features = “Include/features”, glue = “”)
  • public class MyCucumberRunner {}
  • Example #2: Run all Feature files in a specified file/folder
  • 
    
  • import org.junit.runner.RunWith;
  • import cucumber.api.CucumberOptions;
  • import cucumber.api.junit.Cucumber;
  • @RunWith(Cucumber.class)
  • @CucumberOptions(features = “Your_Folder_Or_File_Path”, glue = “”)
  • public class MyCucumberRunner {}
  • Example #3: Run all Feature files in a specified file/folder, generate JUnit Cucumber report with XML pretty
  • format,
  • and copy to a specified folder
  • 
    
  • import org.junit.runner.RunWith;
  • import cucumber.api.CucumberOptions;
  • import cucumber.api.junit.Cucumber;
  • @RunWith(Cucumber.class)
  • @CucumberOptions(features=“Your_Folder_Path”, glue="", plugin = [“pretty”,
  •                  "junit:Folder_Name/cucumber.xml"])
    
  • public class MyCucumberRunner {
  • }
  • Example #4: Run all Feature files in a specified file/folder, generate multi Cucumber reports with XML, JSON,
  • HTML pretty format,
  • and copy to a specified folder
  • 
    
  • import org.junit.runner.RunWith;
  • import cucumber.api.CucumberOptions;
  • import cucumber.api.junit.Cucumber;
  • @RunWith(Cucumber.class)
  • @CucumberOptions(features=“Your_Folder_Path”, glue="", plugin = [“pretty”,
  •                  "junit:Folder_Name/cucumber.xml",
    
  •                  "html:Folder_Name",
    
  •                  "json:Folder_Name/cucumber.json"])
    
  • public class MyCucumberRunner {
  • }
  • an instance of {@link CucumberRunnerResult} that includes status of keyword and JUnit Runner result.
  • @since 5.7
    */
    @Keyword
    public static CucumberRunnerResult runWithCucumberRunner(Class cucumberRunnerClass) {
    return runWithCucumberRunner(cucumberRunnerClass, RunConfiguration.getDefaultFailureHandling())
    }

}

Hello, I don’t know why always failed when I run runFeatureFileWithTags. Can you help me?

Test Cases/Login FAILED.
Reason:
groovy.lang.MissingMethodException: No signature of method: static com.kms.katalon.core.cucumber.keyword.CucumberBuiltinKeywords.runFeatureFileWithTags() is applicable for argument types: (java.lang.String, java.lang.String, java.lang.String) values: [Include/features/Login.feature, @login, @failedlogin]
at Login.run(Login:16)
at com.kms.katalon.core.main.ScriptEngine.run(ScriptEngine.java:194)
at com.kms.katalon.core.main.ScriptEngine.runScriptAsRawText(ScriptEngine.java:119)
at com.kms.katalon.core.main.TestCaseExecutor.runScript(TestCaseExecutor.java:337)
at com.kms.katalon.core.main.TestCaseExecutor.doExecute(TestCaseExecutor.java:328)
at com.kms.katalon.core.main.TestCaseExecutor.processExecutionPhase(TestCaseExecutor.java:307)
at com.kms.katalon.core.main.TestCaseExecutor.accessMainPhase(TestCaseExecutor.java:299)
at com.kms.katalon.core.main.TestCaseExecutor.execute(TestCaseExecutor.java:233)
at com.kms.katalon.core.main.TestCaseMain.runTestCase(TestCaseMain.java:114)
at com.kms.katalon.core.main.TestCaseMain.runTestCase(TestCaseMain.java:105)
at com.kms.katalon.core.main.TestCaseMain$runTestCase$0.call(Unknown Source)
at TempTestCase1593568374091.run(TempTestCase1593568374091.groovy:21)

@m.septianggadirja Please can you provide the details of the test case you run (featurefilewithtag) and which version of katalon you use.

import static com.kms.katalon.core.checkpoint.CheckpointFactory.findCheckpoint
import static com.kms.katalon.core.testcase.TestCaseFactory.findTestCase
import static com.kms.katalon.core.testdata.TestDataFactory.findTestData
import static com.kms.katalon.core.testobject.ObjectRepository.findTestObject
import com.kms.katalon.core.checkpoint.Checkpoint as Checkpoint
import com.kms.katalon.core.cucumber.keyword.CucumberBuiltinKeywords as CucumberKW
import com.kms.katalon.core.mobile.keyword.MobileBuiltInKeywords as Mobile
import com.kms.katalon.core.model.FailureHandling as FailureHandling
import com.kms.katalon.core.testcase.TestCase as TestCase
import com.kms.katalon.core.testdata.TestData as TestData
import com.kms.katalon.core.testobject.TestObject as TestObject
import com.kms.katalon.core.webservice.keyword.WSBuiltInKeywords as WS
import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUI
import internal.GlobalVariable as GlobalVariable

CucumberKW.runFeatureFileWithTags(‘Include/features/Login.feature’, “@login”, “@failedlogin”)

and use Katalon 6.3.3-2ae328334

@chaker because katalon version?

Hi @m.septianggadirja
If you used the “runFeatureFileWithTags” Keyword I posted above
then you should change your test case as :
CucumberKW.runFeatureFileWithTags(‘Include/features/Login.feature’, “@login or @failedlogin”)

instead of :
CucumberKW.runFeatureFileWithTags(‘Include/features/Login.feature’, “@login”, “@failedlogin”)

As I explained in my previous post here how tags work (using OR / AND) :
@tag1 and not @tag2 : Scenarios tagged with @tag1 that aren’t also tagged with @tag12
@tag1 and @tag2 : Scenarios tagged with both @tag1 and @tag2
@tag1 or @tag2 : Scenarios tagged with either @tag1 or @tag2
we can use even parenthesis : (@tag1 or @tag2) and (not @tag3)

Now if you have not added a specific keyword for “runFeatureFileWithTags” then it will not work as this feature “runFeatureFileWithTags” is only available in Katalon 7 versions (not sure which version exactly)

Thanks @chaker , resolved after update to version 7.5

Glad to hear it. I am happy for you :slight_smile:

Sorry for the late response

For your case, please declare the syntax like this:

CucumberKW.runFeatureFileWithTags('Include/features/test.feature', "@Connexion", "not @Identifier","not @Adress")

or

CucumberKW.runFeatureFileWithTags('Include/features/test.feature', "@Connexion and (not @Identifier) and (not @Adress)")

References: https://cucumber.io/docs/cucumber/api#list-configuration-options

1 Like

@chaker - Can you please provide more details on RunConfiguration.getDefaultFailureHandling() method? Actually I am developing a mini framework in Katalon as per the client need and I came across this method - getDefaultFailureHandling in RunConfiguration. So if I define this at global level, will there be no need of separately adding “FailureHandling.CONTINUE_ON_FAILURE”, “FailureHandling.STOP_ON_FAILURE” at Test case / suite level? Currently, I am adding these two methods at the end of each test steps for each operation (like mobile.tap. mobile,waitforElementPresent etc.)