Adding test cases through command line

Web Testing
1

Hi,

Is there a way that I can create a test suite and add test cases to it through command?
What i want is that whatever test cases are failed in my CI-CD pipeline, I want to re-execute them (only failed tests). I want to create a test suite, add those failed TCs in it and re-execute the stage. All in CI-CD so Im looking for such command.

Regards
Soumya

2

I am sure there is no such build-in automatic functionality in Katalon Studio.

In a definition of a Test Suite, each Test Case is marked with attribute “Run” with value On/Off. You can manually select which test case to run.

スクリーンショット 2020-01-14 22.33.53
スクリーンショット 2020-01-14 22.33.531072×713 92.4 KB

In a Katalon project there are files <projectDir>/Test Suites/folder name/<test case name>.ts which has such code snipet:

    <testCaseLink>¶                                                                                                              
       <guid>cbec7f36-1c4f-4ede-ae10-35dbfc07d946</guid>¶                                                                        
       <isReuseDriver>false</isReuseDriver>¶                                                                                     
       <isRun>true</isRun>¶                                                                                                      
       <testCaseId>Test Cases/CURA/restorePreviousScreenshots</testCaseId>¶                                                      
    </testCaseLink>¶

I believe that the <isRun> here denotes On/Off value you check for the Run attribute in the GUI.

If you want to, you can write and run a script that modifies the <isRun> value in the *.ts file,

How do you determin which test case to turn On/Off for <isRun>?

In the <projectDir>/Reports/yyyyMMdd_hhmmss/folders/<test suite name>/yyyyMMdd_hhmmss/JUnit_Report.xml. In the file you can find such code fragments:

 <testsuites name="chronos_capture" time="51" tests="2" failures="1" errors="0">¶                                                
    <testsuite name="chronos_capture" tests="2" failures="1" errors="0" time="51" timestamp="2019-10-10 22:43:08" id="Test Suites/CURA/chronos_capture">¶                                                                   
    ....   
        <testcase name="Test Cases/CURA/restorePreviousScreenshots" status="FAILED">¶

If you develop a script to read the JUnit_Report.xml file, then you can find which test cases of which test case failed so that you want to run again later.

3

Good answer, Kaz.

This requirement has been mentioned before - build a test suite programmatically (with and easy way to take fails form one suite and pass them to a retest-suite).

4

‘Love’ this solution.
With the amendment … this will only confuse the main scope of automated tests, relevancy.
Unless fuzz testing is desired.

5

I agree with @anon46315158.

I would not mind repeating all test cases, regardless passed or failed in the previous run, every time I run a test suite. Especially in a CI/CD process, I would not mind the burden of redoing already-passed tests because my machine takes it and I can be away from the redo.