huyuan
Katalon Apprentice
05/28/2018

Why Lose some suite report when execute suite collection in console mode?

Hello,
I execute a suite collection in console mode,which include 2 test suites,the cmd is as follow:
katalon -runMode=console -projectPath="F:\demo\demo.prj" -reportFolder="D:\project\Reports"  -retry=0 -testSuiteCollectionPath="Test Suites\C1" -browserType="Chrome" 

Some times, there is only one report folder generated in  D:\project\Reports,
while there should be 2 report folders, one report for every suite.
 And the log is like this :
--------------------------------------------------------------------------------
Test Suites/C1 - 20180528_160109....................2/2(100%)
--------------------------------------------------------------------------------
Test Suites/TS2_data - Chrome - 20180528_160110........................1/1(100%)
Test Suites/TS1 - Chrome - 20180528_160110.............................1/1(100%)
--------------------------------------------------------------------------------
And the execution mode is sequential.

Why  one report is lost? Is it because one test suite  report cover another suite report in the same collection? How can I solve the problem?

Upvote
Quote

Comments

  • David Edmondson
    Katalon Ambassador
    05/31/2018
    Try adding another argument to your command script (-reportName="Reports") and see if that helps...

    katalon -runMode=console -projectPath="F:\demo\demo.prj" -reportFolder="D:\project\Reports"  --reportName="Reports" retry=0 -testSuiteCollectionPath="Test Suites\C1" -browserType="Chrome" 
    Upvote
    Quote
  • Nick
    Katalon Apprentice
    06/01/2018
    I have also reported a similar problem in the forum (no response) with the docker image, which also runs in console mode. All suites write to the same report folder, so what you end up with is the results of the last test suite run. I believe it is a bug (although not confirmed), either way, it makes collections unusable.
    Upvote
    Quote
  • Gerard van der Winkel
    Katalon Ambassador
    06/02/2018
    The BUG here is that the timestamp in the report names are the same for different TestSuites in the TestSuiteCollection.

    The original reports are stil  generated correctly for Katalon, because they are placed SEPARATELY under subfolders (using TestSuite name) under the Katalon Project path.

    But when running under console mode these reports are copied ALL to the SAME supplied -reportFolder WITHOUT using the original subfolders name of the TestSuite, resulting in overwritting reports due to same timestamp name.

    Example: 

    >>>>>>
    C:\Katalon_Studio>katalon -noSplash  -runMode=console -projectPath="C:\Katalon_Studio_Projects\ESB API\ESB API.prj" -reportFolder="C:\Jenkins\workspace\Katalon Studio - ESB API\Reports" -reportFileName="report" -retry=0 -testSuiteCollectionPath="Test Suites/TSC-Regression" 
    Request sent successfully.
    Starting Groovy-Eclipse compiler resolver.  Specified compiler level: unspecified
    133 2.4.7.xx-201611170128-e46 = ACTIVE
    
    --------------------------------------------------------------------------------
    Test Suites/TSC-Regression - 20180601_105317.............................0/7(0%)
    --------------------------------------------------------------------------------
    Test Suites/TS-Regression/TS-Retrieve_PUK_Code - Chrome (headless) - 20180601_10
    5317.....................................................................0/2(0%)
    Test Suites/TS-Regression/TS-Retrieve_Balance_Info - Chrome (headless) - 2018060
    1_105317.................................................................0/2(0%)
    Test Suites/TS-Regression/TS-Query_MSISDN - Chrome (headless) - 20180601_105318.
    .........................................................................0/3(0%)
    --------------------------------------------------------------------------------
    
    Request sent successfully.
    
    --------------------------------------------------------------------------------
    Test Suites/TSC-Regression - 20180601_105317.............................0/7(0%)
    --------------------------------------------------------------------------------
    Test Suites/TS-Regression/TS-Retrieve_PUK_Code - Chrome (headless) - 20180601_10
    5317.....................................................................0/2(0%)
    Test Suites/TS-Regression/TS-Retrieve_Balance_Info - Chrome (headless) - 2018060
    1_105317.................................................................0/2(0%)
    Test Suites/TS-Regression/TS-Query_MSISDN - Chrome (headless) - 20180601_105318.
    .........................................................................0/3(0%)
    --------------------------------------------------------------------------------
    
    Copying report to folder C:\Jenkins\workspace\Katalon Studio - ESB API\Reports\20180601_105317...
    Uploading report to JIRA...
    Start sending test result to JIRA issues
    Report has been sent to JIRA
    
    --------------------------------------------------------------------------------
    Test Suites/TSC-Regression - 20180601_105317............................2/7(29%)
    --------------------------------------------------------------------------------
    Test Suites/TS-Regression/TS-Retrieve_Balance_Info - Chrome (headless) - 2018060
    1_105317...............................................................2/2(100%)
    Test Suites/TS-Regression/TS-Retrieve_PUK_Code - Chrome (headless) - 20180601_10
    5317.....................................................................0/2(0%)
    Test Suites/TS-Regression/TS-Query_MSISDN - Chrome (headless) - 20180601_105318.
    .........................................................................0/3(0%)
    --------------------------------------------------------------------------------
    
    Copying report to folder C:\Jenkins\workspace\Katalon Studio - ESB API\Reports\20180601_105317...
    Uploading report to JIRA...
    Start sending test result to JIRA issues
    Report has been sent to JIRA
    
    --------------------------------------------------------------------------------
    Test Suites/TSC-Regression - 20180601_105317............................4/7(57%)
    --------------------------------------------------------------------------------
    Test Suites/TS-Regression/TS-Retrieve_Balance_Info - Chrome (headless) - 2018060
    1_105317...............................................................2/2(100%)
    Test Suites/TS-Regression/TS-Retrieve_PUK_Code - Chrome (headless) - 20180601_10
    5317...................................................................2/2(100%)
    Test Suites/TS-Regression/TS-Query_MSISDN - Chrome (headless) - 20180601_105318.
    .........................................................................0/3(0%)
    --------------------------------------------------------------------------------
    
    Copying report to folder C:\Jenkins\workspace\Katalon Studio - ESB API\Reports\20180601_105318...
    Uploading report to JIRA...
    Start sending test result to JIRA issues
    Report has been sent to JIRA
    
    --------------------------------------------------------------------------------
    Test Suites/TSC-Regression - 20180601_105317...........................7/7(100%)
    --------------------------------------------------------------------------------
    Test Suites/TS-Regression/TS-Retrieve_Balance_Info - Chrome (headless) - 2018060
    1_105317...............................................................2/2(100%)
    Test Suites/TS-Regression/TS-Retrieve_PUK_Code - Chrome (headless) - 20180601_10
    5317...................................................................2/2(100%)
    Test Suites/TS-Regression/TS-Query_MSISDN - Chrome (headless) - 20180601_105318.
    .......................................................................3/3(100%)
    --------------------------------------------------------------------------------
    
    
    C:\Katalon_Studio>exit 1 
    Build step 'Execute Windows batch command' marked build as failure
    Recording test results
    Finished: FAILURE
    
    >>>>>>
    
    The first 2 TestSuites under this TestSuiteCollection BOTH get the same timestamp 20180601_105317 in their name !
    I would suggest that during the copying of the reports while running in console mode the subfolder for each TestSuite is also included.
    Upvote
    Quote
  • Vinh Nguyen
    Katalon Moderator
    06/25/2018
    This issue has been fixed and it will be delivered in the next release.
    Upvote
    Quote
  • Gerard van der Winkel
    Katalon Ambassador
    06/25/2018
    Looking forward to it. Thanx
    Upvote
    Quote
  • Swaminathan Nandakumar
    Katalon Apprentice
    07/11/2018
    Hello, I am facing the same problem but in the docker image (https://github.com/katalon-studio/docker-images).  Will you be updating that also as part of the release?

    Best, Nanda
    Upvote
    Quote
  • Alex
    Katalon Apprentice
    07/25/2018
    This issue has been fixed and it will be delivered in the next release.
    Dear Vingh,
    please let us know if it was already fixed or still waiting for release.
    If I am not mistaken, v.5.5 was published mid of July (after your comment)
    Although I am still facing the issue
    Upvote
    Quote
  • sudarshan
    Katalon Apprentice
    08/06/2018
    Hi,
    I am also facing the same issue in docker image.will you please let us know when ll this be fixed?
    Upvote
    Quote
  • Andrei Zaharia
    Katalon Apprentice
    08/25/2018
    Hi,
    Any updates for this issue? I'm having 5.6.0 version and I'm still facing the issue.
    Upvote
    Quote
  • Sunil Kumar A
    Katalon Apprentice
    09/17/2018
    @"Vinh Nguyen" I have tried it on version 5.7.1 and still the issue exists. Please let us know when this will be fixed. This is one of the major issue as reports are getting overwritten after the execution.
    Upvote
    Quote
  • Gerard van der Winkel
    Katalon Ambassador
    09/17/2018
    You can temporary work with a workaround I described in:
    https://forum.katalon.com/discussion/7327/jenkins-cant-find-junit-reports
    Upvote
    Quote
  • Sunil Kumar A
    Katalon Apprentice
    09/21/2018
    edited September 21
    You can temporary work with a workaround I described in:
    https://forum.katalon.com/discussion/7327/jenkins-cant-find-junit-reports
    I can not try this as Im using katalon docker image for running the test suite collection. This image by default uses -reportfolder option while triggering test cases.

    Please let me know if we have any other workaround ? As this issue is getting very serious. With out this we are completely blocked on our pipeline.
    Upvote
    Quote
  • Gerard van der Winkel
    Katalon Ambassador
    09/21/2018
    even with Katalon docker image, you can "workaround" the issue in a same manner.
    The Reports were correctly placed separately in separate subfolders under the Katalon Projects Report folder.
    The problem happens when these reports are copied incorrectly into one Reports folder (resulting in overwrites due to duplicate subfolder names).
    In Katalon docker image setup you are simply running a script too (as I understand it from the video I have seen on Katalon docker image), where you can add too a personal copy command to it after running Katalon to copy the subfolders correctly into your destination while keeping the subfolder structure intact, instead of trying to store all reports in one folder (as Katalon does).
    Upvote
    Quote
  • Swaminathan Nandakumar
    Katalon Apprentice
    09/21/2018
    Gerard,
      IMHO, What you are saying is not quite correct.   Since it pulls down the docker image every time, there is no way to modify the script which it is running.  I am trying to get around the problem by having a test case called "logWorkaround" which simply copies the entire reports folder to a new folder which is timestamped.  I add this at the end of every test suite, so all the reports for that suite get copied.  This might result in additional logs but it is a small price to pay for a few extra bytes as long as all the logs are available.
    Best,
      Nanda
    Upvote
    Quote
  • Gerard van der Winkel
    Katalon Ambassador
    09/24/2018
    Hi Nanda, 
    I do not have enough info on how you are using Katalon Docker Image in your Automation process . But maybe using docker cp may be an option for you ?
    http://obrown.io/2016/02/20/docker-build.html
    Upvote
    Quote
  • Swaminathan Nandakumar
    Katalon Apprentice
    09/25/2018
    Hi Gerard,
      I am using Katalon Docker as follows:
    docker run -v "/home/circleci/project/Katalon":/katalon/katalon/source:ro -v "/home/circleci/project/report":/katalon/katalon/report -e KATALON_OPTS='-browserType="Chrome (headless)" -retry=0 -statusDelay=150 -reportName="Reports" -testSuiteCollectionPath="Test Suites/Executable Suites/All/SuperSet/CircleCI/AllTests" ' katalonstudio/katalon:latest

      docker cp might work although it is sometime since I have used it.  Will investigate later when time permits.

    Best,
      Nanda
    Upvote
    Quote
  • Christian R
    Katalon Apprentice
    09/27/2018
    Hi Gerard, 

    I am also running in to this issue with the same cirumstances as Nanda. While there might be workarounds such as docker cp, the underlying bug does not (from here) look like a very difficult issue to fix.

    If the bug turns out to be very difficult to resolve then any docker-script related workaround can be fixed by the Katalon Studio team as well, since they are the one that publishes the docker image. 

    best regards, 
    Christian. 
    Upvote
    Quote
  • Steve Seypt
    Katalon Apprentice
    10/16/2018
    even with Katalon docker image, you can "workaround" the issue in a same manner.
    The Reports were correctly placed separately in separate subfolders under the Katalon Projects Report folder....
    As Gerard suggested, the separate subfolders under the Katalon Projects Report is fine.
    Here is my workaround:




    Upvote
    Quote
  • Gerard van der Winkel
    Katalon Ambassador
    10/17/2018
    As Gerard suggested, the separate subfolders under the Katalon Projects Report is fine.
    Here is my workaround:

    Regarding your other comment on Github:
    This might need a bit more refactoring. I'm now facing another problem. eval will exit 1 if the collection has any failed test. Because of that cp -r is not executed.. still no reports :)

    One solution may be:
    In your shell script start with 

    set +e

    and after executing Katalon use the following to remember Katalon's Exit code:

    katalonRC=$?

    and after your other commands you can exit with Katalon's Exit code with:

    exit $katalonRC

    BTW on Linux Im using "rsync -auv" instead of cp command.
    Upvote
    Quote
Sign In or Register to comment.