Why Lose some suite report when execute suite collection in console mode?


#1

Hello,
I execute a suite collection in console mode,which include 2 test suites,the cmd is as follow:
katalon -runMode=console -projectPath=“F:\demo\demo.prj” -reportFolder=“D:\project\Reports” -retry=0 -testSuiteCollectionPath=“Test Suites\C1” -browserType=“Chrome”

Some times, there is only one report folder generated in D:\project\Reports,
while there should be 2 report folders, one report for every suite.
And the log is like this :
--------------------------------------------------------------------------------
Test Suites/C1 - 20180528_160109…2/2(100%)
--------------------------------------------------------------------------------
Test Suites/TS2_data - Chrome - 20180528_160110…1/1(100%)
Test Suites/TS1 - Chrome - 20180528_160110…1/1(100%)
--------------------------------------------------------------------------------
And the execution mode is sequential.

Why one report is lost? Is it because one test suite report cover another suite report in the same collection? How can I solve the problem?


#2

Try adding another argument to your command script (-reportName=“Reports”) and see if that helps…

katalon -runMode=console -projectPath=“F:\demo\demo.prj” -reportFolder=“D:\project\Reports” --reportName=“Reports” retry=0 -testSuiteCollectionPath=“Test Suites\C1” -browserType=“Chrome”


#3

I have also reported a similar problem in the forum (no response) with the docker image, which also runs in console mode. All suites write to the same report folder, so what you end up with is the results of the last test suite run. I believe it is a bug (although not confirmed), either way, it makes collections unusable.


#4

The BUG here is that the timestamp in the report names are the same for different TestSuites in the TestSuiteCollection.

The original reports are stil generated correctly for Katalon, because they are placed SEPARATELY under subfolders (using TestSuite name) under the Katalon Project path.

But when running under console mode these reports are copied ALL to the SAME supplied -reportFolder WITHOUT using the original subfolders name of the TestSuite, resulting in overwritting reports due to same timestamp name.

Example:

C:\Katalon_Studio>katalon -noSplash  -runMode=console -projectPath="C:\Katalon_Studio_Projects\ESB API\ESB API.prj" -reportFolder="C:\Jenkins\workspace\Katalon Studio - ESB API\Reports" -reportFileName="report" -retry=0 -testSuiteCollectionPath="Test Suites/TSC-Regression" Request sent successfully.Starting Groovy-Eclipse compiler resolver.  Specified compiler level: unspecified133 2.4.7.xx-201611170128-e46 = ACTIVE--------------------------------------------------------------------------------Test Suites/TSC-Regression - 20180601_105317.............................0/7(0%)--------------------------------------------------------------------------------Test Suites/TS-Regression/TS-Retrieve_PUK_Code - Chrome (headless) - 20180601_105317.....................................................................0/2(0%)Test Suites/TS-Regression/TS-Retrieve_Balance_Info - Chrome (headless) - 20180601_105317.................................................................0/2(0%)Test Suites/TS-Regression/TS-Query_MSISDN - Chrome (headless) - 20180601_105318..........................................................................0/3(0%)--------------------------------------------------------------------------------Request sent successfully.--------------------------------------------------------------------------------Test Suites/TSC-Regression - 20180601_105317.............................0/7(0%)--------------------------------------------------------------------------------Test Suites/TS-Regression/TS-Retrieve_PUK_Code - Chrome (headless) - 20180601_105317.....................................................................0/2(0%)Test Suites/TS-Regression/TS-Retrieve_Balance_Info - Chrome (headless) - 20180601_105317.................................................................0/2(0%)Test Suites/TS-Regression/TS-Query_MSISDN - Chrome (headless) - 20180601_105318..........................................................................0/3(0%)--------------------------------------------------------------------------------Copying report to folder C:\Jenkins\workspace\Katalon Studio - ESB API\Reports\20180601_105317...Uploading report to JIRA...Start sending test result to JIRA issuesReport has been sent to JIRA--------------------------------------------------------------------------------Test Suites/TSC-Regression - 20180601_105317............................2/7(29%)--------------------------------------------------------------------------------Test Suites/TS-Regression/TS-Retrieve_Balance_Info - Chrome (headless) - 20180601_105317...............................................................2/2(100%)Test Suites/TS-Regression/TS-Retrieve_PUK_Code - Chrome (headless) - 20180601_105317.....................................................................0/2(0%)Test Suites/TS-Regression/TS-Query_MSISDN - Chrome (headless) - 20180601_105318..........................................................................0/3(0%)--------------------------------------------------------------------------------Copying report to folder C:\Jenkins\workspace\Katalon Studio - ESB API\Reports\20180601_105317...Uploading report to JIRA...Start sending test result to JIRA issuesReport has been sent to JIRA--------------------------------------------------------------------------------Test Suites/TSC-Regression - 20180601_105317............................4/7(57%)--------------------------------------------------------------------------------Test Suites/TS-Regression/TS-Retrieve_Balance_Info - Chrome (headless) - 20180601_105317...............................................................2/2(100%)Test Suites/TS-Regression/TS-Retrieve_PUK_Code - Chrome (headless) - 20180601_105317...................................................................2/2(100%)Test Suites/TS-Regression/TS-Query_MSISDN - Chrome (headless) - 20180601_105318..........................................................................0/3(0%)--------------------------------------------------------------------------------Copying report to folder C:\Jenkins\workspace\Katalon Studio - ESB API\Reports\20180601_105318...Uploading report to JIRA...Start sending test result to JIRA issuesReport has been sent to JIRA--------------------------------------------------------------------------------Test Suites/TSC-Regression - 20180601_105317...........................7/7(100%)--------------------------------------------------------------------------------Test Suites/TS-Regression/TS-Retrieve_Balance_Info - Chrome (headless) - 20180601_105317...............................................................2/2(100%)Test Suites/TS-Regression/TS-Retrieve_PUK_Code - Chrome (headless) - 20180601_105317...................................................................2/2(100%)Test Suites/TS-Regression/TS-Query_MSISDN - Chrome (headless) - 20180601_105318........................................................................3/3(100%)--------------------------------------------------------------------------------C:\Katalon_Studio>exit 1 Build step 'Execute Windows batch command' marked build as failureRecording test resultsFinished: FAILURE>>>>>>The first 2 TestSuites under this TestSuiteCollection BOTH get the same timestamp 20180601_105317 in their name !
I would suggest that during the copying of the reports while running in console mode the subfolder for each TestSuite is also included.

#5

This issue has been fixed and it will be delivered in the next release.


#6

Looking forward to it. Thanx


#7

Hello, I am facing the same problem but in the docker image (https://github.com/katalon-studio/docker-images). Will you be updating that also as part of the release?

Best, Nanda


#8

Vinh Nguyen said:

This issue has been fixed and it will be delivered in the next release.

Dear Vingh,
please let us know if it was already fixed or still waiting for release.
If I am not mistaken, v.5.5 was published mid of July (after your comment)
Although I am still facing the issue


#9

Hi,
I am also facing the same issue in docker image.will you please let us know when ll this be fixed?


#10

Hi,
Any updates for this issue? I’m having 5.6.0 version and I’m still facing the issue.


#11

@Vinh Nguyen I have tried it on version 5.7.1 and still the issue exists. Please let us know when this will be fixed. This is one of the major issue as reports are getting overwritten after the execution.


#12

You can temporary work with a workaround I described in:


#13

Gerard van der Winkel said:

You can temporary work with a workaround I described in:
https://forum.katalon.com/discussion/7327/jenkins-cant-find-junit-reports

I can not try this as Im using katalon docker image for running the test suite collection. This image by default uses -reportfolder option while triggering test cases.

Please let me know if we have any other workaround ? As this issue is getting very serious. With out this we are completely blocked on our pipeline.


#14

even with Katalon docker image, you can “workaround” the issue in a same manner.
The Reports were correctly placed separately in separate subfolders under the Katalon Projects Report folder.
The problem happens when these reports are copied incorrectly into one Reports folder (resulting in overwrites due to duplicate subfolder names).
In Katalon docker image setup you are simply running a script too (as I understand it from the video I have seen on Katalon docker image), where you can add too a personal copy command to it after running Katalon to copy the subfolders correctly into your destination while keeping the subfolder structure intact, instead of trying to store all reports in one folder (as Katalon does).


#15

Gerard,
IMHO, What you are saying is not quite correct. Since it pulls down the docker image every time, there is no way to modify the script which it is running. I am trying to get around the problem by having a test case called “logWorkaround” which simply copies the entire reports folder to a new folder which is timestamped. I add this at the end of every test suite, so all the reports for that suite get copied. This might result in additional logs but it is a small price to pay for a few extra bytes as long as all the logs are available.
Best,
Nanda


#16

Hi Nanda,
I do not have enough info on how you are using Katalon Docker Image in your Automation process . But maybe using docker cp may be an option for you ?
http://obrown.io/2016/02/20/docker-build.html


#17

Hi Gerard,
I am using Katalon Docker as follows:
docker run -v “/home/circleci/project/Katalon”:/katalon/katalon/source:ro -v “/home/circleci/project/report”:/katalon/katalon/report -e KATALON_OPTS=’-browserType=“Chrome (headless)” -retry=0 -statusDelay=150 -reportName=“Reports” -testSuiteCollectionPath=“Test Suites/Executable Suites/All/SuperSet/CircleCI/AllTests” ’ katalonstudio/katalon:latest

docker cp might work although it is sometime since I have used it. Will investigate later when time permits.

Best,
Nanda


#18

Hi Gerard,

I am also running in to this issue with the same cirumstances as Nanda. While there might be workarounds such as docker cp, the underlying bug does not (from here) look like a very difficult issue to fix.

If the bug turns out to be very difficult to resolve then any docker-script related workaround can be fixed by the Katalon Studio team as well, since they are the one that publishes the docker image.

best regards,
Christian.


#19

Gerard van der Winkel said:

even with Katalon docker image, you can “workaround” the issue in a same manner.
The Reports were correctly placed separately in separate subfolders under the Katalon Projects Report folder…

As Gerard suggested, the separate subfolders under the Katalon Projects Report is fine.

Here is my workaround:


#20

Steve Seypt said:

As Gerard suggested, the separate subfolders under the Katalon Projects Report is fine.

Here is my workaround:

https://github.com/katalon-studio/docker-images/issues/26#issuecomment-430275065

Regarding your other comment on Github:

This might need a bit more refactoring. I’m now facing another problem. eval will exit 1 if the collection has any failed test. Because of that cp -r is not executed… still no reports :slight_smile:

One solution may be:
In your shell script start with

set +e

and after executing Katalon use the following to remember Katalon’s Exit code:

katalonRC=$?

and after your other commands you can exit with Katalon’s Exit code with:

exit $katalonRC

BTW on Linux Im using “rsync -auv” instead of cp command.