Hello,
I execute a suite collection in console mode,which include 2 test suites,the cmd is as follow:
katalon -runMode=console -projectPath=“F:\demo\demo.prj” -reportFolder=“D:\project\Reports” -retry=0 -testSuiteCollectionPath=“Test Suites\C1” -browserType=“Chrome”
Some times, there is only one report folder generated in D:\project\Reports,
while there should be 2 report folders, one report for every suite.
And the log is like this :
--------------------------------------------------------------------------------
Test Suites/C1 - 20180528_160109…2/2(100%)
--------------------------------------------------------------------------------
Test Suites/TS2_data - Chrome - 20180528_160110…1/1(100%)
Test Suites/TS1 - Chrome - 20180528_160110…1/1(100%)
--------------------------------------------------------------------------------
And the execution mode is sequential.
Why one report is lost? Is it because one test suite report cover another suite report in the same collection? How can I solve the problem?
I have also reported a similar problem in the forum (no response) with the docker image, which also runs in console mode. All suites write to the same report folder, so what you end up with is the results of the last test suite run. I believe it is a bug (although not confirmed), either way, it makes collections unusable.
The BUG here is that the timestamp in the report names are the same for different TestSuites in the TestSuiteCollection.
The original reports are stil generated correctly for Katalon, because they are placed SEPARATELY under subfolders (using TestSuite name) under the Katalon Project path.
But when running under console mode these reports are copied ALL to the SAME supplied -reportFolder WITHOUT using the original subfolders name of the TestSuite, resulting in overwritting reports due to same timestamp name.
Example:
C:\Katalon_Studio>katalon -noSplash -runMode=console -projectPath="C:\Katalon_Studio_Projects\ESB API\ESB API.prj" -reportFolder="C:\Jenkins\workspace\Katalon Studio - ESB API\Reports" -reportFileName="report" -retry=0 -testSuiteCollectionPath="Test Suites/TSC-Regression" Request sent successfully.Starting Groovy-Eclipse compiler resolver. Specified compiler level: unspecified133 2.4.7.xx-201611170128-e46 = ACTIVE--------------------------------------------------------------------------------Test Suites/TSC-Regression - 20180601_105317.............................0/7(0%)--------------------------------------------------------------------------------Test Suites/TS-Regression/TS-Retrieve_PUK_Code - Chrome (headless) - 20180601_105317.....................................................................0/2(0%)Test Suites/TS-Regression/TS-Retrieve_Balance_Info - Chrome (headless) - 20180601_105317.................................................................0/2(0%)Test Suites/TS-Regression/TS-Query_MSISDN - Chrome (headless) - 20180601_105318..........................................................................0/3(0%)--------------------------------------------------------------------------------Request sent successfully.--------------------------------------------------------------------------------Test Suites/TSC-Regression - 20180601_105317.............................0/7(0%)--------------------------------------------------------------------------------Test Suites/TS-Regression/TS-Retrieve_PUK_Code - Chrome (headless) - 20180601_105317.....................................................................0/2(0%)Test Suites/TS-Regression/TS-Retrieve_Balance_Info - Chrome (headless) - 20180601_105317.................................................................0/2(0%)Test Suites/TS-Regression/TS-Query_MSISDN - Chrome (headless) - 20180601_105318..........................................................................0/3(0%)--------------------------------------------------------------------------------Copying report to folder C:\Jenkins\workspace\Katalon Studio - ESB API\Reports\20180601_105317...Uploading report to JIRA...Start sending test result to JIRA issuesReport has been sent to JIRA--------------------------------------------------------------------------------Test Suites/TSC-Regression - 20180601_105317............................2/7(29%)--------------------------------------------------------------------------------Test Suites/TS-Regression/TS-Retrieve_Balance_Info - Chrome (headless) - 20180601_105317...............................................................2/2(100%)Test Suites/TS-Regression/TS-Retrieve_PUK_Code - Chrome (headless) - 20180601_105317.....................................................................0/2(0%)Test Suites/TS-Regression/TS-Query_MSISDN - Chrome (headless) - 20180601_105318..........................................................................0/3(0%)--------------------------------------------------------------------------------Copying report to folder C:\Jenkins\workspace\Katalon Studio - ESB API\Reports\20180601_105317...Uploading report to JIRA...Start sending test result to JIRA issuesReport has been sent to JIRA--------------------------------------------------------------------------------Test Suites/TSC-Regression - 20180601_105317............................4/7(57%)--------------------------------------------------------------------------------Test Suites/TS-Regression/TS-Retrieve_Balance_Info - Chrome (headless) - 20180601_105317...............................................................2/2(100%)Test Suites/TS-Regression/TS-Retrieve_PUK_Code - Chrome (headless) - 20180601_105317...................................................................2/2(100%)Test Suites/TS-Regression/TS-Query_MSISDN - Chrome (headless) - 20180601_105318..........................................................................0/3(0%)--------------------------------------------------------------------------------Copying report to folder C:\Jenkins\workspace\Katalon Studio - ESB API\Reports\20180601_105318...Uploading report to JIRA...Start sending test result to JIRA issuesReport has been sent to JIRA--------------------------------------------------------------------------------Test Suites/TSC-Regression - 20180601_105317...........................7/7(100%)--------------------------------------------------------------------------------Test Suites/TS-Regression/TS-Retrieve_Balance_Info - Chrome (headless) - 20180601_105317...............................................................2/2(100%)Test Suites/TS-Regression/TS-Retrieve_PUK_Code - Chrome (headless) - 20180601_105317...................................................................2/2(100%)Test Suites/TS-Regression/TS-Query_MSISDN - Chrome (headless) - 20180601_105318........................................................................3/3(100%)--------------------------------------------------------------------------------C:\Katalon_Studio>exit 1 Build step 'Execute Windows batch command' marked build as failureRecording test resultsFinished: FAILURE>>>>>>The first 2 TestSuites under this TestSuiteCollection BOTH get the same timestamp 20180601_105317 in their name !
I would suggest that during the copying of the reports while running in console mode the subfolder for each TestSuite is also included.
This issue has been fixed and it will be delivered in the next release.
Dear Vingh,
please let us know if it was already fixed or still waiting for release.
If I am not mistaken, v.5.5 was published mid of July (after your comment)
Although I am still facing the issue
@Vinh Nguyen I have tried it on version 5.7.1 and still the issue exists. Please let us know when this will be fixed. This is one of the major issue as reports are getting overwritten after the execution.
I can not try this as Im using katalon docker image for running the test suite collection. This image by default uses -reportfolder option while triggering test cases.
Please let me know if we have any other workaround ? As this issue is getting very serious. With out this we are completely blocked on our pipeline.
even with Katalon docker image, you can “workaround” the issue in a same manner.
The Reports were correctly placed separately in separate subfolders under the Katalon Projects Report folder.
The problem happens when these reports are copied incorrectly into one Reports folder (resulting in overwrites due to duplicate subfolder names).
In Katalon docker image setup you are simply running a script too (as I understand it from the video I have seen on Katalon docker image), where you can add too a personal copy command to it after running Katalon to copy the subfolders correctly into your destination while keeping the subfolder structure intact, instead of trying to store all reports in one folder (as Katalon does).
Gerard,
IMHO, What you are saying is not quite correct. Since it pulls down the docker image every time, there is no way to modify the script which it is running. I am trying to get around the problem by having a test case called “logWorkaround” which simply copies the entire reports folder to a new folder which is timestamped. I add this at the end of every test suite, so all the reports for that suite get copied. This might result in additional logs but it is a small price to pay for a few extra bytes as long as all the logs are available.
Best,
Nanda
Hi Nanda,
I do not have enough info on how you are using Katalon Docker Image in your Automation process . But maybe using docker cp may be an option for you ? http://obrown.io/2016/02/20/docker-build.html
Hi Gerard,
I am using Katalon Docker as follows:
docker run -v “/home/circleci/project/Katalon”:/katalon/katalon/source:ro -v “/home/circleci/project/report”:/katalon/katalon/report -e KATALON_OPTS=’-browserType=“Chrome (headless)” -retry=0 -statusDelay=150 -reportName=“Reports” -testSuiteCollectionPath=“Test Suites/Executable Suites/All/SuperSet/CircleCI/AllTests” ’ katalonstudio/katalon:latest
docker cp might work although it is sometime since I have used it. Will investigate later when time permits.
I am also running in to this issue with the same cirumstances as Nanda. While there might be workarounds such as docker cp, the underlying bug does not (from here) look like a very difficult issue to fix.
If the bug turns out to be very difficult to resolve then any docker-script related workaround can be fixed by the Katalon Studio team as well, since they are the one that publishes the docker image.
even with Katalon docker image, you can “workaround” the issue in a same manner.
The Reports were correctly placed separately in separate subfolders under the Katalon Projects Report folder…
As Gerard suggested, the separate subfolders under the Katalon Projects Report is fine.
This might need a bit more refactoring. I’m now facing another problem. eval will exit 1 if the collection has any failed test. Because of that cp -r is not executed… still no reports
One solution may be:
In your shell script start with
set +e
and after executing Katalon use the following to remember Katalon’s Exit code:
katalonRC=$?
and after your other commands you can exit with Katalon’s Exit code with:
exit $katalonRC
BTW on Linux Im using “rsync -auv” instead of cp command.