Hi,
I have 2 questions with regard to how TestOps aggregates the execution results and displays them in the reports - test runs.
In a nutshell: I think I miss a parameter, let’s call it “uniqueTestResultsEnvironmentSource” that I could use to group my execution results in TestOps in a meaningful way.
1° Executions from Katalon:
I have a test suite collection that runs against our TEST environment and a test suite collection that runs against our ACCEPTANCE environment. Like good test practice, these are actually the same tests but:
- differently versioned (e.g. test cases on TEST are v.2.0 of the application, while on ACC they are still v.1.0).
- linked to different execution profiles. A global variable will determine which url (TST or ACC) will be opened.
Note: There might be other global variables that are different, like a language (fr vs nl) the application needs to be opened with, or other browsers etc.
My problem (see attached screenshot):
I want to be able to see the execution reports per environment.
As is:
The /executions overview shows me all runs by default. Detail of executions/1073 ACC_1.M makes his offset against run #1072. So for instance there is supposedly a drop of Test Results (-20) & Passed (-20) from 61 to 41.
To be:
Offset should be done against the previous ACC_1.M run, #1069, showing the actual change of Passed -1 and Failed +1.
I know I can filter on the testSuite, like “TestSuiteCollectionEntity.name:ACC_1.M”, but I cannot persist this filter (it’s not in the url) to save it as a default view. The offset in the details of the executions neither are persisted on this filter, so that won’t solve that issue neither. So perhaps Katalon can allow users to define a “uniqueTestResultsEnvironmentSource” for each testSuiteCollection, so that in TestOps reporting this can be used to group exuctions?
2° Test executions from another source, which get uploaded to TestOps in a JUnit via Katalon Report Uploader (as described here https://docs.katalon.com/katalon-analytics/docs/project-management-import-cli.html#usage-for-cli).
My problem (see attached screenshot):
I want to be able to see the execution reports per source.
As is:
#1070 is the run from the TST environment. #1072 is the run from the ACC environment. For these test there is not execution profile or test suite collection name. So I cannot filter these. Plus same problem on the offset as above.
To be:
The (optional) environment parameter “uniqueTestResultsEnvironmentSource” could be provided in the Report Uploader.
Note: I don’t use any of the /test-project configurations in TestOps. Cannot use them at the moment due to the infra is setup at my client.
