How to make the second Visual Basiline in TestOps?

Product Forums Katalon TestOps
1

Hello everyone :slightly_smiling_face:
We use visual testing on a project and have several different test suites that use this feature. When running any of these suites the same baseline is used. How can we make one suite use one baseline and the next suite use a different baseline?

2

Hi Naumchik,

Thank you for the recommendation we will look into this and inform when there’s any update, stay tuned.

1 Like
3

Hi n.naumchik,

Hope you’re doing great.

I believe that you are still using Visual Testing feature. If not, you can try to register the Trial Visual Testing to experience more exclusive features (please let us know if you have any difficulties in the registration). Visual Testing is now able to generate each different baseline collection (a set of baseline images) which is relevant with corresponding test suites.

To make one suite use one baseline collection, you can schedule a test run containing the suite and select the ‘Generate a new Baseline Collection’ option in the ‘Visual Testing’ drop-down list (Please have a look at the attached image for details). After creating the test schedule, you run it and a new baseline collection will be generated and associated with the test run.

image
image1816×1482 242 KB

Thank you.

1 Like
4

@nghi.hua
Hello, I have the same problem.

Do you mean that this feature is only supported by one Test Suite, which means that if we want to use Visualtest on multiple TestSuites, we need a paid license for TestCloud? (Schedule test run requires that license, right?)

(EDIT)
We are currently running multiple tests in Azure DevOps Pipeline and Schedule Test run in TestOps doesn’t seem to be the solution…

5

@n.naumchik
Have you been able to resolve this? If so, could you please share your solution?

6

1/. Generating a new baseline collection or associating the test schedule with one existing baseline collection is now available for the following types:

  • Test Suite (select desire option in the Visual Testing drop-down list)

  • Test Suite Collection (select desire option in the Visual Testing drop-down list)

  • Katalon Command (Katalon Studio from 8.4.0 onwards, add -testOps.baselineCollectionId=<CollectionId> parameter in the command)

    image
    image878×1364 75.2 KB

2/. You are running tests in Azure DevOps Pipeline, then the solution will be:

  • Install the Azure DevOps Extension (please read here for more details)

  • Once Azure has been successfully integrated with TestOps and before you run the tests in Azure, you need to include “-testOps.baselineCollectionId=-1” param to the Command Arguments.

  • After the tests are run in the Azure, the result will be uploaded to TestOps. At the same time, a corresponding baseline collection named “System-generated Common Baseline Collection” will be generated. You can later rename this collection to different name.

3/. Besides trial/paid TestCloud, when scheduling a test run, you can select local test environment.

Please try and let me know if you have any concern.

7

@nghi.hua
Thank you, I’m trying your solution 2.

I don’t think this method is compatible with multiple TestSuites either, isn’t it?

I specified baselineCollectionId=-1 for the first Test Suite Correction and baselineCollectionId=-2 for the second, but both compare to the baseline with ID 1.

8

Currently, Baseline Collection param in the command has limitation to be able to generate each different baseline collection. We are going to improve it very soon.

In the meantime, I’d like to propose you a workaround solution so that you can have each different baseline collection for each corresponding test suite. It’s a little complicated, but hope it can help.

  1. Assumption: you have 2 test suites which are run in Azure DevOps Pipeline
  2. Next steps: you create an Azure Repo which contains all the test suites in TestOps (please refer to detailed info in here)
  3. Create a local agent in TestOps (please refer for detailed info in here)
  4. Schedule a test run within TestOps for each test suite
    image
    image1920×1574 162 KB
  5. Start running the schedule. If you have configured 2 schedules for 2 test suites, then you will have two different generated baseline collections (for example, the baseline collection ID are respectively 1 and 2)
  6. Next, you can use each of those ID in the command which you are configured in the Azure DevOps Pipeline. For example, you can add 'baselineCollectionId=1' param to the command of test suite 1 AND add 'baselineCollectionId=2' param to the command of test suite 2
1 Like