How to configure visual testing for test suites from separate projects to work in the same TestOps project

Product Forums Katalon TestOps
, ,
#1

I’ve been using Katalon Visual Testing on a small scale, partly to see if it fits our requirements before expanding its usage and potentially upgrading to Visual Testing Professional.

I have 2 Katalon projects - ‘WEB’ and ‘MOBILE’ - and I have a visual testing test suites/collection in each project - let’s call them WEB-TS-1 and MOB-TS-1 respectively. Let’s say there are 10 checkpoints in WEB-TS-1 and 4 checkpoints in MOB-TS-1.

When I execute WEB-TS-1 for the first time, then review Visual Test Runs, I have to go through each checkpoint in TestOps for #1 and ‘Mark as Passed’ for all checkpoints (‘New’ images) and then ‘Save to baseline’, so I now have 10 images in my baseline collection (system-generated - I’m executing these via Runtime application). All good so far.

However, when I execute MOB-TS-1 for the first time, when I review the Visual Test Run (#2), I have 14 items to review - 4 ‘new’ images and 10 ‘missing’).

This obviously causes a ‘conflict’ going forwards whatever I do - it seems that unless all checkpoints are executed on every execution of any visual test, then this will happen.

I thought that I may need to use separate Visual Baseline Collections, but this option seems only to be available if I schedule test runs in TestOps and/or (I’m not quite sure) subscribe to Visual Testing Professional?

Is there any way to configure visual testing so that I can execute multiple, separate visual tests, without the checkpoints conflicting with each other please?

#2

1/. I believe that you are selecting the same TestOps project for both Katalon projects. So the results from those Katalon projects will be uploaded to the same baseline collection in the same TestOps project. If you want to use separate Baseline Collections, you should create another TestOps project (for example: Project A and Project B). Then you select ‘Project A’ in the Katalon TestOps configuration and execute test suites in the ‘WEB’ project. After that, in the Katalon TestOps configuration, you change to select ‘Project B’ and continue running the test suites in the ‘MOBILE’ project.

After finishing the above steps, you can access to the Visual Testing in ‘Project A’ and ‘Project B’ in order to see the uploaded images are not merged in the same baseline collection.

image
image1920×1037 96.8 KB

2/. As you said you can have separate Visual Baseline Collections by scheduling test runs in TestOps. This feature is currently available for everyone. That means it does not need to subscribe Visual Testing Professional to use it.

Thank you.

#3

@nghi.hua

Thanks for your response - I understand exactly what you’re saying.

I think that the way we have structured our tests and projects is conventional and in line with what would be expected (i.e. we have separate Katalon projects for our web and mobile projects), and I also think the fact that we point both projects to the same TestOps project is also not abnormal (indeed, you provide a facility to point to any given project in the organisation).

So on that basis, to effectively limit Visual Test comparisons to have only one baseline per TestOps project (unless we use TestOps to schedule test runs - something we choose not to do at present) that can result in the conflict I’ve described seems like either an oversight, or an artificial limitation (I’d be happy to hear a different explanation if these neither of these are the case).

We separate our Katalon projects by web and mobile in-part because when a new project is created, the user is asked to choose a desired project type. One workaround to the issue I reported would be to combine all of our tests into a single project, but this would over-complicate things quite considerably and seems to go against how Katalon suggest we create new projects.

We point both Katalon projects to the same TestOps project because we want to see one overall picture in terms of test results/data; again, this follows your software where the choice of TestOps project is one that we’re free to make ourselves and causes no issues for us other than this visual testing issue.

And if we’re free to point a Katalon project to any TestOps project, is there any reason why we can’t select the TestOps Visual Testing Baseline for each Katalon project?

We try to be pragmatic about such things, so as an alternative way of looking at this, is there, or could there be a way to view test results in TestOps consolidated across all Projects? Perhaps like the following, but showing results for all projects…

Katalon TestOps?
Katalon TestOps?

This seems to be a useful thing to have anyway, but this would enable us to point each Katalon project to its own corresponding TestOps project (thereby solving the issue of conflicting visual baselines and comparisons), and also enable us to be able to view test results on an overall basis?

#4

@kevin.mcandrew1

It’s really exciting when you are interested in using Visual Testing and have insightful questions for us.

I believe that ‘only one baseline per TestOps project’ was the exact purpose when Visual Testing was first introduced in 2020. At that time, we would want to provide users with an additional feature to assist them in checking the user interfaces. So we only have only one baseline collection to help users be able to successfully compare the UI and return corresponding results first.

We continue listening to users’ feedback and enhancing the Visual Testing to have multiple baseline collections in 2022. Regarding your concern about having the ability to select different baseline collections for each Katalon project, it is definitely great feedback to continue improving Visual Testing so that it can associate each Katalon project with each baseline collection in the same TestOps project. In 2023, we cannot tell you exactly the timeline to implement and release this improvement because we’ve had planned prioritization. In the meantime, we hope you can continue using Visual Testing as we recommend:

1/. Select different TestOps projects for different Katalon projects
2/. Schedule test runs directly in TestOps

Regarding the concern about having the ability to view test results across all TestOps projects, we have not thought of it yet. We will pass your request to the relevant team to consider the plan for it.

Thank you so much.

#5

@nghi.hua

Thanks for your response - certain the ability to specify baseline collections would provide flexibility to both arrange our projects optimally for our purposes, and organise visual testing flexibly too, and the ability to gain an overall / consolidated view of all/selected projects in TestOps would also I believe be a great enhancement - I look forward to seeing these developments in the future.

Whilst I fully understand your recommendations on how to use visual testing, I hadn’t seen such recommendations up until now - please can you provide links to this information please (we do try and follow recommended practises wherever possible).

Thank you

#6

@kevin.mcandrew1

To schedule a test run for visual testing purposes, please follow the following steps:

1/. Prepare a git repository that contains all the test suites/test suite collections from all Katalon projects
2/. Integrate the git repository to the TestOps (see the link)
3/. Schedule a test run in the Test Execution page:

  • Select the script repository which is integrated in step #2
  • In the “Test Suite” or “Test Suite Collection” tab, select a desired repository path containing the test scripts in the “Object Name” drop-down list
  • Select an environment to execute the test
  • Open the “Advanced Settings” to see the “Visual Testing” drop-down list. The default option is “Generate a new baseline collection”.

4/. Then click “Schedule” and wait until the test execution is done.

Please see the recorded clip for more details.

Just let us know if you have any difficulties while doing these steps. Thank you.

#7

@nghi.hua

Thank you for this information, I was broadly aware of what was required and this is a very handy guide (and video) - thank you - though I was more intrigued about this being the ‘recommended’ option as much as anything :slight_smile: