About Scheduling Test Suite Collections on TestCloud through TestOps

The Katalon platform allows teams to involve business people in the software quality process by allowing them to easily select what, where, and how often to run automated tests (via TestOps).

And through many conversations with users, we find out there’s always a need for other parties, mostly business people, to get involved in the test run scheduling and execution triggering.

With TestCloud, business people can join in testing activities more easily

Previously, users have to set up TestOps agents to schedule test runs. But the setup process seems to be complicated and hassle-some for non-technical people.

But now, with TestCloud in place, users can get access to a cloud-based infrastructure easily, removing the friction of setting up an orchestration or physical infrastructure. Scheduling test runs on Testcloud environments can be done with just a few clicks in TestOps.

But there’s a roadblock to scheduling test suite collections

When using TestOps to schedule test runs on TestOps agents, users can override execution environments for a test suite collection. But they cannot do the same with TestCloud.

When scheduling a test suite collection on TestCloud (via TestOps), the only environment option is a Linux instance with all the latest versions of popular browsers installed. This setting provides the best execution speed.

We’ve previously decided to trade the flexibility in exchange for the ease of getting started and the execution speed.

But what happens if somebody wants to override the above execution environment for a test suite collection? TestCloud does not offer this flexibility, at least for now. And we want to know if there’s really a need for this.

So, we go on and collect some insights into how QAs organize their test suite collections

There are more questions we’re wondering: To what extent should that flexibility be? Do users want to override for an entire test suite collection, or do they want to override different environments for different test suites in the collection? The answer takes us in different directions.

Let’s say you have a test suite collection that contains test suites A, B, and C.

  • Do you want to override the entire collection? Let’s say, you want all the three included test suites to be executed on a desktop Chrome browser.

  • Or do you want to override the environment for each test suite in the collection? For example, you want to run test suite A on a desktop Chrome browser, but B and C on a mobile Chrome browser.

That’d depend on how the test suites are organized:

If the test suites in a collection test the same functionality (i.e they constitute some kind of flow), then it would make sense that all of them have to be executed in the same environment. These test suites depend on one another and their sequential executions make the suite collection meaningful.

On the other hand, if the test suites in a collection are grouped based on a shared purpose (for example, regression), then there are scenarios where it’d be necessary for each test suite to be executed in a different environment (to see if a feature “Purchase order” works across desktop and mobile browsers, for example). These test suites are independent and thus their executions can be parallelized.

After asking the experts on our forum the way they’re organizing test suites and our own automation engineers for advice, the evidence seems to support the pattern that QAs would generally organize test suites independently in a test suite collection. It is a best practice to minimize dependency between test suites so that one failure does stop other executions.

Read the discussion about this matter on the forum.

Improvement is Needed

Since a majority of QAs choose to organize test suites independently from one another, it’s clear that they need the ability to override the environment for each test suite within a collection.

We believe by allowing users to have this flexibility when scheduling test suite collections on TestCloud environments, different roles in software development, including business stakeholders, can obtain the platform more quickly and join in testing activities more easily.

Following what has been described, we’ll continue to look into the way to make enhancements to TestCloud, allowing users to select a set of test suites and override their execution environments in a test suite collection when scheduling via TestOps.

Hope this small article helps you get more insights into Katalon. Have a nice day!


Thank you @ThanhTo for sharing the information and collaboration on this article :star_struck:. Looking forward to more contents from our community.