[Open Discussion] How are you organizing your test suite collections?

Hello all,

We are looking into improving the test scheduling capability of Katalon, and I want to get a conversation going so that we know more about your thoughts and needs. Now, I know you’re wondering what’s that gotta do with the title of this topic, so let me get started with that.

In TestOps you can schedule your Katalon tests to be executed at a specific time and repeated after a certain interval (don’t worry this isn’t a piece of marketing content). From TestOps, go to a project, go to Test Execution tab, click Schedule Test Run , and you will see the following screen.

Previously, with TestCloud (our cloud test infrastructure), we chose to limit flexibility in exchange for ease of getting started. A Linux instance with all the latest versions of popular browsers installed provides the best execution speed.

But you can see how some people would want to override that default environments. What if I want to execute with Windows and on Firefox only? When you schedule a test execution, overriding execution environments configured in KS is clearly desirable.

But there are some questions that we’re wondering: do the majority of users want to override for the entire TSC only? Or do they also want to override for specific TS within the TSC?

Let’s say you have a TSC called TSC A which contains test suite TS A , TS B and TS C .

  • Do you want to just override for TSC A , that is, " I want to have all three test suites inside executed on desktop Chrome browser".
  • Or do you want to do something like, “Okay for TS A I want to execute on desktop Chrome browser, but for TS B and TS C I want to execute on mobile Chrome browser”.

I think that would depend on how you’re organizing your TSC.

  • If the test suites in your collection test the same functionality (i.e constitution some kind of flow), then it would make sense that all of them have to be executed on the same environment. These test suites depend on one another so that the TSC can execute meaningfully.
  • But if the test suites in your collection are grouped based on some purpose (let’s say “regression”), then I can imagine a scenario where each test suite would be executed on a different environment (to see if a feature “Purchase order” works across desktop and mobile browsers, for example). These test suites are essentially independent.

So, what is your thoughts on this? How are you doing things right now? Let us know!

  • Most of my TSCs are organized so that the test suites are dependent on one another.
  • Most of my TSCs are organized so that the test suites are independent.
  • Roughly half of my TSCs is organized in one way, the other half is organized in the other way

0 voters

I hope you would see your expert opinions reflected in our product decisions.

Regards!

2 Likes

For some reason I cannot vote, I get this:
vote

using chrome browser on Linux.

@anon46315158

Hmm that’s weird, are you using any extension at all? Maybe refresh and see if it works.

It worked now … voted.
My thoughts here, in addition to the vote:

  • I am organizing my suites independently (and as much as possible tescases should be independent also)
  • However, to override the environment for a given collection, overriding it for all suites in the collection make more sense for me.

Why?

To organize my workflow, I am using a ‘Testrail like’ approach, e.g:

  • for any new feature to test I create a new ‘milestone’
  • for any environment subject to test I create a new test run.
  • If not all test are relevant for the given feature, I will exclude them (either from the suite of from the collection). For such, ‘dynamic test suites’ feature i think it is pretty handy.

Side notes: I am not an active Katalon user, I am just describing how I will use it …

2 Likes

Thank you, that’s interesting, let me recap and let me know if I got that right.

For a feature, let’s say “Purchase Order”, you’d create a new milestone. I assume this would be something equivalent of a release, or is it the TSC itself?

If you want to test that “Purchase Order” on 3 environments, Chrome, Firefox and Edge, then you’d create 3 test suites within a test suite collection. When you execute that collection, you’d want to override the environment for each test suite, right?

Sorry, @ThanhTo, I don’t routinely use TSCs. The few that I have are used only to test/debug forum questions. :upside_down_face:

I don’t routinely use Test Suite Collections either. I would rather try to avoid TSC as much as possible.

The only case I have used TSC was this case: Background story: why I developed this library

The reason is that Katalon Studio does not provide any means of loading an Execution Profile in test script programmatically. Katalon Studio forces me to create a Test Suite Collection, in which I execute the Test Suite processURL 2 twice while applying 2 Profiles demoProductionEnv and demoEnvironmentEnv .

In this case, I did not require testing with multiple browsers, I did not require running tests parallel in this case. So, I did not wanted TSC, but was forced to use it.

I developed my Custom keywords to load Execution Profiles by Test Case scripts to avoid TSC.

I think that the parallel-ness of Test Suite Collection is not very useful.

Why? — I would tell you.

Let me assume I have a test scenario that I want to run with multiple browsers and in parallel. I would want to run 3 types of processing steps.

  1. pre-process ---- create work directory, etc
  2. processURL ---- run the test
  3. post-process ---- for example, compiling my custom reports

I would want to execute the preprocess → processURL → postprocess SEQUENTIALLY as they are dependent. Obviously the processURL should launch after the prepress. And the postprocess should start after the processURL.

And also I would want to execute multiple times of processURL IN PARALLEL; multiple browser types, multiple Execution Profiles. Of course all processURL will be independent each other.

Hopefully I want to distribute the processURLs across multiple machines on Cloud for ultimate speed. If I could use 8 boxes in parallel, simply the duration would be 8 times shorter. I know some users have mighty TSCs which run for days. They may appreciate this.

This diagram makes me remember the code structure of the good-old programming language AWK, for example:

BEGIN {
    FS="[^a-zA-Z]+"
}
{
    for (i=1; i<=NF; i++)
        words[tolower($i)]++
}
END {
    for (i in words)
        print i, words[i]
}

But, the current Test Suite Collection forces me to chose either of SEQUENTIAL or IN PARALLEL for all Test Suites in a Test Suite Collection. The organization of TSC makes it less useful.

At the risk of drifting the topic just a little…

That’s interesting @kazurayam – thank you. I do something VERY similar.

I think I’ve shown this technique to @ThanhTo in the past and maybe others have seen my posts about it. One of the reasons I don’t “need” TSCs is because I developed something called STCs in my early days using Katalon:

STC == Suite Test Case – a Test Case that behaves like a Suite.

An STC is my way of collecting related/connected TCs together in a chain. I addition, TCs stored in an STC can use “before” and “after” settings which are stored in JSON format in the TC properties. So each TC can call a TC before and after it is executed:

This allows me to treat Test Cases as “steps” in a much larger process.

STCs also support repetition over a period of time. The STC will loop for the period specified at startup. Using this facility, I can easily create a “pulse” – taking the pulse of a critical server. Because my reports are live the moment an STC is started, I can run an STC 24/7 monitoring critical servers for unexpected outages (Johnson & Johnson and Merck cloud systems for the last couple of years due to Covid19). Katalon is probably not the best tool for this, but I’m just checking our cloud provider is doing their job (the fine details can be left to them).

Not sure any of that helps you @ThanhTo… thought you might find it interesting.

1 Like

Many wheels have been reinvented.

2 Likes

@Russ_Thomas

You have reinvented a wheel you liked.

May I ask you, why did you decided to use Katalon Studio?

I doubt it if you need KS at all.

You’ve said that before.

@kazurayam

That’s interesting, but a bit back to the point. It seems like you’d still opt for the sequential mode of execution, that is, grouping test suites into some sort of flow and execute that. I think parts of the reasons you want to combine sequentially and in parallel is because you’d want to speed things up. In other words, if you were given a kind of perfect TSC that can execute in the fastest way possible, you’d organize the test suites sequentially in that TSC, right?

@Russ_Thomas Ah I remember that approach.

I think you two are experienced enough to customize Katalon to whatever direction you wish to choose, but what I’m interested here is the problem space that you’re operating in. Would you try to test a whole flow (using a bunch of TS or whatever) or would you try to test one flow (contained within a TC or TS) on different environments? What use case do you think is more relevant to you?

1 Like

Correct. For me, faster execution of tests is the ONLY possible reason why I would execute things in parallel. Is there any other values? If yes, please tell me. If the tests do not get faster, I find no reason why executing tests in parallel.

When I first learned Test Suite Collection’s “Execution Mode: parallel”, I hoped that this would auto-magically make my long running test far faster. I grouped my 40 Test Cases into 4 Test Suites. I made 1 Test Suite Collection with 4 Parallelism. I expected that parallel TCS will run 4 times faster than a sequential TSC. But I got no speed gain. In fact Parallel execution on a single PC run even worse. I learned that, if I want to make my long running tests run faster, 1 single PC is not enough. I need 4 or more number of machines where I distribute the processes to run them in parallel.

Katalon’s TSC does not support distributing processes over multiple cloud boxes. So I find TCS is not valuable for me at all.

1 Like

Not exactly.
For me, a test suite consist of a bunch of testcases relevant for a specific validation target …
I will rather organize them based on ‘regression, smoke, integration’ etc criteria (e.g a smoke suite can be a subset of regression, new tests for the new feature will always be added to regression and so on)

Based on the ‘stage’ (deploy environment) I will execute one, or the other … e.g in staging / qa I will always execute a ‘full suite’ (regression with the new features added) but later for production deploy I will only run a smoke to confirm the deploy is right.

Therefore, for a given milestone (feature) I will either create a single test collection with whatever environment and execute it for each environment by overriding the execution params from CI … or one collection for each environment.

1 Like

That’s hard to say without a real world need presenting itself for my consideration.

  1. I don’t need to support multiple environments. >0.5M users all using Windows desktop browsers set the specifications for the AUT. >80% use IE/Edge, the rest Chrome.

  2. My AUT is a broad-spectrum app covering many flows in the domain. One suite is all I need and that is what is routinely executed every day.

I have a bunch of smaller suites where the TCs are drawn from the main suite but I rarely run them. I can’t even remember the last time I ran them. I guess I could use them to run in parallel but then I’d lose the single report the devs are expecting. Remember, I (we) have a philosophy that “passes are noise” here – all we care about are fails. So my reports are focused on giving the devs EXACTLY what they want – a list of failures, all steps intact.

Again, not sure any of that helps you much.

@Dave_Evers do you have anything to contribute?

2 Likes

speaking about paralellization… i never felt the need to use this particular katalon feature.
if i need to run a huge bunch of tests faster, i will do this by cleverly splitting the project in independent components.
smaller and targeted projects are better to maintain, imho, than a huge one
to run them in parallel i will abuse the ci power: spin more nodes, create more jobs etc

1 Like

Hi @ThanhTo,

We use Test Suite Collections to drive our Jenkins/Azure DevOps triggered continuous integration and continuous deployment pipelines.

For example, we have a daily environment health check for our Production and Demo environments. We can use the same tests cases for each environment by simply picking the Test Suite Collection profile to use.

We also have multiple testing environments so using Test Suite Collections management allows us to run the same test cases on each environment via our continuous integration and continuous deployment pipelines or on demand.

3 Likes

the most enjoying part of the current thread is… somehow people think that, by using acronims or whatever are named, we look professional.
tsc, tcs, ssct, xxls, kkux, and so on
cmoon guys, it is not a live chat, your fingers wont die if you type more letters

1 Like

I must admit, I am a bit confused reading TCS and TSC :rofl: So are the test suites in your test suite collection organized independently and you configure a different profile for each test suite?

1 Like