I have made and published a GitHub project:
This post discusses about the same problem as previous
but proposes a new / better solution.
Problem to solve
I have developed and published a tool named VisualTestingInKatalonStudio project on top of Katalon Studio, which enables me to take screenshots of Web services on browser and to compare the images between environments (e.g. Development and Production) or between 2 different timings (e.g. before and after work of system changes).
Last year (2020) I worked for an Application Service Provider (ASP) in Financial industry. They had approximately 40 customers, they provided many URLs to customers: 50 URL as average per a single customer. They maintained 3 environments (Development, Staging, Production) for every customers. This resulted 40 * 50 * 3 = 6000 URL for developers to test frequently to ensure the service quality.
A diligent staff in charge of the Web service mainteined a large Excel file which contained 40 sheets where the urls for each customers were listed. The following picture shows how it looked like:
I developed a Katalon Studio project which works on top of the Visual Testing framework supplied with the Excel file. I wanted to perform visual testing over the mass of URLs.
However, a problem arose. The Visual Testing tool takes approximately 10 seconds per a single URL to test. So if I ran the tool against all 6000 URL as a batch, it would take me 6000 * 10 seconds = over 17 hours. Obviously it’s too long to repeat frequently.
Therefore I wanted to select smaller number of URLs by some criteria out of the Excel file to make the test run in a shorter time period. If I choose 60 URLs, then the tool will run in 600 seconds = 10 minutes. That’s OK. 10 minutes break of work for tea will be welcomed.
Also I wanted the tool to be flexible enough. For example, yesterday, I tested the production URLs for CompanyA; today, I want to test the development URLs for CommpanyB; tomorrow, I would want to test the staging URLs for CompanyL + CompanyM + CompanyN; next week, I will test the production URLs which ends with a string login.html of all 40 customers; etc.
I tried to find a solution, and decided to introduce a set of GlobalVariables to the testing project which would express URL selection criteria. I enumerated 5 GlobalVariables, each of which may take a range of possible values as follows:
| No. | GlobalVariable name | possible values |
|---|---|---|
| 1 | CONFIG |
'./Include/fixture/Config.xlsx' |
| 2 | ENVIRONMENT |
|
| 3 | CATEGORY |
|
| 4 | INCLUDE_SHEETS |
|
| 5 | INCLUDE_URLs |
|
| OK. I can restate my problem to solve. How can I specify a particular set of values for these 5 GlobalVariables when I execute my test in Katalon Studio? |
Katalon Studio provides a feature named Execution Profile . In an Exceution Profile, you can define a set of name=value pairs of GlobalVariables. You can prepare as many Execution Profiles as you want using Katalon Studio GUI. So I createded bunch of Profiles, each of which contains 5 GlobalVariables with values assigned. The following screenshot shows an example:
Now you should remember, Katalon Studio allows you to appoint only a single Execution Profile for a test execution.
I realized a difficulty. I had a lot of possible selection criterias: 1 CONFIG * 3 ENVIRONMENTS * 5 CATEGORIES * 6 INCLUDE_SHEETS * 3 INCLUDE_URLs = 180. Therefore I had to prepare 180 Execution Profiles in Katalon Studio GUI. It was a crasy job. The following screenshot shows how the project looked like:
Further more, the range of `GlobalVariable.INCLUDE_SHEETS’ values was initially 4, but it could possibly increase to 40. In that case, should I prepare 1800 Profiles maually? No way!
I found a fundamental design problem in Katalon Studio here. This issue has remained unresolved for me since May 2020.
Solution
In the Keywords directory, I will write a few Groovy enums which declares GlobalVariables with enumeration of possible values. I will call them as Domain Enums .
Also I will write a Groovy script (actually a Test Case script) which generates hundreds of *.glbl files derived from the comibination of the Domain Enums. All of *.glbl files are in a valid XML format as a serialized Execution Profile of Katalon Studio.
I will put the generated *.glbl files into the Profiles folder of a Katalon Studio project. Consequently I will obtain some hundreds of Excecution Profiles ready to use.
With this facilities provided, I do not have to manually create hundreds of Execution Profiles any longer. The tool scripts will smoothly generate them for me.
Solution Description
Domain Enums
The following Groovy code reporesents the GlobalVariables and their possible values:
- their.globalvariable.domain.Category
- their.globalvariable.domain.Env
- their.globalvariable.domain.IncludeSheets
- their.globalvariable.domain.IncludeURLs
- their.globalvariable.domain.SaveHTML
The source code of Env is quoted here to give you better idea what “Domain Enum” looks like:
package their.globalvariable.domain
import com.kazurayam.ks.globalvariable.ExecutionProfile.GlobalVariableEntity as GVE
import com.kazurayam.ks.globalvariable.SerializableToGlobalVariableEntity
enum Env implements SerializableToGlobalVariableEntity {
DEVELOPMENT("Development"),
PRODUCTION("Production"),
STAGING("Staging");
private String id
String getId() {
return this.id
}
Env(String id) {
this.id = id
}
GVE toGlobalVariableEntity() {
GVE gve = new GVE()
gve.description("")
gve.initValue(this.getId())
gve.name("ENV")
return gve
}
}
Execution Profile
Files under the Profiles folder is a XML file. For example, main_AllSheets_AllURLs_DEVELOPMENT_0_NoSave.glbl files would look like:
<?xml version="1.0" encoding="UTF-8"?><GlobalVariableEntities>
<description/>
<name>main_AllSheets_AllURLs_DEVELOPMENT_0_NoSave.glbl</name>
<tag/>
<defaultProfile>false</defaultProfile>
<GlobalVariableEntity>
<description/>
<initValue>Development</initValue>
<name>ENV</name>
</GlobalVariableEntity>
<GlobalVariableEntity>
<description/>
<initValue>0</initValue>
<name>CATEGORY</name>
</GlobalVariableEntity>
<GlobalVariableEntity>
<description/>
<initValue>[]</initValue>
<name>INCLUDE_SHEETS</name>
</GlobalVariableEntity>
<GlobalVariableEntity>
<description/>
<initValue>[]</initValue>
<name>INCLUDE_URLS</name>
</GlobalVariableEntity>
<GlobalVariableEntity>
<description/>
<initValue>false</initValue>
<name>SAVE_HTML</name>
</GlobalVariableEntity>
</GlobalVariableEntities>
Reading and writing this XML format requires some complexed coding. Enveryting is encapsulated in the following class.
Execution Profiles Generator tool
The following is the groovy script that generates a bunch of Execution Profiles. The script enumerates the combinations of possilbe values of the Domain Enums.
When I execute it, I got the following output in the console:
2021-03-10 20:50:43.162 INFO c.k.katalon.core.main.TestCaseExecutor - --------------------
2021-03-10 20:50:43.166 INFO c.k.katalon.core.main.TestCaseExecutor - START Test Cases/tools/generateExecutionProfilesWithDomainEnums
/Users/username/ExecutionProfilesGeneratedWithDomainEnums/Profiles/main_AllSheets_AllURLs_DEVELOPMENT_0_Save.glbl
/Users/ExecutionProfilesGeneratedWithDomainEnums/Profiles/main_AllSheets_AllURLs_DEVELOPMENT_0_NoSave.glbl
/Users/ExecutionProfilesGeneratedWithDomainEnums/Profiles/main_AllSheets_Login_DEVELOPMENT_0_Save.glbl
... <over 400 lines>
/Users/ExecutionProfilesGeneratedWithDomainEnums/Profiles/main_CompanyC_Login_STAGING_3_NoSave.glbl
/Users/ExecutionProfilesGeneratedWithDomainEnums/Profiles/main_CompanyC_Top_STAGING_3_Save.glbl
/Users/ExecutionProfilesGeneratedWithDomainEnums/Profiles/main_CompanyC_Top_STAGING_3_NoSave.glbl
/Users/ExecutionProfilesGeneratedWithDomainEnums/Profiles/main_GroupG_AllURLs_STAGING_3_Save.glbl
/Users/ExecutionProfilesGeneratedWithDomainEnums/Profiles/main_GroupG_AllURLs_STAGING_3_NoSave.glbl
/Users/ExecutionProfilesGeneratedWithDomainEnums/Profiles/main_GroupG_Login_STAGING_3_Save.glbl
/Users/ExecutionProfilesGeneratedWithDomainEnums/Profiles/main_GroupG_Login_STAGING_3_NoSave.glbl
/Users/ExecutionProfilesGeneratedWithDomainEnums/Profiles/main_GroupG_Top_STAGING_3_Save.glbl
/Users/ExecutionProfilesGeneratedWithDomainEnums/Profiles/main_GroupG_Top_STAGING_3_NoSave.glbl
2021-03-10 20:51:03.394 INFO c.k.katalon.core.main.TestCaseExecutor - END Test Cases/tools/generateExecutionProfilesWithDomainEnums
The Katalon Studio GUI showed some hundreds of Execution Profiles were generated and are ready for use.
Benefit
The changes in Domain Enums was really problematic. For example, when we need to support 2 more INCLUDE_SHEETS values, so that I need to create 120 Execution Profiles, how much volume of works involved?
If I were manually creating Execution Profiles, I will just exclaim No, I can’t do it!
But once I am provided with the tool, I will just do 2 things.
- modify a Domain Enum IncludeSheets slightly; add 2 enums.
- kick the Test Cases/tools/generateExecutionProfilesWithDomainEnums
Wait for approx. 10 seconds, then stop KS and restart it. The new set of Executiion Profiles will be ready for use. No hussle will be involved.
How to reuse this solution in your Katalon Project
The idea of Domain Enums and generating Profiles/*.glbl files by script is essential and can be reused. The codes in this project are small. A seasoned Groovy programmer would be able to read them through and mimic as needed.