TeamCity and Katalon Integration


#1

We are using Katalon Studio 5.9.1 to automate about 100 tests for our AngularJS applications. We did not find much on the Internet about TeamCity Continuous Integration, so I am posting this here for feedback.

There are three parts: creating the TC job, configuring the Junit results, and adding the Katalon HTML report.

1. Creating Katalon Job in TeamCity

Created one project with a separate build configuraiton for each Katalon test suite.

When setting up checkout rules, we had to check out to a subdirectory of the work folder. E.g., /Katalon

Next, create a Command-line Build Step to run the Katalon command. You can find the command on the Internet or generate it from Katalon studio.

D:\\Katalon\_Studio\_Windows\_64\\katalon.exe -runMode=console -projectPath="%system.teamcity.build.checkoutDir%\\Katalon\\My-Katalon-Project.prj" -testSuitePath="Test Suites/My\_Test_Suite" -reportFolder "Reports/%system.teamcity.buildType.id%/%build.number%" -browserType="Chrome"

Note that we used the -reportFolder option to specify the output. This was to ensure that each report was unique by build type and number. Notice the project ID and build number in the report path.

2. Configuring JUnit Results

This configuration is to display the test counts in the job run status, as well as to see the Tests tab. The test tab shows each test that passed or failed. Without this step, you have to comb the build log for failures.

Create a Build Feature of type XML Report Processing, report type of Ant Junit.

Set the path to:

+:Katalon/Reports/%system.teamcity.buildType.id%/%build.number%/JUnit_Report.xml

Note the path matches that specified about in the report folder. The JUnit_Report.xml file is generated by Katalon.

3. Add Katalon HTML Report

Edit on 12/13/2018, fix artifact path and report path.

From General Settings, Build Configuration, set the artifact path

+:Katalon/Reports/%system.teamcity.buildType.id%/%build.number%

From project, Report tabs, Create new build report tab.

We named it “HTML Report”. Enter the path: report.html

The reports.html file is a nice report that Katalon generates which includes screenshots of failed tests.

Good luck!


TeamCity as CI for Katalon Studio
Publish Katalon Project as a Website
#2

If you see an emoji in step 1, the frown face was the drive letter D followed by a colon.


#3

Thanks @Harold_Owen I’ll check it.


#4

I need to edit the original post. We started with the artifact path +:Katalon/Reports, and the project report path %system.teamcity.buildType.id%/%build.number%/report.html.

We had to change this to artifact path +:Katalon/Reports/%system.teamcity.buildType.id%/%build.number% and project report path report.html.

The original configuration caused TeamCity to publish all artifacts for ALL SUITES under the report path for one suite. This worked, but it caused TeamCity to retain huge amounts of artifacts with each run of each single suite. This lead to disk space issues.

Apologies if the original configuration affected anyone else.

Edit: TeamCity is happier. :slight_smile:


#5

Thank you for sharing this in great detail. I’ve moved it into Tips and Tricks to help people with similar needs.


#6

#7

Thanks for this first of all. Worked for me while trying to integrate my API tests in TeamCity.

The double quotes on the testSuitePath, reportFolder, and browserType are a bit confusing though. They’re different from the ones used for projectPath and may cause problems for newbies like me.


#8

@addimanarig, thank you for reporting this. I fixed in the original post. Sorry for the trouble!


#9

One of the advantages of integrating Katalon with TeamCity is that TeamCity tracks statistics for each test suite. Over time you can see how the number of tests in this suite grew initially. Now we are in a phase of trying to make the tests more robust/reliable. For some reason, the testers are finding tests pass locally when run from their own machines, but fail occasionally in TeamCity. So far we see no reason to blame TeamCity–rather we expect Katalon runs more slowly on the build machine.

Green dots show number of passed tests, and red dots are failures.


#10

I like this. Do you have any other tips for managing your test results?


#11

TeamCity also shows the Build Duration, the time required to execute the suite. The X-axis is the build number, 1 through 109, representing the number of times the suite has run in TeamCity. The duration increased over time as new tests and test steps were added to the suite.

The most striking take-away is just how much testing is accomplished by Katalon in a few minutes. This graph depicts a suite that covers a very critical part of our application. It would take a tester several days to manually test what this suite accomplishes in under 20 minutes. And the suite runs over and over, day after day.

The highlighted downturn at the end of the graph represents a performance improvement implemented by one of the developers. These tests work on a table of data that required several seconds to load. The developer tweaked the queries to make the table load in under a second, and the improvement was immediately seen in the graph.

I have not taken the time to investigate the two spikes in the graph. But, you can see how having this information may allow us to correlate spikes with other server event logs when they occur.