Using AWS SDK in Katalon Studio, how to resolve external dependencies with Gradle

I have made a demo project in GitHub.


Problems to solve

(1) I want to use Amazon S3 in my test cases

My Visual Testing in Katalon Studio project enabled me to do visual regression testing. It compares 2 URLs of my AUT (production & development) visually in a time-slicing manner.

Now I have a plan to develop new way of visual regression testing. It would be a chronological approach. The new project will compare screen shots of the current AUT with another set of screen shots taken previously — taken 3 hours ago, taken yesterday evening, or taken last week. Chronological visual regression testing would enable me to ensure the system’s stability before/after the AUT changes in production. I would be able to automatically compare hundreds of pages against the previous images taken before/after the change.

But this new idea raises an issue to me. I need to store a lot of versions of screen shot sets. Driving the chronological test in a Continuous Integration system, for example once a day, will result huge number of screen shot files (over 10000 easily). How can I manage the accumulated screen shot files? One idea has come up to my mind. Why not using Cloud Storage service such as Amazon S3?

Cloud Storage seems to be promising for me. With it, I do not have to worry about the capacity, it’s cheap, files older than 1 month will be automatically deleted, … etc.

Therefore I want to call AWS Java SDK for S3 in Katalon Studio so that my test case can transport screenshot files to and from Amazon S3.

(2) How to resolve external dependencies?

AWS document provides enough information how to use AWS Java SDK for S3 in my Java/Groovy codes. So I tried to write a test case in Groovy in Katalon Studio which calls AWS API for S3. I wanted to list my S3 Buckets in my AWS account. Soon I encountered a blocking problem.

Katalon Studio requires all of the external dependencies (jar files) put into the <projectDir>/Drivers directory. I knew I need the aws-java-sdk-s3-1.11.470.jar. But there are a lot more dependencies (in fact 14 jars). Unfortunately Katalon Studio does not provide any dependency management.

A quote from Gradle document

What is dependency management?

Software projects rarely work in isolation. In most cases, a project relies on reusable functionality in the form of libraries or is broken up into individual components to compose a modularized system. Dependency management is a technique for declaring, resolving and using dependencies required by the project in an automated fashion.

It looked impossible for me to look up all the necessary jar files for running a test case with AWS SDK in Katalon Studio.

Solution

Use com.katalon.gradle-plugin.

Sample build.gradle is here

$ gradle katalonCopyDependencies

the plugin lets Gradle download the specified jar file and other dependencies from the Maven Central repository, and it copies them into the <projectDir>/Drivers directory. For example, I got:

$ ls Drivers
katalon_generated_aws-java-sdk-core-1.11.470.jar
katalon_generated_aws-java-sdk-kms-1.11.470.jar
katalon_generated_aws-java-sdk-s3-1.11.470.jar
katalon_generated_commons-codec-1.10.jar
katalon_generated_commons-logging-1.2.jar
katalon_generated_httpclient-4.5.5.jar
katalon_generated_httpcore-4.4.9.jar
katalon_generated_ion-java-1.0.2.jar
katalon_generated_jackson-annotations-2.6.0.jar
katalon_generated_jackson-core-2.6.7.jar
katalon_generated_jackson-databind-2.6.7.2.jar
katalon_generated_jackson-dataformat-cbor-2.6.7.jar
katalon_generated_jmespath-java-1.11.470.jar
katalon_generated_joda-time-2.8.1.jar

Description

Prerequisite

Here I assume you have some experience of AWS. You should already have an AWS account for you; you already have prepared AWS credential info in the <HOME>/.aws directory of your PC; you should have installed AWS CLI. I assume you can successfully do the following operation in the command line:

$ aws s3 ls
2017-12-27 06:30:04 myBucket1
2016-03-31 23:11:02 myBucket2
2015-03-01 13:01:59 myBucket3
...

I am going to do the same operation (list my S3 Buckets) by a groovy script as test case in Katalon Studio.

Resolving dependencies

Here I assume you have got started with Gradle. Installation guide is here.

I added <projectDir>/build.gradle file. In there I declared only one jar as dependency of the project.

dependencies {
    // https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk-s3
    compile group: 'com.amazonaws', name: 'aws-java-sdk-s3', version: '1.11.470'
}

You can find the jar in the Maven Central repository https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk-s3

I the command line, I executed the following command:

$ cd <katalon project path>
$ gradle katalonCopyDependencies

Gradle took a few minutes to finish the katalonCopyDependencies task. It downloaded 14 jar files into the local <HOME>/.m2 diretory — this took a few minutes. And the task copied necessary jar files into <projectDir>/Drivers .

I closed the project once, and reopened it in Katalon Studio. This was necessary in order to let Katalon Studio acknowledge the new jar files in the Drivers directory.

Now I run it

I wrote a test case ListMyS3Buckets.

I ran the test case, and got the following output:

My Amazon S3 buckets are:
* myBucket1
* myBucket2
* myBucket3
...

It worked.

Note on .gitignore

If you are using git for the Katalon project, you should exclude the external jar files out of the repository by adding the following line in the .gitignore file.

/Drivers
8 Likes

It is really helpful. Thank you so much.

Completely awesome and excellently put together, as usual, @kazurayam. Demonstrates Katalon’s integration capabilities across multiple paradigms: Katalon, Java, Gradle, and AWS.

(With such integration, it makes me wonder why it’s considered a Tip – it certainly isn’t a trick!)

1 Like

I often imagine: in future a project of Katalon Studio is reorganized as a Gradle project within Eclipse.

  • It’s directory structure will be Gradle-standard format.
  • You can utilize the external dependency management features of Gradle in full scale.
  • You can do unit-testing for your custom classes (namely, Keywords) using any testing framework you like (JUnit, Spock, Cucumber, etc)
  • You can generate groovy-docs of your custom classes (and test cases if possible) from the groovy-doc comments in your codes.
  • You can use any JVM-based programming languages together: Java, Groovy, kotlin, jython, etc.
  • All in Eclipse GUI!
  • Will have 2 eclipse Perspectives and you can easily switch between: the original Katalon Studio perspective, and good old Eclipse IDE perspective.
4 Likes

Using com.katalon.gradle-plugin is potentially dangerous.

Let me tell you what. I can write in build.gradle

dependencies {
    compile group: 'org.slf4j', name: 'slf4j-api', version: "1.7.4"
...

and execute the plugin. I will have slf4j version1.7.4 in the Drivers directory.

On the other hads Katalon Studio itself bundles slf4j too, but the version is different: 1.6.

Therefore it is likely that I see the following error in runtime

SLF4J: The requested version 1.7.16 by your slf4j binding is not compatible with [1.6]
SLF4J: See http://www.slf4j.org/codes.html#version_mismatch for further details.

Fundamentally, we should not have 2 isolated library paths naively mixed: Drivers directory and Katalon’s plugin dir.

We should have only 1 integrated library path. This can be achived by relying on Gradle’s dependency management totally.

2 Likes

Yes, I agree. However, dependency hell is an issue with almost every language. One way to resolve it is that the user has to repackage their libraries (i.e. “to build shaded JARs”) before adding them to Katalon Studio projects.

2 Likes

Thanks for this post.
I configure it locally and it is download all jars I need to my project.
However, I use bitbucket pipelines, and the jars aren’ being download.
Can you please help me to configure the yml, maybe my script conf is wrong.
I really appreciate your time

Are you using Katalon Docker image in your pipeline?

1 Like

Thanks for your quick answer, yes, seems that at the moment that katalon execute command is happening the dependencies are not there, I might be missing a step or the execution order is wrong. Do you have any working sample of your pipeline configuration?

yml file:

image: katalonstudio/katalon

pipelines:
default:
- step:
services:
- docker
caches:
- gradle
script:
# Run Test
- chmod +x gradlew
- bash ./gradlew katalonCopyDependencies
- bash ./gradlew build
- echo app.env = $ENV
- katalon-execute.sh -browserType=“Chrome” -retry=0 -statusDelay=15 -testSuitePath=“Test Suites/integration-test-suite”
# defining the artifacts .
artifacts:
- report/**

Thank you. We’ll test the config and get back to you soon.

Currently, both of steps I believe should be integrated in the katalon executed.sh. I mean, I should not do the following: - chmod +x gradlew

  • bash ./gradlew katalonCopyDependencies
  • bash ./gradlew build

As Katalon is creating a tmp file in which all my project is being copied, them Driver folders is empty, as I never it is not running the cmd: gradle katalonCopyDependencies

I really appreciate your help.

In the worst scenario, I need to copy the jars to bitbucket, but I shouldn’t do that.

Thanks again :slight_smile:

Hi @devalex88,
I’ve started in a new branch everything, to check if that was part of my configuration:
I added to .gitIgnore the /Drivers.
When the pipelines started to build throws the following exception:

java.lang.NoClassDefFoundError: AmazonS3
at com.kms.katalon.core.main.CustomKeywordDelegatingMetaClass.getCustomKeywordClassAndSetMetaClass(CustomKeywordDelegatingMetaClass.java:100)

I really appreciate your help and time.

Just a question: Also I added to .gitIgnore the folder .gradle
Does it right?

Thanks!

Hi @devalex88,

Sorry for bothering you again,
Have you done the configuration on your yml to run this plugin on CI? We need to download dependencies to connect with AWS?
Many thanks!

I have updated the sample project so that it works with the latest versions of Gradle, katalon-gradle-plugin, AWS SDK.

1 Like