How to write Katalon Studio tests with IntelliJ IDEA and other IDEs

kazurayam · December 26, 2018, 6:40am

I’ve got a question.

In an arbitrary Katalon Studio project created by version 5.10.0, in the .classpath file, I found the following lines:

<?xml version="1.0" encoding="UTF-8"?>
<classpath>
    ...
    <classpathentry kind="src" output="bin/keyword"  path="Keywords"/>
    <classpathentry kind="src" output="bin/listener" path="Test Listeners"/>
    <classpathentry kind="src" output="bin/lib"      path="Libs"/>
    <classpathentry kind="src" output="bin/groovy"   path="Include/scripts/groovy"/>
    ...

The last line which has path="Include/scripts/groovy" looks very interesting for me. This line implies that a user (like me) can locate source files of arbitrary custom Groovy classes in the Include/scripts/groovy folder.

As far as I know up until now, the Keywords folder was the only source folder where I could locate my *.groovy files. Now I have another source folder Include/scripts/groovy as well. — Am I right? For what purpose is this folder intended?

This is a significant design change.
@Russ_Thomas, are you aware of it?

devalex88 · December 26, 2018, 7:49am

That folder was introduced for defining BDD steps.

https://docs.katalon.com/katalon-studio/docs/cucumber-features-file.html

It’s also leveraged for the open-source component:

By the way, the first place that Java’s class loader looks for classes is in the current Katalon Studio project. Therefore, there is a way to “patch” the execution engine before the desired changes make it into official builds. If you put a modified version with the same class name and package into Include/scripts/groovy, the other version in JAR packages will be ignored by the class loader.

For example, create a new Test Case that throw Katalon Studio’s StepFailedException:

import com.kms.katalon.core.exception.StepFailedException

throw new StepFailedException("");

Under Include/scripts/groovy/com/kms/katalon/core/exception, add a new file named StepFailedException.java:

package com.kms.katalon.core.exception;

import com.kms.katalon.core.util.internal.ExceptionsUtil;

/**
 * Exception to stop execution and mark keyword or test case as FAILED
 *
 */
public class StepFailedException extends RuntimeException {
    private static final long serialVersionUID = 1L;

    public StepFailedException(String message) {
        super(message);
        System.out.println("Patched!");
    }

    public StepFailedException(Throwable t) {
        super(ExceptionsUtil.getMessageForThrowable(t), t);
    }
}

The log would be:

2018-12-26 14:40:53.208 DEBUG testcase.New Test Case - throw new com.kms.katalon.core.exception.StepFailedException()
Patched!

Delete this class, the Patched! line will be gone.

Please remember to refresh the project to make sure Katalon Studio synchronizes changes in the file system.

This trick is actually helpful sometimes when combined with the above open-source repository.

kazurayam · December 26, 2018, 8:04am

OK, I will try patching com.kms.katalon.core.webui.keyword.internal.WebUIKeywordMain so that it prints stacktrace of an exception thrown by Closure, as mentioned in the Cannot find elements when XPath expression is null

devalex88 · December 26, 2018, 8:09am

You can also patch the class com.kms.katalon.core.util.internal.ExceptionsUtil. Replace the part

    private static String getExceptionMessage(Throwable throwable) {
        if (throwable instanceof MissingPropertyException) {
            return getExceptionMessage((MissingPropertyException) throwable);
        } else {
            return throwable.getClass().getName()
                    + (throwable.getMessage() != null ? (": " + throwable.getMessage()) : "");
        }
    }

with the following one

private static String getExceptionMessage(Throwable throwable) {
    if (throwable instanceof MissingPropertyException) {
        return getExceptionMessage((MissingPropertyException) throwable);
    } else {
        return getStackTraceForThrowable(throwable);
    }
}

Russ_Thomas · December 26, 2018, 1:34pm

This is a significant design change.
@Russ_Thomas, are you aware of it?

@kazurayam No, and thanks for the detective work, Kaz. I was aware of the UI changes for BDD includes but didn’t know about the detail exposed by you and @devalex88. Thanks both of you.

More places to hook into… that should keep me busy a while

kazurayam · December 27, 2018, 2:33am

I tried patching the Katalon Studio engine by writing Groovy classes in the Include/scripts/groovy folder as devalex88 suggested. I worked on my GitHub project Cannot_find_elements_when_XPath_expression_is_null. You can checkout the tag 0.4 to my last code.

patching the class `com.kms.katalon.core.util.internal.ExceptionsUtil`

devalex88 suggested this patch.

The patch resulted the following stacktrace: log_withPatchToExceptionsUtil.txt

com.kms.katalon.core.exception.StepFailedException: Unable to verify object 'Object Repository/Page_AngularJS Hub  Text Inputs/iframe_Example_exampleIFrame' is present (Root cause: java.lang.IllegalArgumentException: Cannot find elements when the XPath expression is null.)
	at com.kms.katalon.core.keyword.internal.KeywordMain.stepFailed(KeywordMain.groovy:36)
	at com.kms.katalon.core.webui.keyword.internal.WebUIKeywordMain.stepFailed(WebUIKeywordMain.groovy:65)
	at com.kms.katalon.core.webui.keyword.internal.WebUIKeywordMain.runKeyword(WebUIKeywordMain.groovy:27)
	at com.kms.katalon.core.webui.keyword.builtin.VerifyElementPresentKeyword.verifyElementPresent(VerifyElementPresentKeyword.groovy:92)
	at com.kms.katalon.core.webui.keyword.builtin.VerifyElementPresentKeyword.execute(VerifyElementPresentKeyword.groovy:68)
	at com.kms.katalon.core.keyword.internal.KeywordExecutor.executeKeywordForPlatform(KeywordExecutor.groovy:53)
...

This stacktrace is NOT interesting. I want to know which line of code in which method of which class raied the IllegalArgumentException. But this statcktrace does not tell me.

patching the class `com.kms.katalon.core.webui.keyword.internal.WebUIKeywordMain` class.

I suggested this at Cannot find elements when XPath expression is null.

This patch resulted the following statcktrace: log_withPatchToWebUIKeywordMain.txt

...
	at com.kms.katalon.core.webui.common.WebUiCommonHelper.buildLocator(WebUiCommonHelper.java:497)
...
	at com.kms.katalon.core.webui.keyword.builtin.VerifyElementPresentKeyword$_verifyElementPresent_closure1.doCall(VerifyElementPresentKeyword.groovy:79)

These lines are informative. I could find that the Exception was raised at line497 in WebUiCommonHelper

           case XPATH:
                return By.xpath(to.getSelectorCollection().get(selectorMethod));

I can conclude that the expression to.getSelectedCollection().get(selectorMethod) returned null. This is the root cause of the problem!

My conclusion

Patching ExceptionsUtil class is not useful, but patching WebUIKeywordMain class is useful.

kazurayam · December 27, 2018, 2:51am

The original com.kms.katalon.core.util.internal.ExceptionsUtil is coded in Java. I wanted to patch it so I added a Groovy code named ExceptionsUtil.groovy. Java and Groovy is almost similar but different in detail. The following page shows how I modified the Java code into Groovy.

I needed to translate a Java’s lambda expression into a Groovy’s closure.
The way to cast a List<StackTraceElement> to an array StackTraceElement ; Groovy requires ‘as’ notation rather than .toArray()

kazurayam · December 27, 2018, 2:53am

@devalex88

Why not we have Include/scripts/java?

devalex88 · December 27, 2018, 3:09am

Actually, .java files will also be compiled - for patching purpose you can use whatever you prefer. We try to use the Groovy name as much as possible in order to avoid confusing non-developer users and to simplify the support process.

By the way (again ), Groovy also has some advanced methods that I think would also help this purpose.

http://docs.groovy-lang.org/latest/html/documentation/core-metaprogramming.html#_magic_package

kazurayam · December 27, 2018, 4:15am

OK, I understand it.

kazurayam · December 28, 2018, 1:56am

@devalex88

I tried writing JUnit tests for my custom Keywords in Katalon Studio and running the tests with Eclipse. I got a partial success, but encountered a difficulty. Let me report my research and ask for your support.

Successful case

I created a keyword Keywords/junittutorial/Calculator. Please note that this keyword class is a Plain-Old-Java-Object which has no dependency upon the Katalon Studio’s runtime, e.g, the com.kms.katalon.core.configuration.RunConfiguration object)

And I created a JUnit test for it Include/scripts/groovy/junittutorial/CalculatorTest.groovy

I started Eclipse photon, opened the project, and ran the JUnit test. The test ran successfully as expected.

Difficult case

I created another keyword Keywords/my/keywor/MyWebUI. This keyword class tries to instantiate a ChromeDriver. This class has dependency upon Katalon Studio’s runtime.

I created another JUnit test Include/scripts/groovy/my/keyword/MyWebUITest.groovy. This test simply invokes my.keyword.MyWebUI.openBrowser().

When I ran this test in Eclipse, got an Exception with the following statcktrace:

java.lang.NullPointerException
	at com.kms.katalon.core.configuration.RunConfiguration.getDriverExecutionProperties(RunConfiguration.java:226)
	at com.kms.katalon.core.configuration.RunConfiguration.getDriverSystemProperties(RunConfiguration.java:252)
	at com.kms.katalon.core.webui.driver.DriverFactory.isUsingExistingDriver(DriverFactory.java:198)
	at com.kms.katalon.core.webui.driver.DriverFactory.getExecutedBrowser(DriverFactory.java:853)
	at com.kms.katalon.core.webui.driver.DriverFactory$getExecutedBrowser.call(Unknown Source)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:120)
	at my.keyword.MyWebUI.openBrowser(MyWebUI.groovy:22)
	at my.keyword.MyWebUI$openBrowser.call(Unknown Source)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:120)
	at my.keyword.MyWebUITest.testOpenBrowser(MyWebUITest.groovy:13)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86)
	at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:538)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:760)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:460)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:206)

Why NullPointerException? It is easily seen. The keyword requires Katalon Studio runtime environment. But my JUnit test does not construct the runtime environment before invoking the keyword. Therefore the keyword can not run anyway.

My thought

Previously I ran my JUnit tests as a TestCase in Katalon Studio, for example see https://github.com/kazurayam/RunningJUnitInKatalonStudio/blob/0.2/Scripts/test/junittutorial.test/CalculatorTestRunner/Script1532666027182.groovy In this way, I could mix codes calling org.junit.* classes and com.kms.katalon.** classes with no problem.

Question

I want to write a custom keyword which has dependency on the Katalon Studio RunConfiguration. I want to do unit-testing for those keywords with JUnit. I want to run the JUnit tests with Eclipse. Then, how can I setup the Katalon runtime environment for each JUnit tests in Eclipse (not in Katalon Studio)?

devalex88 · December 28, 2018, 2:08am

Do you want to actually interact with browsers in your JUnit tests? My initial thought is that unit tests should be fast, reliable, and isolated so dependencies should be provided as mocks or stubs.

kazurayam · December 28, 2018, 2:14am

No. Many of the custom keywords I have written so far have no dependency on Katalon Studio runitme environment. I am happy testing them with Eclipse now.

But a few of my custom keywords, as MyWebUI above, is closely related to the Katalon runtime. How to do unit testings for those? My first intension is not to interact with browser, but it is inevitable. Mock or stub — fine, but I do not see how to create a mock of the RunConfiguration.

kazurayam · December 28, 2018, 2:20am

Can you provide an junit tests which creates a mock of RunConfiguration?

devalex88 · December 28, 2018, 2:24am

I think Mockito can help here.

https://site.mockito.org

Could you please share your tests? I’m on vacation but I’ll take a look and have a sample based on them next week.

kazurayam · December 28, 2018, 2:29am

Ah, Mockito. I know that of course. However, I could not remeber the name in combination with Katalon Studio ever.

kazurayam · December 28, 2018, 9:59pm

I want to perform JUnit test for a custom keyword which calls the RunConfiguration.getProjectDir(). Of course this method is expected to return the path of a Katalon Studio’s project directory. I made a test and ran it in Eclipse, then I got a failure because RunConfiguration.getProjectDir() returned null. I understood that RunConfiguration.getProjectDir() method requires the runtime environment of a Katalon Studio project.

Taking advice from @devalex88, I thought I should isolate my JUnit test from Katalon Studio runtime environment, wanted to find out a way to implement it. So I tried ExpandoMetaClass of Groovy language, and got a success. Let me tell you what I did.

I made a Groovy file as a JUnit test Include/scripts/groovy/my/keyword/GroovyExpandoMetaClassTest.groovy:

package my.keyword

import static org.hamcrest.CoreMatchers.*
import static org.junit.Assert.*

import org.junit.Before
import org.junit.Test

import com.kms.katalon.core.configuration.RunConfiguration

/**
 * This test demonstrates how to use so called "ExpandoMetaClass" of Groovy language.
 * See http://groovy-lang.org/metaprogramming.html#metaprogramming_emc for detail.
 * 
 * This JUnit test shows that you can dynamically override the static 
 *     getProjectDir() 
 * method of 
 *     com.kms.katalon.core.configuration.RunConfiguration
 * class. 
 * 
 * Overriding the Katalon Studio runtime engine by ExpandoMetaClass helps 
 * simplifying JUnit testcases for your custom keywords. This technique enables you 
 * to "mock" the runtime engine with just a few lines of Groovy code.
 * 
 * @author kazurayam
 *
 */
class GroovyExpandoMetaClassTest {
    private static final String PATH_X = "/Users/kazurayam/katalon-workspace/projectX"
 
    @Before
    void setup() {
        RunConfiguration.metaClass.static.getProjectDir = { -> return PATH_X }
    }

    @Test
    void testExpandoMetaClass() {
        String s = RunConfiguration.getProjectDir()
        println "[testExpandoMetaClass] " + s
        assertThat(s, is(PATH_X))
    }
}

I ran this test with Eclipse, and it passed!

With just a few line of Groovy code, I could override the static method (getProjectDir) of Katalon Studio’s core engine class (com.kms.katalon.core.configuration.RunConfiguration) runtime. This case demonstrates the power of Groovy’s metaprogramming. I feel quite comfortable with it.

Now I’ve got to know how to write JUnit tests for my custom Katalon Keywords, how to run them and see the results in Eclipse. I like this approach very much.

The source code is here:

GitHub - kazurayam/junit4ks at develop

devalex88 · December 29, 2018, 2:57am

This is awesome @kazurayam. It deserves an article of its own .

YoungNgo · December 29, 2018, 9:23am

Russ_Thomas · December 29, 2018, 12:30pm

I’m convinced @kazurayam is writing a book - Katalon: The Missing Manual