Test Suite Collection switching between Appium mobile driver and selenium web driver

bmeyer · April 9, 2020, 8:49pm

Our team is using saucelabs to run mobile test scripts against a variety of real remote mobile devices. As part of this testing we require switching between the appium mobile driver and the selenium web driver - there exist certain actions we perform in the web driver, and then switch back to the mobile driver to verify outcome of said web driver action.

That being said, we cannot make this sort of functionality work (nor can we find any documentation regarding this). Here’s our configuration:

Android desired capabilities set up through Project > Settings > Desired Capabilities > Custom
- We know this functionality works fine - our mobile scripts are running on real devices no problem
Test suite collection set up with following configuration:
- “Run with” set to the various custom mobile capabilities (i.e. android tablet running version 9)
- “Profile” set to our desired execution profile
The test suite run in the test suite collection consists of 3 different test cases; first 2 test cases running on mobile device, and the last in chrome browser. Here’s what the setup block of the test suite:

def setUp() {
    	System.setProperty('webdriver.chrome.driver', DriverFactory.getChromeDriverPath())
    	GlobalVariable.MCDriver = new ChromeDriver()
    	DriverFactory.changeWebDriver(GlobalVariable.MCDriver)
}

The test suite collection fails out immediately at the first test case; as I mentioned, the first test case is a mobile test case, and starts with:
Mobile.startApplication(GlobalVariable.androidApplication, false)
It bears repeating that the mobile test cases run without issue until I try to introduce a web driver

Any help on this would be much appreciated

kazurayam · April 9, 2020, 9:43pm

Could you please share the error log?

Can you run the first test case successfully (outside the test suite collection)?

kazurayam · April 9, 2020, 9:45pm

I think that you are the 1st one who initiated a discussion how to run Appium and WebDriver together in a test project. You are great pioneer!

bmeyer · April 10, 2020, 3:14pm

@kazurayam thanks for the quick reply! Either I’m a pioneer or I’m trying to do something that shouldn’t be done

I’ll break this down into 2 possible scenarios (and the outcomes) of how I could approach testing with Appium Driver and then WebDriver.

Scenario 1: Creating WebDriver in test suite setup
This is the scenario I outlined in my original comment. The test suite starts and stops in ~1 second. Unfortunately when I try to run this test suite I don’t get much information in the console tab in Katalon:

2020-04-10 07:56:03.711 INFO  c.k.katalon.core.main.TestSuiteExecutor  - START Test Suites/AND_Android/Suites/AND_RegMedDriverSwap
2020-04-10 07:56:04.234 INFO  c.k.katalon.core.main.TestSuiteExecutor  - --------------------
2020-04-10 07:56:04.234 INFO  c.k.katalon.core.main.TestSuiteExecutor  - END Test Suites/AND_Android/Suites/AND_RegMedDriverSwap
2020-04-10 07:56:04.234 INFO  c.k.katalon.core.main.TestSuiteExecutor  - ====================

To note, the first test case in the test suite does pass without issue when run outside of the test suite (seems the setting of the WebDriver in test suite setup is causing an issue)

Scenario 2: No WebDriver creation in the test suite setup, instead create WebDriver at the start of the WebUI Test Case

I start the final test case in the test suite with:

System.setProperty('webdriver.chrome.driver', DriverFactory.getChromeDriverPath())
WebDriver MCDriver = new ChromeDriver()
WebUI.navigateToUrl(GlobalVariable.mcUrl)

Once the test cases using Appium Driver are completed, the last test case in the test suite (test case using the WebDriver) fails with this error:

2020-04-10 08:12:01.883 ERROR c.k.katalon.core.main.TestCaseExecutor   - ❌ Test Cases/MC_ManagementConsole/MC_UnregisterDriverSwap FAILED.
Reason:
java.lang.NullPointerException
	at java_lang_System$setProperty$0.call(Unknown Source)
	at MC_UnregisterDriverSwap.run(MC_UnregisterDriverSwap:11)
	at com.kms.katalon.core.main.ScriptEngine.run(ScriptEngine.java:194)
	at com.kms.katalon.core.main.ScriptEngine.runScriptAsRawText(ScriptEngine.java:119)
	at com.kms.katalon.core.main.TestCaseExecutor.runScript(TestCaseExecutor.java:337)
	at com.kms.katalon.core.main.TestCaseExecutor.doExecute(TestCaseExecutor.java:328)
	at com.kms.katalon.core.main.TestCaseExecutor.processExecutionPhase(TestCaseExecutor.java:307)
	at com.kms.katalon.core.main.TestCaseExecutor.accessMainPhase(TestCaseExecutor.java:299)
	at com.kms.katalon.core.main.TestCaseExecutor.execute(TestCaseExecutor.java:233)
	at com.kms.katalon.core.main.TestSuiteExecutor.accessTestCaseMainPhase(TestSuiteExecutor.java:169)
	at com.kms.katalon.core.main.TestSuiteExecutor.accessTestSuiteMainPhase(TestSuiteExecutor.java:142)
	at com.kms.katalon.core.main.TestSuiteExecutor.execute(TestSuiteExecutor.java:91)
	at com.kms.katalon.core.main.TestCaseMain.startTestSuite(TestCaseMain.java:157)
	at com.kms.katalon.core.main.TestCaseMain$startTestSuite$0.call(Unknown Source)
	at TempTestSuite1586531357579.run(TempTestSuite1586531357579.groovy:36)

To note, that same test case does pass when run on its own outside of the test suite. Please let me know if more information is necessary

kazurayam · April 10, 2020, 8:28pm

Which version of Katalon Studio do you use? Possibly 7.3.0 or 7.3.1?

As Project Settings > Executions > Enable/Disable logging executed steps; is it deprecated? describes, Katalon Studio ver 7.3.0 ~ 7.3.1 has the “Log executed test steps” option in “Project > Settings > Execution” default “Disabled”. Therefore you don’t get much information in the console tab. @duyluong wrote that they will change it to be “Enabled” in the next ver 7.3.2. If you bring your KS down to ver 7.2.x and try the same test suites, you will probably see some lines more of logs. However I suppose that these additional lines of message will not be very informative. The lines will just show which test steps were executed.

kazurayam · April 10, 2020, 8:37pm

In the Scenario 2, you open Chrome browser:

When do you close Chrome browser?
Do you close Chrome before you start working on Appium Driver?
Or do you need to keep Chrome up and running while you work on Appium Driver?

The stack trace :

Reason: java.lang.NullPointerException
    at java_lang_System$setProperty$0.call(Unknown Source)
    at MC_UnregisterDriverSwap.run(MC_UnregisterDriverSwap:11)
    at com.kms.katalon.core.main.ScriptEngine.run(ScriptEngine.java:194)
    ...

tells me there occured something unexpected for ScriptEngine class.

What is unexpected?

I guess, the ScriptEngine class does not expect to communicate with both of WebDriver and Appium at the same time.

I guess, the ScriptEngine class is not designed defensive enough. At least it should not throw NPE but throw some other type of RuntimeException with more informative message saying “Please do not open WebDriver and Appium at the same time. I can not manage it!”

I guess, you should close Chrome browser as soon as you finished business with it before you start working on Appium Driver.

bmeyer · April 14, 2020, 3:25pm

@kazurayam swinging back around to this thread after my update to 7.3.1 - happy to report my test steps are back in the console

I’ve implemented one of your suggestions, and I’m unfortunately still met with the NPE. My first test case runs on an Android tablet. I grab the mobile driver with:

AndroidDriver testDriver = MobileDriverFactory.getDriver()

Then I proceed to run the rest of the mobile steps, before closing the application and quitting the driver:

Mobile.closeApplication()
testDriver.quit()

All is well up until this point. Finally, my test suite runs the test case that runs with web driver, which starts off with:

System.setProperty('webdriver.chrome.driver', DriverFactory.getChromeDriverPath())

WebDriver MCDriver = new ChromeDriver()

And immediately we get the NPE:

2020-04-14 08:14:53.868 DEBUG testcase.MC_UnregisterDriverSwap         - 1: System.setProperty("webdriver.chrome.driver", getChromeDriverPath())
2020-04-14 08:14:53.887 ERROR c.k.katalon.core.main.TestCaseExecutor   - ❌ Test Cases/MC_ManagementConsole/MC_UnregisterDriverSwap FAILED.
Reason:
java.lang.NullPointerException
	at java_lang_System$setProperty$0.call(Unknown Source)
	at MC_UnregisterDriverSwap.run(MC_UnregisterDriverSwap:11)
	at com.kms.katalon.core.main.ScriptEngine.run(ScriptEngine.java:194)
	at com.kms.katalon.core.main.ScriptEngine.runScriptAsRawText(ScriptEngine.java:119)
	at com.kms.katalon.core.main.TestCaseExecutor.runScript(TestCaseExecutor.java:337)
	at com.kms.katalon.core.main.TestCaseExecutor.doExecute(TestCaseExecutor.java:328)
	at com.kms.katalon.core.main.TestCaseExecutor.processExecutionPhase(TestCaseExecutor.java:307)
	at com.kms.katalon.core.main.TestCaseExecutor.accessMainPhase(TestCaseExecutor.java:299)
	at com.kms.katalon.core.main.TestCaseExecutor.execute(TestCaseExecutor.java:233)
	at com.kms.katalon.core.main.TestSuiteExecutor.accessTestCaseMainPhase(TestSuiteExecutor.java:169)
	at com.kms.katalon.core.main.TestSuiteExecutor.accessTestSuiteMainPhase(TestSuiteExecutor.java:142)
	at com.kms.katalon.core.main.TestSuiteExecutor.execute(TestSuiteExecutor.java:91)
	at com.kms.katalon.core.main.TestCaseMain.startTestSuite(TestCaseMain.java:157)
	at com.kms.katalon.core.main.TestCaseMain$startTestSuite$0.call(Unknown Source)
	at TempTestSuite1586877171783.run(TempTestSuite1586877171783.groovy:36)

kazurayam · April 14, 2020, 9:23pm

Please try asserting that DriverFactory.getChromeDriverPath() returns a non-null value.

I suppose that DriverFactory.getChromeDriverPath() returns null. That is the cause of NPE you got.

kazurayam · April 14, 2020, 9:35pm

In MC_UnregisterDriverSwap, why DriverFactory.getChromeDriverPath() returns null?

I know it. I will tell you. See the following dropdown list.

As you know, Katalon Studio’s GUI has this dropdown list. When you are to invoke your test case, you will operate this to choose one of the types of driver you want for your test to run with.

Possibly you chose “Android” driver when you ran your tests. In other words, “Chrome” driver was not chosen.

As “Chrome” driver was NOT chosen, DriverFactory.getChromeDriverPath() returned null. Katalon Studio is designed as such. It seems to be designed with a naive assumption that

User is required to choose, in this dropdown list, a single type of driver to run their test. Once chosen, a single driver should be sufficient for a test run. The test is supposed NOT to swap drivers during a run.

@bmeyer

In your test, you are swapping drivers, which is unexpected for Katalon Team .

You can study how DriverFactory behaves according to which type of WebDriver you chose to invoke your test. Please add the following TestCase code and run it with Chrome; and later with Firefox.

import com.kms.katalon.core.webui.driver.DriverFactory;

println "executed driver is: " + DriverFactory.getExecutedBrowser().toString()
println "chrome driver path is : " + DriverFactory.getChromeDriverPath()
println "firefox Gecko driver path is : " + DriverFactory.getGeckoDriverPath()

You will see the following output:

2020-04-15 15:23:11.226 INFO  c.k.katalon.core.main.TestCaseExecutor   - START Test Cases/TC1
2020-04-15 15:23:11.913 DEBUG testcase.TC1                             - 1: println("executed driver is: " + getExecutedBrowser().toString())
executed driver is: Firefox
2020-04-15 15:23:12.019 DEBUG testcase.TC1                             - 2: println("chrome driver path is : " + getChromeDriverPath())
chrome driver path is : null
2020-04-15 15:23:12.033 DEBUG testcase.TC1                             - 3: println("firefox Gecko driver path is : " + getGeckoDriverPath())
firefox Gecko driver path is : /Applications/Katalon Studio.app/Contents/Eclipse/configuration/resources/drivers/firefox_mac/geckodriver
2020-04-15 15:23:12.084 INFO  c.k.katalon.core.main.TestCaseExecutor   - END Test Cases/TC1

How to work around NPE?

You just want to refrain from calling DriverFactory.getChromeDriverPath(). You can write the path as a constant string as follows.

On my macOS:

System.setProperty("webdriver.chrome.driver", "/Applications/Katalon Studio.app/Contents/Eclipse/configuration/resources/drivers/chromedriver_mac/chromedriver")
WebDriver MCDriver = new ChromeDriver()

This code looks ugly, but it will hopefully work without throwing a NPE.

@bmeyer

If you choose “Chrome” here, it may run successful. I am not sure.