Web Tests failing inconsistently on mobile devices for no obvious reason

I have a number of web tests that are passing almost 100% in Chrome (desktop), but I’m seeing inconsistent failures when executing on real Android devices. These tests are spread over 5 test suites, all kicked off from one Test Suite collection executed using Katalon Runtime (kicked off via Azure).

The error - for each failed test case - seems to relate to step 1 of each failed test case, which is a custom keyword (openBrowserSetLanguage), which basically opens the browser with a URL (defined by a variable - I’m using data-driven testing through test suites data-binding), and then selects the relevant language in the web page through a series of clicks, wait-for statements, and delays).

A typical error message, which all point to this keyword is shown below (I’ve hidden the URLs purposefully, but it’s the same URL for every test case).

I’m seeing failures when running it on a Nokia 5 (Android 9) and Sony Z5 Compact (Android 7), but running it once on my own Google Pixel 2XL resulted in no failures.

Coincidentally or not, where there are failures, they’ve so far all been from the same test suite. I have 5 test suites, and in my most recent execution, 22 out of 30 tests failed all within test suite ‘A’. However, on the previous run, all test cases in test suite ‘A’ passed, but 12 out of 22 test cases in test suite ‘B’ failed.

It’s almost like once a Test Suite has a failed test case, further test cases fail. This makes me think that perhaps the device / operating system has had an issue that it has somehow recovered from (after an amount of time), but this is just a theory.

Any suggestions as to what could be causing this, and how to rectify these issues would be appreciated.

Should I expect to be able to run these tests - trouble free - on older devices? Does the java.io.IOException error below have any significance? Are there any preferred strategies for running tests in volume on mobile devices (e.g. could there be memory/leak issues etc., and is there anything I can do to negate this type of thin?)?

I’m using Katalon Runtime Engine 64-7.2.1. Chrome version on all mobile devices is 80.0.3987.99, and I have Chromedriver.exe v80.

Unable to open browser with url: ‘https://[URL]’ (Root cause: com.kms.katalon.core.exception.StepFailedException: Unable to open browser with url: ‘https://[URL]’ at com.kms.katalon.core.webui.keyword.internal.WebUIKeywordMain.stepFailed(WebUIKeywordMain.groovy:64) at com.kms.katalon.core.webui.keyword.internal.WebUIKeywordMain.runKeyword(WebUIKeywordMain.groovy:26) at com.kms.katalon.core.webui.keyword.builtin.OpenBrowserKeyword.openBrowser(OpenBrowserKeyword.groovy:81) at com.kms.katalon.core.webui.keyword.builtin.OpenBrowserKeyword.execute(OpenBrowserKeyword.groovy:67) at com.kms.katalon.core.keyword.internal.KeywordExecutor.executeKeywordForPlatform(KeywordExecutor.groovy:72) at com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords.openBrowser(WebUiBuiltInKeywords.groovy:60) at com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords$openBrowser.call(Unknown Source) at Apples.openBrowserSetLanguage(Apples.groovy:28) at Apples.invokeMethod(Apples.groovy) at com.kms.katalon.core.main.CustomKeywordDelegatingMetaClass.invokeStaticMethod(CustomKeywordDelegatingMetaClass.java:50) at Verify category action button.run(Verify category action button:19) at com.kms.katalon.core.main.ScriptEngine.run(ScriptEngine.java:194) at com.kms.katalon.core.main.ScriptEngine.runScriptAsRawText(ScriptEngine.java:119) at com.kms.katalon.core.main.TestCaseExecutor.runScript(TestCaseExecutor.java:337) at com.kms.katalon.core.main.TestCaseExecutor.doExecute(TestCaseExecutor.java:328) at com.kms.katalon.core.main.TestCaseExecutor.processExecutionPhase(TestCaseExecutor.java:307) at com.kms.katalon.core.main.TestCaseExecutor.accessMainPhase(TestCaseExecutor.java:299) at com.kms.katalon.core.main.TestCaseExecutor.execute(TestCaseExecutor.java:233) at com.kms.katalon.core.main.TestSuiteExecutor.accessTestCaseMainPhase(TestSuiteExecutor.java:169) at com.kms.katalon.core.main.TestSuiteExecutor.accessTestSuiteMainPhase(TestSuiteExecutor.java:142) at com.kms.katalon.core.main.TestSuiteExecutor.execute(TestSuiteExecutor.java:91) at com.kms.katalon.core.main.TestCaseMain.startTestSuite(TestCaseMain.java:157) at com.kms.katalon.core.main.TestCaseMain$startTestSuite$0.call(Unknown Source) at TempTestSuite1582284081015.run(TempTestSuite1582284081015.groovy:36) Caused by: java.io.IOException: GetExitCodeProcess error=6, The handle is invalid at com.kms.katalon.core.appium.driver.AppiumDriverManager.isAppiumServerStarted(AppiumDriverManager.java:193) at com.kms.katalon.core.appium.driver.AppiumDriverManager.startAppiumServerJS(AppiumDriverManager.java:257) at com.kms.katalon.core.appium.driver.AppiumDriverManager.startAppiumServerJS(AppiumDriverManager.java:406) at com.kms.katalon.core.appium.driver.AppiumDriverManager.ensureServicesStarted(AppiumDriverManager.java:243) at com.kms.katalon.core.appium.driver.AppiumDriverManager.createMobileDriver(AppiumDriverManager.java:472) at com.kms.katalon.core.webui.driver.WebMobileDriverFactory.createMobileDriver(WebMobileDriverFactory.java:139) at com.kms.katalon.core.webui.driver.DriverFactory.startNewBrowser(DriverFactory.java:325) at com.kms.katalon.core.webui.driver.DriverFactory.openWebDriver(DriverFactory.java:208) at com.kms.katalon.core.webui.keyword.builtin.OpenBrowserKeyword$_openBrowser_closure1.doCall(OpenBrowserKeyword.groovy:74) at com.kms.katalon.core.webui.keyword.builtin.OpenBrowserKeyword$_openBrowser_closure1.call(OpenBrowserKeyword.groovy) at com.kms.katalon.core.webui.keyword.internal.WebUIKeywordMain.runKeyword(WebUIKeywordMain.groovy:20) at com.kms.katalon.core.webui.keyword.builtin.OpenBrowserKeyword.openBrowser(OpenBrowserKeyword.groovy:81) at com.kms.katalon.core.webui.keyword.builtin.OpenBrowserKeyword.execute(OpenBrowserKeyword.groovy:67) at com.kms.katalon.core.keyword.internal.KeywordExecutor.executeKeywordForPlatform(KeywordExecutor.groovy:72) at com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords.openBrowser(WebUiBuiltInKeywords.groovy:60) at com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords$openBrowser.call(Unknown Source) at Apples.openBrowserSetLanguage(Apples.groovy:28) at Apples.invokeMethod(Apples.groovy) at com.kms.katalon.core.main.CustomKeywordDelegatingMetaClass.invokeStaticMethod(CustomKeywordDelegatingMetaClass.java:50) at Script1581345333335.run(Script1581345333335.groovy:19) … 13 more ) %

@kevin.mcandrew

TempTestSuite1582284081015.run(TempTestSuite1582284081015.groovy:36) Caused by: java.io.IOException: GetExitCodeProcess error=6, The handle is invalid at com.kms.katalon.core.appium.driver.AppiumDriverManager.isAppiumServerStarted(AppiumDriverManager.java:193) at

When Open Browser keyword is called, Katalon Studio will re-use the existing Appium Server (if possible) to launch new Appium session.
In your case, Appium Server process got hung, you need to use: WebUI.closeBrowser() to close the current Appium Server process then use Open Browser keyword to let Katalon Studio start new Appium Server process.
Sample code:

import com.kms.katalon.core.webui.driver.DriverFactory
import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUI

try {
    WebUI.openBrowser('')
}
catch (Exception e) {
    if (DriverFactory.getWebDriver() != null) {
		WebUI.closeBrowser();
		WebUI.openBrowser('')
    }
}

Thanks @duyluong, appreciate your response.

Unfortunately, after implementing your code into my keyword, my tests now stop responding quite early on (latest example…out of ~80 test case, the first failed (maybe for other reasons), but after three further passes, the test has seemingly stopped…when I look in the logs from Azure, I see…


Test Suites/A******/L****** Content/T0001 - A****** L****** Content Tests -
Nokia 2.2 - 20200304_145907…4/78(5%)

I think from a previous execution, this typically lasts for 30 minutes but doesn’t seem to move on. Prior to implementing your code, the test sutie seemed to complete, enabling me to see the logs from Test-Ops, but now I can’t…where could I look in any logs that would be on our ‘execution machine’ (these are executed using Katalon Runtime) to find more info / supply it here, please?

Going back to your code, I tried WebUI.openBrowser(’’) (with no URL) and a later step navigating to URL, and I also tried populating the URL (which is a global variable) in the open browser step, but neither worked. I’ve got both the import statements included at the top of the script.

Is there anything else I can try? Please let me know what logs I can potentially gather - thanks

@kevin.mcandrew

Can you share us the execution log file? The above code only close and re-open browser if KS could not open at the previous step.

Related issue was created at: https://github.com/katalon-studio/katalon-studio/issues/295

@duyluong I’m sorry for the delayed response - is there a way to send logs privately to you please? (there could be certain URLs and other information in the logs that I may not want exposed publicly).

Thanks for your help,
Kevin

@duyluong it strikes me we have potentially two separate issues here…

  1. The test execution is not completing (and because of that, reports are not uploaded to TestOps) and the logs don’t tell me much

  2. ‘Unable to open browser with url…’ errors

Both issues are intermittent. Both do not happen with specific test cases, but can seemingly happen at any time during execution.

The problem I have is that when I use your script above, I can’t tell what’s happening because typically the execution doesn’t complete - it may be resolving the ‘Unable to open browser with url…’ errors but I can’t tell. When I don’t use your script (and have the Desired Capabilities setting mentioned above set to com.*), the execution typically completes, but typically with ‘Unable to open browser with url…’ errors.

Some other information:

I’ve downgraded Appium to 1.8.1 (from 1.16), and after doing this, I saw a rise in reliability (but not a full resolution).

Could there be some important configuration I’m not doing? Are there any specific Desired Capabilities that I need to be configuring? I’ve tried looking at your documentation but I can’t see where I might be going wrong?

This is becoming a real problem for us and we’d really appreciate it if you could investigate this urgently - thank you.

ps in your documentation https://docs.katalon.com/katalon-studio/videos/intro_mobile_app_testing.html - there is a broken link (titled ‘environment setup for mobile app automation’) - does this contain any information that could be useful?

@kevin.mcandrew

The problem may come from Appium server. We are investing on this issue. We need to add some tweaks in our code to make Appium works better.

Document for configuring desired capabilities, you can refer to these links:
http://appium.io/docs/en/writing-running-appium/caps/

@duyluong thanks for the response - it’s good to know that there may be an issue and that you’re on to it - thank you.

In the meantime, obviously please let me know if you need any logs (preferably through a private channel), or additional information from us on the nature of our tests if that could be of any help.

Finally…and importantly…do you have an idea of timescales on this please? I appreciate that you’ve only just confirmed that you’re investigating the issue, but I’m also conscious that we can’t allow this to drag on indefinitely. Having invested in building a Katalon-based framework, we have created suites of hundreds of test cases, many of them executing on a variety of real devices, and reliability is everything.

Please can you just confirm how you plan to address this issue, and that you see this as a high priority?

Many thanks,
Kevin