I have a number of web tests that are passing almost 100% in Chrome (desktop), but I’m seeing inconsistent failures when executing on real Android devices. These tests are spread over 5 test suites, all kicked off from one Test Suite collection executed using Katalon Runtime (kicked off via Azure).
The error - for each failed test case - seems to relate to step 1 of each failed test case, which is a custom keyword (openBrowserSetLanguage), which basically opens the browser with a URL (defined by a variable - I’m using data-driven testing through test suites data-binding), and then selects the relevant language in the web page through a series of clicks, wait-for statements, and delays).
A typical error message, which all point to this keyword is shown below (I’ve hidden the URLs purposefully, but it’s the same URL for every test case).
I’m seeing failures when running it on a Nokia 5 (Android 9) and Sony Z5 Compact (Android 7), but running it once on my own Google Pixel 2XL resulted in no failures.
Coincidentally or not, where there are failures, they’ve so far all been from the same test suite. I have 5 test suites, and in my most recent execution, 22 out of 30 tests failed all within test suite ‘A’. However, on the previous run, all test cases in test suite ‘A’ passed, but 12 out of 22 test cases in test suite ‘B’ failed.
It’s almost like once a Test Suite has a failed test case, further test cases fail. This makes me think that perhaps the device / operating system has had an issue that it has somehow recovered from (after an amount of time), but this is just a theory.
Any suggestions as to what could be causing this, and how to rectify these issues would be appreciated.
Should I expect to be able to run these tests - trouble free - on older devices? Does the java.io.IOException error below have any significance? Are there any preferred strategies for running tests in volume on mobile devices (e.g. could there be memory/leak issues etc., and is there anything I can do to negate this type of thin?)?
I’m using Katalon Runtime Engine 64-7.2.1. Chrome version on all mobile devices is 80.0.3987.99, and I have Chromedriver.exe v80.
Unable to open browser with url: ‘https://[URL]’ (Root cause: com.kms.katalon.core.exception.StepFailedException: Unable to open browser with url: ‘https://[URL]’ at com.kms.katalon.core.webui.keyword.internal.WebUIKeywordMain.stepFailed(WebUIKeywordMain.groovy:64) at com.kms.katalon.core.webui.keyword.internal.WebUIKeywordMain.runKeyword(WebUIKeywordMain.groovy:26) at com.kms.katalon.core.webui.keyword.builtin.OpenBrowserKeyword.openBrowser(OpenBrowserKeyword.groovy:81) at com.kms.katalon.core.webui.keyword.builtin.OpenBrowserKeyword.execute(OpenBrowserKeyword.groovy:67) at com.kms.katalon.core.keyword.internal.KeywordExecutor.executeKeywordForPlatform(KeywordExecutor.groovy:72) at com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords.openBrowser(WebUiBuiltInKeywords.groovy:60) at com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords$openBrowser.call(Unknown Source) at Apples.openBrowserSetLanguage(Apples.groovy:28) at Apples.invokeMethod(Apples.groovy) at com.kms.katalon.core.main.CustomKeywordDelegatingMetaClass.invokeStaticMethod(CustomKeywordDelegatingMetaClass.java:50) at Verify category action button.run(Verify category action button:19) at com.kms.katalon.core.main.ScriptEngine.run(ScriptEngine.java:194) at com.kms.katalon.core.main.ScriptEngine.runScriptAsRawText(ScriptEngine.java:119) at com.kms.katalon.core.main.TestCaseExecutor.runScript(TestCaseExecutor.java:337) at com.kms.katalon.core.main.TestCaseExecutor.doExecute(TestCaseExecutor.java:328) at com.kms.katalon.core.main.TestCaseExecutor.processExecutionPhase(TestCaseExecutor.java:307) at com.kms.katalon.core.main.TestCaseExecutor.accessMainPhase(TestCaseExecutor.java:299) at com.kms.katalon.core.main.TestCaseExecutor.execute(TestCaseExecutor.java:233) at com.kms.katalon.core.main.TestSuiteExecutor.accessTestCaseMainPhase(TestSuiteExecutor.java:169) at com.kms.katalon.core.main.TestSuiteExecutor.accessTestSuiteMainPhase(TestSuiteExecutor.java:142) at com.kms.katalon.core.main.TestSuiteExecutor.execute(TestSuiteExecutor.java:91) at com.kms.katalon.core.main.TestCaseMain.startTestSuite(TestCaseMain.java:157) at com.kms.katalon.core.main.TestCaseMain$startTestSuite$0.call(Unknown Source) at TempTestSuite1582284081015.run(TempTestSuite1582284081015.groovy:36) Caused by: java.io.IOException: GetExitCodeProcess error=6, The handle is invalid at com.kms.katalon.core.appium.driver.AppiumDriverManager.isAppiumServerStarted(AppiumDriverManager.java:193) at com.kms.katalon.core.appium.driver.AppiumDriverManager.startAppiumServerJS(AppiumDriverManager.java:257) at com.kms.katalon.core.appium.driver.AppiumDriverManager.startAppiumServerJS(AppiumDriverManager.java:406) at com.kms.katalon.core.appium.driver.AppiumDriverManager.ensureServicesStarted(AppiumDriverManager.java:243) at com.kms.katalon.core.appium.driver.AppiumDriverManager.createMobileDriver(AppiumDriverManager.java:472) at com.kms.katalon.core.webui.driver.WebMobileDriverFactory.createMobileDriver(WebMobileDriverFactory.java:139) at com.kms.katalon.core.webui.driver.DriverFactory.startNewBrowser(DriverFactory.java:325) at com.kms.katalon.core.webui.driver.DriverFactory.openWebDriver(DriverFactory.java:208) at com.kms.katalon.core.webui.keyword.builtin.OpenBrowserKeyword$_openBrowser_closure1.doCall(OpenBrowserKeyword.groovy:74) at com.kms.katalon.core.webui.keyword.builtin.OpenBrowserKeyword$_openBrowser_closure1.call(OpenBrowserKeyword.groovy) at com.kms.katalon.core.webui.keyword.internal.WebUIKeywordMain.runKeyword(WebUIKeywordMain.groovy:20) at com.kms.katalon.core.webui.keyword.builtin.OpenBrowserKeyword.openBrowser(OpenBrowserKeyword.groovy:81) at com.kms.katalon.core.webui.keyword.builtin.OpenBrowserKeyword.execute(OpenBrowserKeyword.groovy:67) at com.kms.katalon.core.keyword.internal.KeywordExecutor.executeKeywordForPlatform(KeywordExecutor.groovy:72) at com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords.openBrowser(WebUiBuiltInKeywords.groovy:60) at com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords$openBrowser.call(Unknown Source) at Apples.openBrowserSetLanguage(Apples.groovy:28) at Apples.invokeMethod(Apples.groovy) at com.kms.katalon.core.main.CustomKeywordDelegatingMetaClass.invokeStaticMethod(CustomKeywordDelegatingMetaClass.java:50) at Script1581345333335.run(Script1581345333335.groovy:19) … 13 more ) %