Edge Chromium - Scheduled tests are failing, for this browser only

Greg_Vowles · October 23, 2020, 9:19am

I have 30 Test Suites consisting of 122 Test Cases. The tests suites run perfectly when instigated from Katalon Studio directly. However, I use a set of batch files to automate these tests nightly and until recently these have been working fine, but suddenly I find all tests are failing.

The tests run twice, once using Chrome and then using Edge Chromium. I installed the latest Webdrivers for both browsers but the tests kept failing. I then added the following config setting to the script: –config -webui.autoUpdateDrivers=true

This fixed the problem for Chrome, but the problem still persists for tests using Edge Chromium.

Here is an example of my scripts:

cd F:\Katalon-Studio-Engine

katalonc.exe -noSplash -runMode=console -projectPath=“F:\katalon-tests\katalon-tests.prj” -retry=0 -testSuiteCollectionPath=“Test Suites/Core Functionality/Dashboard/_Dashboard (Test Suite Collection)” -summaryReport -apiKey=“SANITISED” --config -webui.autoUpdateDrivers=true"

Katalon Studio Version: 7.7.2
OS Version: Windows 10
Browser Version: Microsoft Edge (Chromium) Version 86.0.622.51 (Official build) (64-bit)

As you can see, the tests pass for Chrome, but fail every time for Edge:

duyluong · October 23, 2020, 9:34am

Hi @Greg_Vowles

Can you share the message of the failed test on Edge Chrominum?

Greg_Vowles · October 23, 2020, 11:06am

Hi, thanks very much for replying - would you require the Execution log? If so, here is a snippet which I think is highlighting the fault:

<failure type="FAILED" message="Test Cases/General/Log On FAILED. Reason: com.kms.katalon.core.exception.StepFailedException: Unable to open browser with url: 'SANITISED/' at com.kms.katalon.core.webui.keyword.internal.WebUIKeywordMain.stepFailed(WebUIKeywordMain.groovy:64) at com.kms.katalon.core.webui.keyword.internal.WebUIKeywordMain.runKeyword(WebUIKeywordMain.groovy:26) at com.kms.katalon.core.webui.keyword.builtin.OpenBrowserKeyword.openBrowser(OpenBrowserKeyword.groovy:81) at com.kms.katalon.core.webui.keyword.builtin.OpenBrowserKeyword.execute(OpenBrowserKeyword.groovy:67) at com.kms.katalon.core.keyword.internal.KeywordExecutor.executeKeywordForPlatform(KeywordExecutor.groovy:72) at com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords.openBrowser(WebUiBuiltInKeywords.groovy:60) at Log On.run(Log On:17) at com.kms.katalon.core.main.ScriptEngine.run(ScriptEngine.java:194) at com.kms.katalon.core.main.ScriptEngine.runScriptAsRawText(ScriptEngine.java:119) at com.kms.katalon.core.main.TestCaseExecutor.runScript(TestCaseExecutor.java:337) at com.kms.katalon.core.main.TestCaseExecutor.doExecute(TestCaseExecutor.java:328) at com.kms.katalon.core.main.TestCaseExecutor.processExecutionPhase(TestCaseExecutor.java:307) at com.kms.katalon.core.main.TestCaseExecutor.accessMainPhase(TestCaseExecutor.java:299) at com.kms.katalon.core.main.TestCaseExecutor.execute(TestCaseExecutor.java:233) at com.kms.katalon.core.main.TestSuiteExecutor.accessTestCaseMainPhase(TestSuiteExecutor.java:169) at com.kms.katalon.core.main.TestSuiteExecutor.accessTestSuiteMainPhase(TestSuiteExecutor.java:142) at com.kms.katalon.core.main.TestSuiteExecutor.execute(TestSuiteExecutor.java:91) at com.kms.katalon.core.main.TestCaseMain.startTestSuite(TestCaseMain.java:157) at com.kms.katalon.core.main.TestCaseMain$startTestSuite$0.call(Unknown Source) at TempTestSuite1603445842977.run(TempTestSuite1603445842977.groovy:37) Caused by: org.openqa.selenium.SessionNotCreatedException: session not created: This version of MSEdgeDriver only supports MSEdge version 88 Current browser version is 86.0.622.51 with binary path C:\Program Files (x86)\Microsoft\Edge\Application\msedge.exe Build info: version: ‘3.141.59’, revision: ‘e82be7d358’, time: ‘2018-11-14T08:25:53’ System info: host: ‘PC-NAME’, ip: ‘1XXX’, os.name: ‘Windows 10’, os.arch: ‘amd64’, os.version: ‘10.0’, java.version: ‘1.8.0_181’ Driver info: driver.version: CEdgeDriver

Greg_Vowles · October 23, 2020, 9:48am

Note that when I check my version of Edge Chromium, it tells me that it’s up to date - where do I find version 88? (which the error implies I need).

Microsoft Edge is up to date.
Version 86.0.622.51 (Official build) (64-bit)

Greg_Vowles · October 23, 2020, 11:05am

UPDATE: I have fixed this by doing three things:

Downgrading the EDGE webdriver to version 86, to match my browser (Note that version 88 of Edge Chromium isn’t even available yet, so the webdriver shouldn’t be downloaded when I choose to update my drivers in Katalon IMO).
Downloading the correct Google Chrome webdriver to match my browser.

I had to replace the webdriver in two locations:

C:\Katalon-Studio-Engine\configuration\resources\temp\webdriver
C:\Katalon-Studio-Engine\configuration\resources\drivers

It seems the one in the temp location is the copy the tests were using.

Removing the UpdateWebdriver config from all of my schedules.

This has solved the problem. Hope this helps someone!

Russ_Thomas · October 23, 2020, 1:53pm

@duyluong

That sounds like a race condition.

Does Tools>Update WebDrivers clean the temp folder before executing the update?

Is it safe to empty that folder folder as part of a manual clean up before updating the drivers?

Greg_Vowles · January 23, 2021, 12:37pm

I’ve noticed all of my edge tests are failing again. I would guess that edge has automatically updated and now the web driver doesn’t match. I’ve again tried updating the web driver via Katalon but the tests continue to fail, so looks like I’ll need to carry out the same steps I did before to fix it.

Greg_Vowles · January 25, 2021, 10:08am

UPDATE: Katalon 7.9.0 seems to fix the issues above. I now only see Webdrivers under F:\Katalon-Studio-Engine\configuration\resources\drivers (no longer seeing Webdrivers in a temp folder) and all tests are running properly again.

So: Replace the 7.8.x version entirely with 7.9.0, then from Katalon, choose to update webdrivers. All good.