Hi,
I had configured a pipeline on GitLab by following the instructions as mentoined here GitLab Integration | Katalon Docs. Pipeline is working but script execution is something that is frustrating me now. The reason
- Every time I ran script htrough pipeline, it started failing after first / second script with errors like
– Unable to take screenshot
– click / set text / verify elements actions failing because they are not identifiable with their XPATH
This is the command I am using in pipeline
- katalonc.sh -projectPath=$KATALON_PROJECT_DIR -apiKey=$KATALON_API_KEY -browserType="Chrome" -retry=0 -maxFailedTests=3 -testSuitePath="Test Suites/Test Suites/Unified Agent/Agent_Common_Scenarios" -executionProfile="CIM" --config -webui.autoUpdateDrivers=true
Here are one of the failed execution logs
Running with gitlab-runner 15.7.1 (6d480948)
on Primary-Shared qSohhN5_
Preparing the "docker" executor
00:36
Using Docker executor with image katalonstudio/katalon ...
Starting service docker:dind ...
Pulling docker image docker:dind ...
Using docker image sha256:cd27a71ea4555b89fda3a6fd0118200e247b9f05cd07d3c16c58e67bee7c290d for docker:dind with digest docker@sha256:3cf33ff6e893a39262bd9f6b85ff46e068fa2a98d201326e9c3058a22a21a6bb ...
Waiting for services to be up and running (timeout 30 seconds)...
*** WARNING: Service runner-qsohhn5-project-512-concurrent-0-ee00f2f0ef1d317d-docker-0 probably didn't start properly.
Health check error:
service "runner-qsohhn5-project-512-concurrent-0-ee00f2f0ef1d317d-docker-0-wait-for-service" timeout
Health check container logs:
Service container logs:
2023-03-03T07:21:12.494785519Z Certificate request self-signature ok
2023-03-03T07:21:12.494861734Z subject=CN = docker:dind server
2023-03-03T07:21:12.520588770Z /certs/server/cert.pem: OK
2023-03-03T07:21:18.681428697Z Certificate request self-signature ok
2023-03-03T07:21:18.681466275Z subject=CN = docker:dind client
2023-03-03T07:21:18.706903464Z /certs/client/cert.pem: OK
2023-03-03T07:21:19.336631779Z time="2023-03-03T07:21:19.336342692Z" level=info msg="Starting up"
2023-03-03T07:21:19.342892878Z time="2023-03-03T07:21:19.342739996Z" level=warning msg="could not change group /var/run/docker.sock to docker: group docker not found"
2023-03-03T07:21:19.343047498Z failed to load listeners: can't create unix socket /var/run/docker.sock: device or resource busy
*********
Pulling docker image katalonstudio/katalon ...
Using docker image sha256:b1b382d16c3c76aa13bf52b5f694e5cdcaac547a0843d90c40760e2a64e0335c for katalonstudio/katalon with digest katalonstudio/katalon@sha256:f38539714c2ecf20a08df244333d7a2439e0e30d90ddecac1aca2ceb8b55ad00 ...
Preparing environment
00:00
Running on runner-qsohhn5-project-512-concurrent-0 via vmi758837.contaboserver.net...
Getting source from Git repository
00:02
Fetching changes with git depth set to 50...
Reinitialized existing Git repository in /builds/qa/cim_automation_scripts/.git/
Checking out b9ad3cfb as Develop...
Removing .classpath
Removing .project
Removing Checkpoints/
Removing Include/features/
Removing Include/scripts/
Removing Libs/
Removing Plugins/
Removing Reports/
Removing bin/
Removing core
Skipping Git submodules setup
Executing "step_script" stage of the job script
14:48
Using docker image sha256:b1b382d16c3c76aa13bf52b5f694e5cdcaac547a0843d90c40760e2a64e0335c for katalonstudio/katalon with digest katalonstudio/katalon@sha256:f38539714c2ecf20a08df244333d7a2439e0e30d90ddecac1aca2ceb8b55ad00 ...
Entrypoint
+ echo Entrypoint
+ '[' -z '' ']'
+ exec sh -c 'if [ -x /usr/local/bin/bash ]; then
exec /usr/local/bin/bash
elif [ -x /usr/bin/bash ]; then
exec /usr/bin/bash
elif [ -x /bin/bash ]; then
exec /bin/bash
elif [ -x /usr/local/bin/sh ]; then
exec /usr/local/bin/sh
elif [ -x /usr/bin/sh ]; then
exec /usr/bin/sh
elif [ -x /bin/sh ]; then
exec /bin/sh
elif [ -x /busybox/sh ]; then
exec /busybox/sh
else
echo shell not found
exit 1
fi
'
$ katalonc.sh -projectPath=$KATALON_PROJECT_DIR -apiKey=$KATALON_API_KEY -browserType="Chrome" -retry=0 -statusDelay=20 -testSuitePath="Test Suites/Test Suites/Unified Agent/Agent_Login_Logout_Scenarios" -executionProfile="CIM" --config -webui.autoUpdateDrivers=true
+ echo 'Starting Katalon Studio'
+ cat /katalon/version
Starting Katalon Studio
Mozilla Firefox 107.0
Google Chrome 108.0.5359.94
Microsoft Edge 108.0.1462.42
Katalon Studio
+ args=("$KATALON_KATALON_INSTALL_DIR/katalonc" "$@")
+ xvfb-run -s '-screen 0 1024x768x24' /opt/katalonstudio/katalonc -projectPath=/builds/qa/cim_automation_scripts -apiKey=$KATALON_API_KEY -browserType=Chrome -retry=0 -statusDelay=20 '-testSuitePath=Test Suites/Test Suites/Unified Agent/Agent_Login_Logout_Scenarios' -executionProfile=CIM --config -webui.autoUpdateDrivers=true
Katalon workspace folder is set to default location: /tmp/session-ae7d659f
Starting Groovy-Eclipse compiler resolver. Specified compiler level: unspecified
139 org.codehaus.groovy_2.4.20.v202009301404-e2006-RELEASE ACTIVE
Activating...
Start getting machine ID on Linux
End getting machine ID on Linux f65c27a93a6238c9028ce1df638ee760
Start appending additional signatures
hardwareSerialNumber
osDependentUsername root
End appending additional signatures f65c27a93a6238c9028ce1df638ee760__root
Start activating offline...
Search for valid offline licenses in folder: /root/.katalon/license
The number of valid offline licenses: 0
Start activating online...
INFO: Katalon Version: 8.5.5
INFO: Katalon Edition: Standalone
INFO: Command-line arguments: -projectPath=/builds/qa/cim_automation_scripts -apiKey=******** -browserType=Chrome -retry=0 -statusDelay=20 -testSuitePath=Test Suites/Test Suites/Unified Agent/Agent_Login_Logout_Scenarios -executionProfile=CIM --config -webui.autoUpdateDrivers=true
INFO: User working dir: /builds/qa/cim_automation_scripts
INFO: Error log: /tmp/session-ae7d659f/.metadata/.log
INFO: Katalon KatOne server URL: https://admin.katalon.com
INFO: Katalon TestOps server URL: https://testops.katalon.io
INFO: Katalon Store server URL: https://store.katalon.com
INFO: User home: /root
INFO: Java vendor: Private Build
INFO: Java version: 1.8.0_352
INFO: Local OS: Linux 64bit
INFO: CPU load: 0%
INFO: Total memory: 9961 MB
INFO: Free memory: 6312 MB
INFO: Machine ID: 3d13faaf2b45912565749dbf4e92ccb3
Project path is a folder.
Examine file 'SSH Key File.pub'.
Examine folder 'Test Cases'.
Examine folder 'Include'.
Examine file 'build.gradle'.
Examine folder 'settings'.
Examine file '.gitlab-ci.yml'.
Examine folder 'Object Repository'.
Examine file 'CIM_Test_Project.prj'.
Found project file '/builds/qa/cim_automation_scripts/CIM_Test_Project.prj'.
Examine folder 'Data Files'.
Examine folder 'Screenshots'.
Examine file 'GlobalVariables.glbl'.
Examine file 'SSH Key File'.
Examine folder 'Keywords'.
Examine folder 'Test Suites'.
Examine folder 'Drivers'.
Examine folder '.git'.
Examine file 'mintty.exe.stackdump'.
Examine file '.gitignore'.
Examine folder '.settings'.
Examine file 'console.properties'.
Examine folder 'Profiles'.
Examine folder 'Test Listeners'.
Examine folder 'Scripts'.
Cleaning up workspace
Opening project file: /builds/qa/cim_automation_scripts/CIM_Test_Project.prj
Warning! Please run Katalon execution command outside of the project folder.
Generating global variables...
Project 'CIM_Test_Project' opened
Start reloading plugins...
Katalon version: 8.5.5
Plugin info URL: https://store.katalon.com/api/products/ks?appVersion=8.5.5&appType=ENGINE&licenseType=TRIAL
Start check license task
Chrome driver is located at: /opt/katalonstudio/configuration/resources/drivers/chromedriver_linux64/chromedriver
INFO: KATALON_JAVA_HOME: null
--------------------------------------------------------------------------------
Test Suites/Test Suites/Unified Agent/Agent_Login_Logout_Scenarios - Chrome - 20
230303_072206............................................................0/1(0%)
--------------------------------------------------------------------------------
2023-03-03 07:22:08.878 INFO c.k.katalon.core.main.TestSuiteExecutor - START Test Suites/Test Suites/Unified Agent/Agent_Login_Logout_Scenarios
2023-03-03 07:22:08.914 INFO c.k.katalon.core.main.TestSuiteExecutor - hostName = root - runner-qsohhn5-project-512-concurrent-0
2023-03-03 07:22:08.915 INFO c.k.katalon.core.main.TestSuiteExecutor - os = Linux 64bit
2023-03-03 07:22:08.916 INFO c.k.katalon.core.main.TestSuiteExecutor - hostAddress = 172.17.0.2
2023-03-03 07:22:08.917 INFO c.k.katalon.core.main.TestSuiteExecutor - katalonVersion = 8.5.5.208
Test Suites/Test Suites/Unified Agent/Agent_Login_Logout_Scenarios
inside of start of test suite.
2023-03-03 07:22:09.816 INFO c.k.katalon.core.main.TestCaseExecutor - --------------------
2023-03-03 07:22:09.816 INFO c.k.katalon.core.main.TestCaseExecutor - CALL Test Cases/Unified_Agent_Scripts/Common_Test_Scripts/StartBrowserForAgent
2023-03-03 07:22:09.847 INFO c.k.katalon.core.main.TestCaseExecutor - varWebProtocol = https
2023-03-03 07:22:09.848 INFO c.k.katalon.core.main.TestCaseExecutor - varFQDN_IPAddress = cim5.expertflow.com
2023-03-03 07:22:09.850 INFO c.k.katalon.core.main.TestCaseExecutor - varAppend_URL = /unified-agent/#/login
2023-03-03 07:22:10.179 DEBUG testcase.StartBrowserForAgent - 1: System.setProperty("webdriver.chrome.driver", getChromeDriverPath())
2023-03-03 07:22:10.214 DEBUG testcase.StartBrowserForAgent - 2: prefs = new java.util.HashMap()
2023-03-03 07:22:10.215 DEBUG testcase.StartBrowserForAgent - 3: prefs.put("profile.default_content_setting_values.notifications", 2)
2023-03-03 07:22:10.216 DEBUG testcase.StartBrowserForAgent - 4: options = new org.openqa.selenium.chrome.ChromeOptions()
2023-03-03 07:22:10.231 DEBUG testcase.StartBrowserForAgent - 5: options.setExperimentalOption("prefs", prefs)
2023-03-03 07:22:10.233 DEBUG testcase.StartBrowserForAgent - 6: agentDriver = new org.openqa.selenium.chrome.ChromeDriver(options)
Starting ChromeDriver 107.0.5304.62 (1eec40d3a5764881c92085aaee66d25075c159aa-refs/branch-heads/5304@{#942}) on port 30202
Only local connections are allowed.
Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
[1677828130.619][SEVERE]: bind() failed: Cannot assign requested address (99)
ChromeDriver was started successfully.
[1677828133.823][WARNING]: This version of ChromeDriver has not been tested with Chrome version 108.
Mar 03, 2023 7:22:14 AM org.openqa.selenium.remote.ProtocolHandshake createSession
INFO: Detected dialect: W3C
2023-03-03 07:22:14.116 DEBUG testcase.StartBrowserForAgent - 7: changeWebDriver(agentDriver)
2023-03-03 07:22:14.129 INFO c.k.k.core.webui.driver.DriverFactory - sessionId = 32f7bd813550127423211fd82a0dc31a
2023-03-03 07:22:14.152 INFO c.k.k.core.webui.driver.DriverFactory - browser = Chrome 108.0.0.0
2023-03-03 07:22:14.153 INFO c.k.k.core.webui.driver.DriverFactory - platform = Linux
2023-03-03 07:22:14.153 INFO c.k.k.core.webui.driver.DriverFactory - seleniumVersion = 3.141.59
2023-03-03 07:22:14.157 INFO c.k.k.core.webui.driver.DriverFactory - proxyInformation = ProxyInformation { proxyOption=NO_PROXY, proxyServerType=HTTP, username=, password=********, proxyServerAddress=, proxyServerPort=0, executionList="", isApplyToDesiredCapabilities=true }
2023-03-03 07:22:14.190 DEBUG testcase.StartBrowserForAgent - 8: maximizeWindow(STOP_ON_FAILURE)
2023-03-03 07:22:14.490 DEBUG testcase.StartBrowserForAgent - 9: URLToNavigate = customPackage.GenericCIMMethods.buildURL(varWebProtocol, varFQDN_IPAddress, varAppend_URL)
2023-03-03 07:22:14.499 INFO k.k.c.m.CustomKeywordDelegatingMetaClass - customPackage.GenericCIMMethods.buildURL is PASSED
2023-03-03 07:22:14.500 DEBUG testcase.StartBrowserForAgent - 10: println("User want to start browser with URL " + URLToNavigate)
User want to start browser with URL https://cim5.expertflow.com/unified-agent/#/login
2023-03-03 07:22:14.504 DEBUG testcase.StartBrowserForAgent - 11: navigateToUrl(URLToNavigate, STOP_ON_FAILURE)
2023-03-03 07:22:16.683 DEBUG testcase.StartBrowserForAgent - 12: waitForPageLoad(60, STOP_ON_FAILURE)
2023-03-03 07:22:16.768 DEBUG testcase.StartBrowserForAgent - 13: delay(5, STOP_ON_FAILURE)
2023-03-03 07:22:21.776 DEBUG testcase.StartBrowserForAgent - 14: takeScreenshot(STOP_ON_FAILURE)
2023-03-03 07:22:22.127 INFO c.k.katalon.core.main.TestCaseExecutor - END CALL Test Cases/Unified_Agent_Scripts/Common_Test_Scripts/StartBrowserForAgent
2023-03-03 07:22:22.128 INFO c.k.katalon.core.main.TestCaseExecutor - --------------------
2023-03-03 07:22:22.142 INFO c.k.katalon.core.main.TestCaseExecutor - --------------------
2023-03-03 07:22:22.143 INFO c.k.katalon.core.main.TestCaseExecutor - START Test Cases/Unified_Agent_Scripts/Login_Scripts/LoginAgentWithoutKeycloakRecord(CIM-1081)
Test Cases/Unified_Agent_Scripts/Login_Scripts/LoginAgentWithoutKeycloakRecord(CIM-1081)
[:]
inside of start of test case.
2023-03-03 07:22:22.383 INFO c.k.katalon.core.main.TestCaseExecutor - --------------------
2023-03-03 07:22:22.383 INFO c.k.katalon.core.main.TestCaseExecutor - CALL Test Cases/Misc_Scripts/Keycloak_Scripts/LoginOnKeycloak
2023-03-03 07:22:22.391 INFO c.k.katalon.core.main.TestCaseExecutor - varWebProtocol = https
2023-03-03 07:22:22.392 INFO c.k.katalon.core.main.TestCaseExecutor - varURLToAppend = /auth/
2023-03-03 07:22:22.392 INFO c.k.katalon.core.main.TestCaseExecutor - varUserName = admin
2023-03-03 07:22:22.392 INFO c.k.katalon.core.main.TestCaseExecutor - varUserPwd = admin
2023-03-03 07:22:22.580 INFO c.k.katalon.core.main.TestCaseExecutor - --------------------
2023-03-03 07:22:22.581 INFO c.k.katalon.core.main.TestCaseExecutor - CALL Test Cases/Common_Scripts/StartBrowser
2023-03-03 07:22:22.587 INFO c.k.katalon.core.main.TestCaseExecutor - varWebProtocol = https
2023-03-03 07:22:22.587 INFO c.k.katalon.core.main.TestCaseExecutor - varFQDN_IPAddress = cim5.expertflow.com
2023-03-03 07:22:22.587 INFO c.k.katalon.core.main.TestCaseExecutor - varAppend_URL = /auth/
2023-03-03 07:22:22.866 DEBUG testcase.StartBrowser - 1: System.setProperty("webdriver.chrome.driver", getChromeDriverPath())
2023-03-03 07:22:22.867 DEBUG testcase.StartBrowser - 2: adminDriver = new org.openqa.selenium.chrome.ChromeDriver()
Starting ChromeDriver 107.0.5304.62 (1eec40d3a5764881c92085aaee66d25075c159aa-refs/branch-heads/5304@{#942}) on port 11010
Only local connections are allowed.
Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
ChromeDriver was started successfully.
[1677828142.879][SEVERE]: bind() failed: Cannot assign requested address (99)
[1677828143.657][WARNING]: This version of ChromeDriver has not been tested with Chrome version 108.
Mar 03, 2023 7:22:23 AM org.openqa.selenium.remote.ProtocolHandshake createSession
INFO: Detected dialect: W3C
2023-03-03 07:22:23.818 DEBUG testcase.StartBrowser - 3: changeWebDriver(adminDriver)
2023-03-03 07:22:23.823 INFO c.k.k.core.webui.driver.DriverFactory - sessionId = 0377412b5fd67b497a2ebdc601e0114a
2023-03-03 07:22:23.833 INFO c.k.k.core.webui.driver.DriverFactory - browser = Chrome 108.0.0.0
2023-03-03 07:22:23.836 INFO c.k.k.core.webui.driver.DriverFactory - platform = Linux
2023-03-03 07:22:23.837 INFO c.k.k.core.webui.driver.DriverFactory - seleniumVersion = 3.141.59
2023-03-03 07:22:23.838 INFO c.k.k.core.webui.driver.DriverFactory - proxyInformation = ProxyInformation { proxyOption=NO_PROXY, proxyServerType=HTTP, username=, password=********, proxyServerAddress=, proxyServerPort=0, executionList="", isApplyToDesiredCapabilities=true }
2023-03-03 07:22:23.844 DEBUG testcase.StartBrowser - 4: maximizeWindow()
2023-03-03 07:22:23.973 DEBUG testcase.StartBrowser - 5: delay(3)
--------------------------------------------------------------------------------
Test Suites/Test Suites/Unified Agent/Agent_Login_Logout_Scenarios - Chrome - 20
230303_072206............................................................0/1(0%)
--------------------------------------------------------------------------------
2023-03-03 07:22:26.978 DEBUG testcase.StartBrowser - 6: URLToNavigate = customPackage.GenericCIMMethods.buildURL(varWebProtocol, varFQDN_IPAddress, varAppend_URL)
2023-03-03 07:22:26.978 INFO k.k.c.m.CustomKeywordDelegatingMetaClass - customPackage.GenericCIMMethods.buildURL is PASSED
2023-03-03 07:22:26.979 DEBUG testcase.StartBrowser - 7: println("User want to start browser with URL " + URLToNavigate)
User want to start browser with URL https://cim5.expertflow.com/auth/
2023-03-03 07:22:26.980 DEBUG testcase.StartBrowser - 8: navigateToUrl(URLToNavigate)
2023-03-03 07:22:28.483 DEBUG testcase.StartBrowser - 9: waitForPageLoad(15, CONTINUE_ON_FAILURE)
2023-03-03 07:22:28.501 DEBUG testcase.StartBrowser - 10: takeScreenshot()
2023-03-03 07:22:28.615 INFO c.k.katalon.core.main.TestCaseExecutor - END CALL Test Cases/Common_Scripts/StartBrowser
2023-03-03 07:24:47.881 INFO c.k.k.c.webui.common.WebUiCommonHelper - Unable to find the element located by 'By.xpath: //a[@id='nav-item-users'][text()='Users']'. Please recheck the objects properties to make sure the desired element is located.
2023-03-03 07:24:47.898 WARN k.k.c.w.k.b.WaitForElementVisibleKeyword - Web element with id: 'Object Repository/Keycloak_Objects/AddUsers/option_Users' located by '//a[@id='nav-item-users'][text()='Users']' not found
2023-03-03 07:24:47.997 INFO c.k.katalon.core.main.TestCaseExecutor - END CALL Test Cases/Misc_Scripts/Keycloak_Scripts/LoginOnKeycloak
2023-03-03 07:24:47.997 INFO c.k.katalon.core.main.TestCaseExecutor - --------------------
2023-03-03 07:24:48.054 INFO c.k.katalon.core.main.TestCaseExecutor - --------------------
2023-03-03 07:24:48.054 INFO c.k.katalon.core.main.TestCaseExecutor - CALL Test Cases/Misc_Scripts/Keycloak_Scripts/AddEFRealm
2023-03-03 07:24:48.254 DEBUG testcase.AddEFRealm - 1: takeScreenshot()
2023-03-03 07:24:48.341 DEBUG testcase.AddEFRealm - 2: isEFRealmNotAvailable = verifyElementNotPresent(findTestObject("Keycloak_Objects/AddRealm/option_ExpertflowRealm"), 3, OPTIONAL)
2023-03-03 07:24:48.432 DEBUG testcase.AddEFRealm - 3: if (isEFRealmNotAvailable)
2023-03-03 07:24:48.433 DEBUG testcase.AddEFRealm - 1: println("EF realm not uploaded or not selected.")
EF realm not uploaded or not selected.
2023-03-03 07:24:48.434 DEBUG testcase.AddEFRealm - 2: click(findTestObject("Keycloak_Objects/AddRealm/dropdown_RealmHolder"))
--------------------------------------------------------------------------------
Test Suites/Test Suites/Unified Agent/Agent_Login_Logout_Scenarios - Chrome - 20
230303_072206............................................................0/1(0%)
--------------------------------------------------------------------------------
2023-03-03 07:25:19.478 INFO c.k.k.c.webui.common.WebUiCommonHelper - Unable to find the element located by 'By.xpath: //button[@data-testid='realmSelectorToggle']'. Please recheck the objects properties to make sure the desired element is located.
The frustrating thing is the same tests are working absolutely fine with KS or in console mode execution which proves that the tests are not flaky but the problem lies somewhere else 
I am tired and not know what to check/do to execute tests without issue through pipeline. One thing worth to mention that our GitLab server is hosted on cloud not on-prem but not sure if that can impact the script execution in any way.
I am sure that will not be the case but is there a chance that the browses inside of katalon image are not opearting or malfunctioning for some reason? Anyone currently executing scripts using GitLab integration the same way as I am doing?
Any thoughts/opinions are most welcome which can give me clue to the root cause of what is happening?