Stale Element Reference

[devin.brown]

I am getting the “Stale Element Reference” seemingly at random. I have a page where I capture the text on a vehicle make link and store it to a variable. I then click the link which updates the panel with a new report (not reloading the page). One the new report loads, I then capture vehicle makes from all of the rows in the grid and loop through them to verify that all are the same. Then I click on the 1st rows, model link and do the same process of checking.

This is one point (clicking the model link) that I am randomly getting the error. If it passes this point, it then clicks a bread crumb that takes me back to the original report where I click the Make link, but this time it captures the text and clicks the link for the Model. This seems to be the point that throws this error the most. Again, it seems random as to whether it will pass or get the Stale Element Reference.

Here is a snipping of code where it is doing the checks:

CustomKeywords.'dwKeywords.WaitForSpinner.spinnerNotVisible'('#panel3 multi-comp div div datatable progress-bar div')
			
			CustomKeywords.'dwKeywords.MakeModelCheck.verifyMakeModel'(reportName, 'Make')
			
			CustomKeywords.'dwKeywords.WaitForSpinner.spinnerNotVisible'('#panel3 multi-comp div div datatable progress-bar div')
			
			CustomKeywords.'dwKeywords.MakeModelCheck.verifyMakeModel'(reportName, 'Model')
			
			// New Inventory Details Breadcrumb
			WebUI.executeJavaScript('document.querySelectorAll(\'#panel3 a\')[1].click()', null)
			
			CustomKeywords.'dwKeywords.WaitForSpinner.spinnerNotVisible'('#panel3 multi-comp div div datatable progress-bar div')
			
			CustomKeywords.'dwKeywords.MakeModelCheck.verifyMakeModel'(reportName, 'Model')

Here is where the CustomKeyword is defined:

public class WaitForSpinner {
	@Keyword
	def spinnerNotVisible(String selector) {
		def count = 0

		def spinnerHidden = false

		if (count >= 20) {
			return
		} else {
			if (spinnerHidden == false) {
				def classList = WebUI.executeJavaScript('return document.querySelectorAll(\'' + selector + '\')[0].classList', null)

				if (classList.contains('ng-hide')) {
					spinnerHidden = true

					return spinnerHidden
				} else {
					count++

					WebUI.delay(0.5)

					spinnerNotVisible(selector)
				}
			}
		}
	}
}

And here is the Error Log from the Katalon Console:

=============== ROOT CAUSE =====================

For trouble shooting, please visit: https://docs.katalon.com/katalon-studio/docs/troubleshooting.html

07-01-2021 09:18:43 AM Test Cases/VariableOperations/NewCarSalesReports

Elapsed time: 1m - 11.664s

Test Cases/VariableOperations/NewCarSalesReports FAILED.
Reason:
org.openqa.selenium.StaleElementReferenceException: stale element reference: element is not attached to the page document
(Session info: chrome=91.0.4472.124)
For documentation on this error, please visit: /documentation/webdriver/troubleshooting/errors/
Build info: version: ‘3.141.59’, revision: ‘e82be7d358’, time: ‘2018-11-14T08:25:53’
System info: host: ‘DWDEVWS008’, ip: ‘172.16.1.220’, os.name: ‘Windows 10’, os.arch: ‘amd64’, os.version: ‘10.0’, java.version: ‘1.8.0_181’
Driver info: com.kms.katalon.selenium.driver.CChromeDriver
Capabilities {acceptInsecureCerts: false, browserName: chrome, browserVersion: 91.0.4472.124, chrome: {chromedriverVersion: 90.0.4430.24 (4c6d850f087da…, userDataDir: C:\Users\DEVIN~1.BRO\AppDat…}, goog:chromeOptions: {debuggerAddress: localhost:51745}, javascriptEnabled: true, networkConnectionEnabled: false, pageLoadStrategy: normal, platform: WINDOWS, platformName: WINDOWS, proxy: Proxy(), setWindowRect: true, strictFileInteractability: false, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior: dismiss and notify, webauthn:extension:largeBlob: true, webauthn:virtualAuthenticators: true}
Session ID: 2d0245a6651b3aa85bb2510110a68abe
at org.openqa.selenium.remote.http.W3CHttpResponseCodec.createException(W3CHttpResponseCodec.java:187)
at org.openqa.selenium.remote.http.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:122)
at org.openqa.selenium.remote.http.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:49)
at org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:158)
at org.openqa.selenium.remote.service.DriverCommandExecutor.execute(DriverCommandExecutor.java:83)
at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:552)
at com.kms.katalon.selenium.driver.CChromeDriver.execute(CChromeDriver.java:19)
at org.openqa.selenium.remote.RemoteWebElement.execute(RemoteWebElement.java:285)
at org.openqa.selenium.remote.RemoteWebElement.getText(RemoteWebElement.java:166)
at org.openqa.selenium.support.events.EventFiringWebDriver$EventFiringWebElement.lambda$new$0(EventFiringWebDriver.java:404)
at com.sun.proxy.$Proxy11.getText(Unknown Source)
at org.openqa.selenium.support.events.EventFiringWebDriver$EventFiringWebElement.getText(EventFiringWebDriver.java:463)
at dwKeywords.MakeModelCheck.verifyMakeModel(MakeModelCheck.groovy:80)
at dwKeywords.MakeModelCheck.invokeMethod(MakeModelCheck.groovy)
at com.kms.katalon.core.main.CustomKeywordDelegatingMetaClass.invokeStaticMethod(CustomKeywordDelegatingMetaClass.java:50)
at NewCarSalesReports.processReports(NewCarSalesReports:126)
at Script1623172074810$processReports.callCurrent(Unknown Source)
at NewCarSalesReports.run(NewCarSalesReports:57)
at com.kms.katalon.core.main.ScriptEngine.run(ScriptEngine.java:194)
at com.kms.katalon.core.main.ScriptEngine.runScriptAsRawText(ScriptEngine.java:119)
at com.kms.katalon.core.main.TestCaseExecutor.runScript(TestCaseExecutor.java:369)
at com.kms.katalon.core.main.TestCaseExecutor.doExecute(TestCaseExecutor.java:360)
at com.kms.katalon.core.main.TestCaseExecutor.processExecutionPhase(TestCaseExecutor.java:339)
at com.kms.katalon.core.main.TestCaseExecutor.accessMainPhase(TestCaseExecutor.java:331)
at com.kms.katalon.core.main.TestCaseExecutor.execute(TestCaseExecutor.java:248)
at com.kms.katalon.core.main.TestCaseMain.runTestCase(TestCaseMain.java:142)
at com.kms.katalon.core.main.TestCaseMain.runTestCase(TestCaseMain.java:133)
at com.kms.katalon.core.main.TestCaseMain$runTestCase$0.call(Unknown Source)
at TempTestCase1625145519841.run(TempTestCase1625145519841.groovy:25)

Any help would be greatly appreciated!

[Brandon_Hein]

I will provide the long answer, and a tl;dr. I think it’s helpful to understand what this exception means so that you can avoid it more generally in the future.

Long Answer

You may already be aware of what a StaleElementReferenceException is, but I will give some context anyway.

This error occurs when you attempt to use a reference to a WebElement that is no longer valid against the current DOM (i.e. the reference is “stale”). In other words, you have captured a WebElement, but at some point the page has changed in such a way that this element reference is no longer valid. This does not have to be an entire page reload or a redirect; It can happen even if only a portion of the page has changed in some way (after an AJAX request, for example). The randomness of this exception comes from the fact that page load times vary across iterations, so sometimes your script is able to use the element before it is reloaded, sometimes not.

This can be hard to visualize in the HTML itself. Imagine for a moment a very real scenario: You’ve located a link element. At some point, the element for your link is then removed from the DOM, then reloaded into the DOM as the exact same element. To you as the observer, this element doesn’t appear to have changed, but it’s technically not the same element, it’s a “copy”. If you located this link prior to it being removed/added back, then tried to use that reference after the reload, you would get the same exception. This is just one example among countless others.

Now, in reference to your exact custom keyword, I don’t have line numbers, but according to the error log its at dwKeywords.MakeModelCheck.verifyMakeModel(MakeModelCheck.groovy:80). If you can point me to exactly which line of code that is, I can give a specific answer as to what’s going on. However, I think you’d be better off following my advice below to handle this error more generally.

"Short" Answer

More generally, here’s what you can do to avoid StaleElementExceptions across the board. If you follow these simple principles, you will rarely ever see this exception in your future automation efforts:

1.) Ensure that EVERY element reference is 1-time use. In other words, locate the element, use it, and DO NOT use that same reference again. If you want to reference that element again, you should re-locate it in the HTML every single time.

2.) Ensure that you have proper wait conditions in place for every action you take in your application. Always wait for the page to be “static” prior to locating elements and using them.

I’m happy to elaborate, but this might be too much info already. Let me know

[devin.brown]

Sorry about that…I posted the code for the spinner check…here is the MakeModelCheck. I believe that line 80 is going to be right near the bottom: If(value.toUpperCase() != rowValue.toUpperCase()) {

package dwKeywords

import static com.kms.katalon.core.checkpoint.CheckpointFactory.findCheckpoint
import static com.kms.katalon.core.testcase.TestCaseFactory.findTestCase
import static com.kms.katalon.core.testdata.TestDataFactory.findTestData
import static com.kms.katalon.core.testobject.ObjectRepository.findTestObject
import static com.kms.katalon.core.testobject.ObjectRepository.findWindowsObject
import com.kms.katalon.core.annotation.Keyword
import com.kms.katalon.core.checkpoint.Checkpoint
import com.kms.katalon.core.cucumber.keyword.CucumberBuiltinKeywords as CucumberKW
import com.kms.katalon.core.mobile.keyword.MobileBuiltInKeywords as Mobile
import com.kms.katalon.core.model.FailureHandling
import com.kms.katalon.core.testcase.TestCase
import com.kms.katalon.core.testdata.TestData
import com.kms.katalon.core.testng.keyword.TestNGBuiltinKeywords as TestNGKW
import com.kms.katalon.core.testobject.TestObject
import com.kms.katalon.core.webservice.keyword.WSBuiltInKeywords as WS
import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUI
import com.kms.katalon.core.webui.keyword.builtin.ConvertWebElementToTestObjectKeyword as ConvertWebElementToTestObjectKeyword
import com.kms.katalon.core.windows.keyword.WindowsBuiltinKeywords as Windows
import internal.GlobalVariable
import org.openqa.selenium.Keys as Keys

public class MakeModelCheck {
	@Keyword
	def verifyMakeModel(String reportName, String type) {
		String value = ''

		boolean checkPassed = true
		boolean reportOpened = true

		int columnNumber = 0

		WebUI.delay(2)

		def headerArray = WebUI.executeJavaScript('return document.querySelectorAll(\'#data-table thead th\')', null)

//		WebUI.delay(2)

		for (def val : headerArray) {
			String header = val.text

			if (header == type) {
				break
			} else {
				columnNumber++
			}
		}

//		WebUI.delay(2)

		if (type == 'Make') {
			value = WebUI.executeJavaScript('return document.querySelectorAll(\'#data-table tbody td\')[' + columnNumber.toString() + '].innerText', null)

//			WebUI.delay(2)

			WebUI.executeJavaScript('document.querySelectorAll(\'#data-table tbody td\')[' + columnNumber.toString() + '].children[0].click()', null)
		} else if (type == 'Model') {
			value = WebUI.executeJavaScript('return document.querySelectorAll(\'#data-table tbody td\')[' + columnNumber.toString() + '].innerText', null)

//			WebUI.delay(2)

			WebUI.executeJavaScript('document.querySelectorAll(\'#data-table tbody td\')[' + columnNumber.toString() + '].children[0].click()', null)
		}

//		if (!(reportOpened = CustomKeywords.'dwKeywords.WaitForSpinner.spinnerNotVisible'('#panel3 multi-comp div div datatable progress-bar div'))) {
//			WebUI.comment(reportName + ': ' + type + ' report did not open!')
//
//			assert false
//		}

		if (!(reportOpened = new dwKeywords.WaitForSpinner().spinnerNotVisible('#panel3 multi-comp div div datatable progress-bar div'))) {
			WebUI.comment(reportName + ': ' + type + ' report did not open!')

			assert false
		}

		columnNumber = 0

		headerArray = []

		headerArray = WebUI.executeJavaScript('return document.querySelectorAll(\'#data-table thead th\')', null)

//		WebUI.delay(2)

		for (def val : headerArray) {
			String header = val.text

			if (header == type) {
				break
			} else {
				columnNumber++
			}
		}

//		WebUI.delay(2)

		def rowArray = WebUI.executeJavaScript('return document.querySelectorAll(\'#data-table tbody tr td:nth-child(' + (columnNumber + 1).toString() + ')\')', null)

//		WebUI.delay(2)

		for (def row : rowArray) {
			String rowValue = row.text

			println rowValue

			if (value.toUpperCase()!= rowValue.toUpperCase()) {
				WebUI.comment(rowArray.toString())

				WebUI.comment(reportName + ': ' + type + ' report not all ' + type + 's match!')

				assert false
			}
		}

		return checkPassed
	}
}

As you can see here, I am capturing the elements in my tables header and looping through to determine which column holds Make or Model depending on what is passed in. This to me is why it is so baffling because I am using an executeJavaScript every time it is called, so to me, it should either find the element or return an element not found error.

As for you suggestion on the wait conditions, as you can see, I have them sprinkled throughout trying to find the right balance of speed vs. accuracy. I do not believe this would affect the stale elements issue, but the timing is causing issues with my results by capturing a record count or list of Makes/Models from the current table, after I click the link to open the next child table. It should be capturing that data from the child table.

[Brandon_Hein]

Using hard waits like WebUI.delay() is a cardinal sin of test automation, and will be the bane of your existence as an automation engineer. Never use them. The only reason to use a hard wait like that is to confirm that you have a timing issue, after which you need to implement a legitimate wait condition to handle it. (Also, I see that they are commented out, so I’m not sure that you are even waiting the 2 seconds like you’re intending to).

This is an art in it’s own right, but usually it involves waiting for some condition on the page to be true. If your application has a spinner or an overlay of some sort that appears when it’s loading, it’s easy enough to wait for that to disappear, indicating that the page is ready for further input. If you’re not that lucky, you’ll have to watch for specific elements and/or attributes in the DOM to appear/disappear, indicating that the page is in a static state. See the built-in WebUI.waitForElement___() family of methods.

Now, as for your specific issue, it violates point #1 that I made above. When you do something like this:

def rowArray = WebUI.executeJavaScript('return document.querySelectorAll(\'#data-table tbody tr td:nth-child(' + (columnNumber + 1).toString() + ')\')', null)

You get a reference to an array of elements, and you are reusing this reference multiple times. While you iterate through these elements and do work, your page has plenty of time to potentially reload, which would completely invalidate your list. There are two ways to mitigate this:

1.) Re-locate the list of elements after each iteration:

def rowArray = WebUI.executeJavaScript('return document.querySelectorAll(\'#data-table tbody tr td:nth-child(' + (columnNumber + 1).toString() + ')\')', null)
for (def row : rowArray) {
    String rowValue = row.text
    println rowValue
    if (value.toUpperCase()!= rowValue.toUpperCase()) {
        WebUI.comment(rowArray.toString())
        WebUI.comment(reportName + ': ' + type + ' report not all ' + type + 's match!')
        assert false
    }
    rowArray = WebUI.executeJavaScript('return document.querySelectorAll(\'#data-table tbody tr td:nth-child(' + (columnNumber + 1).toString() + ')\')', null)
}

2.) (Preferred method) Grab the size of the array once, then grab each row individually by index and get the text:

def size = WebUI.executeJavaScript('return document.querySelectorAll(\'#data-table tbody tr td:nth-child(' + (columnNumber + 1).toString() + ')\').length', null)
for (i = 0; i < size; i++) {
    def row = WebUI.executeJavaScript('return document.querySelectorAll(\'#data-table tbody tr td:nth-child(' + (columnNumber + 1).toString() + ')\').item(' + i + ')', null)
    String rowValue = row.text
    println rowValue
    if (value.toUpperCase()!= rowValue.toUpperCase()) {
        WebUI.comment(rowArray.toString())
        WebUI.comment(reportName + ': ' + type + ' report not all ' + type + 's match!')
        assert false
    }
}

(You should probably check my JavaScript here, I don’t normally use it for element location.)

[devin.brown]

Method 2 seems to have cleared up the stale element issue. Thank you so much for that!

[bookemwa]

I get Stale Element Reference not found when I schedule my test on TestOps.
When I execute locally the code runs successfully.
How can I resolve it?

[kazurayam]

@bookemwa

I get Stale Element Reference not found when I schedule my test on TestOps.
When I execute locally the code runs successfully.

This implies that your test case script is coded careless. You do not respect the mantra by Brandon_Hein.

@bookemwa

How can I resolve it?

Review his post and fix your test case script.

[bookemwa]

that’s interesting thought the same code that runs locally should be able to run in test ops when schedule will review post and see

[kazurayam]

It seems that you expect a single code set runs just the same on your local PC and remotely on TestOps. That is too much opportunistic.

If your test case code is not making defensive waits, it is quite likely that a single set of test performs differently on two environments.

Local machine and remote machine (TestOps server) are two different environments. They may have different version of software installed (especially browsers). They are hosted in different newtwork; possibly TestOps is hosted on a cloud environment with super-wide bandwidth so that queries to remote URLs will be responded far quicker than the local PC. They would have different size of memory, they would have different CPU. Your tests may perform differently on different environments in term of execution speed in mili-second order.

If your test case code is not making defensive waits, difference in a few milli-second order may cause various timing issues in selenium-based software. @Brandon_Hein suggested “Always wait for the page to be static prior to locating elements and using them”. His advice is golden.

[bookemwa]

that’s well understood and it not being opportunistic just trying to be objective and will check the code again

[thierry.flament]

Bit of a Katalon novice here but I have been struggling with this issue like many of us seemingly have.
Found a very simple solution that seems to be missing on these fora
First a bit of context. We use a standard vue datapicker. You can click on arrows to navigate, or on the month to go upward a level to replace the days with the months, then you can click on the year to see the years instead of the months.

The issue I was having is I could not interact with any element on the datepicker (which is hidden untill you click the datefield.).

The simple solution is to put each interaction of hidden elements in a separate test case and then to call that test case from the original test case.

Example:

What won’t work
Original test case: Select the date 2020/1/15

• WebUI.click(findTestObject(‘Page_Crescendo Evaluation/meeting/input_meeting_date’)) → PASS
• WebUI.click(findTestObject(‘Page_Crescendo Evaluation/meeting/date-picker/button_vdp_up’)) → FAIL
• WebUI.click(findTestObject(‘Page_Crescendo Evaluation/meeting/date-picker/button_januari’)) → FAIL
• WebUI.click(findTestObject(‘Page_Crescendo Evaluation/meeting/date-picker/button_15’)) → FAIL

What will work:
Original test case:

• WebUI.click(findTestObject(‘Page_Crescendo Evaluation/meeting/input_meeting_date’)) → PASS
• WebUI.callTestCase(findTestCase(‘Click on vdp_up’), [:], FailureHandling.STOP_ON_FAILURE) → PASS
• WebUI.callTestCase(findTestCase(‘Click on januari’), [:], FailureHandling.STOP_ON_FAILURE) → PASS
• WebUI.callTestCase(findTestCase(‘Click on day 15’), [:], FailureHandling.STOP_ON_FAILURE)

Whereby the called test cases are each 1 action in the datepicker, the very same action that didn’t work in the original test case.
Example: Click on vdp up

• WebUI.click(findTestObject(‘Page_Crescendo Evaluation/meeting/date-picker/button_vdp_up’))

Conclusion
If you need to interact with a datepicker. You need to use a fresh test case do get the DOM refreshed before each action on hidden elements. Given the limited impact on performance, I suggest you simply do it for each action on EVERY element of the datepicker. This way you are sure you will never get a stale element exception and it will run smoothly every time.

Small note on performance
I noticed almost no delay by calling separate testcases, but naturally, the more additional steps you take (such as find test case) the slower your testing will run. I found this to be absolutely negligible.

[Dan_Bown]

Logically, I don’t believe there is any difference between clicking a test object in one script and calling another test case and clicking the same test object. Have you determined whether inserting a wait or delay in your first example would give success? I would start with a delay of 1 or 2 seconds just to see whether it works and if so, proceed to find a smarter way to wait (like Wait for Element Clickable for example instead of a hardcoded delay). Just trying to cut down on your workload and improve readability!

[thierry.flament]

HI Dan.

I believe (but cannot confirm) the difference is that the DOM is retrieved at the beginning of the test case. The stale element exception is caused due to an element which you refer to, not being found. Given the fact the datepicker is not show untill you click on the datefield, it’s not on the DOM. It won’t appear on it unless you retrieve the DOM after the action that makes the element of your choosing appear. Consequently: introducing a wait period will not make the test work. (Yes I tried that initially.)

I assume one can retrieve the DOM manually during a test case using katalon. (For example after clicking a test object that makes your element appear). In that case you can do away with the separate calls. Doing so, however, is more technical than my solution which works perfectly every time so far and with good performance.

[thierry.flament]

UPDATE:

Re-tested today. The workaround no longer works. It worked PERFECTLY previously so I can only assume a change was made to Katalon

[mwarren04011990]

Give this a try

[thierry.flament]

That worked, Also got it back to work in the ‘workaround’ way by changing the identifiers (even though the old identifiers still worked for power automate)

In the meantime we changed the js-framework and version of my application has changed from vue.js to nuxt.js V3.
This resolved the issue where I need to call separate test cases.
Currently I’m able to use all the elements of the datepicker directly. I can confirm this has sped up the test to the point where I actually need to slow it down instead (was able to do it by verifying the toast messages appeard with level = success)

In conclusion (for others who are interested):

• Use relative x-path
• focus on immutable attributes or use ‘contains()’ instead of equal
• careful for test execution which is faster than the validation of your application. Don’t set a date & then verify that date but verify EACH STEP of setting that date before executing the check if the date has accurately been set
• to verify the date was accurate i had to fetch the value of a property

def attribute = WebUI.getAttribute(findTestObject('Object Repository/Module_Evalfed/meeting/form_element_meeting_date'), 
    'value')

def meetingDate = (((String.format('%02d', dayNumber) + '/') + String.format('%02d', monthNumber)) + '/') + yearNumber

assert attribute == meetingDate

And the identifier of form_element_meeting_date:

//div[@data-component='datepicker'][@data-meeting-date='datepicker']/div/div/input

Hope this can help some people out.