Ways to avoid having to sleep()/delay() your code

There are very few legitimate cases where you would want to delay() your test case, or put it to sleep()

Main legit use case for this, is to poll an API until you get some expected result from it…or to do some performance testing (namely to simulate an impatient user).

It is a code smell, because it:

  • is literally stopping the execution of your test case for some hard-coded amount of time, which means:
  • a lot of wasted time, or
  • situations where “it works on my machine (but not yours)!”
  • is a brute force solution to something where either a built-in waitForXXX() method exist, or some simple custom keyword can be written to solve the problem

Yet there are so many people using those methods because there seems to be no other choice…

How can we prevent these?

Let’s go look at some examples:

Examples of preventable uses of delay()/sleep()

Keep in mind, most if not all of these are based on what I’ve had to face in testing a Zoho app, and based on my complete overhauling of the Katalon project for that…

With that in mind, let’s take a look at the first use case I encountered when being tasked with that project…

Elements that are supposed to be on the page (especially after page load)

Bad way

The laziest way is to just do this:

WebUI.navigateTo(homePage);

WebUI.delay(5);

WebUI.click(findTestObject("Home page/Dashboard item"));

Better way

This is better way to do it…

WebUI.navigateTo(homePage);

final TestObject loader = findTestObject("Home page/loader");

WebUI.waitForElementVisible(loader, 2, FailureHandling.OPTIONAL);

WebUI.waitForElementNotVisible(loader, 5); 

final TestObject dashboardItem = findTestObject("Home page/Dashboard item");

WebUI.waitForElementVisible(dashboardItem, 2);

WebUI.click(dashboardItem);

Custom widgets

Auto-complete

In Zoho (namely CRM), an auto-complete has three major parts:

  • text field
  • loading icon
  • dropdown list
Bad way

A noob may handle an autocomplete like:

WebUI.sendKeys(findTestObject("Page/My Autocomplete/input field"), "some text");

WebUI.delay(3); // DON'T DO THIS, PLEASE!!

WebUI.click(findTestObject("Page/My Autocomplete/first dropdown option"));

Heaven forbid it be a more complicated use case, where the autocomplete be fetch-on-scroll, and the dropdown option you’re looking for be on the nth page, for n > 1.

Then what will you do? Some scroll, then some hard-coded delay(), inside for loop?

Better way

This strategy-based way involves a lot more code, but is way more correct:

We’re going to create Custom Keyword for it!

public final class GeneralWebUIUtils { 
    public static void HandleAutoComplete(TestObject textField, String input, TestObject loader, TestObject dropdownOption, BaseSelectionStrategy strategy = null) throws StepFailedException {
        WebUI.click(textField)

        WebUI.sendKeys(textField, input)

        TimeLoggerUtil.LogAction({
            return WebUI.waitForElementNotVisible(loader, 3)
        },
        "Loader",
        "disappear");

        TimeLoggerUtil.LogAction({
            return WebUI.waitForElementPresent(dropdownOption, 3, FailureHandling.STOP_ON_FAILURE);
        },
        "Dropdown option",
        "become present")

        BaseSelectionStrategy selectionStrategy = strategy;
        if (strategy == null)
            selectionStrategy = new BaseSelectionStrategy(input);

        selectionStrategy.doSelect(dropdownOption);
    }

    public static boolean WaitForElementCondition(Closure<Boolean> onCheckCondition, Closure onContinue, TestObject to, int timeOut, FailureHandling failureHandling = FailureHandling.STOP_ON_FAILURE) {
        final long startTime = System.currentTimeMillis()
        boolean isConditionSatisfied = false;
        while ((System.currentTimeMillis() < startTime + timeOut * 1000) && (!isConditionSatisfied)) {
            isConditionSatisfied = WebUI.waitForElementPresent(to, 1, failureHandling) && onCheckCondition(to);
            if (onContinue != null)
                onContinue(isConditionSatisfied, to);
        }
        if ((!isConditionSatisfied) && (failureHandling.equals(FailureHandling.STOP_ON_FAILURE))) {
            KeywordUtil.markFailedAndStop("Condition for TestObject '${to.getObjectId()}' not met after ${(System.currentTimeMillis() - startTime) / 1000} seconds");
        }
        return isConditionSatisfied;
    }

    public static boolean WaitForElementHasText(TestObject to, String expectedText, int timeOut, FailureHandling failureHandling = FailureHandling.STOP_ON_FAILURE) {
        return this.WaitForElementCondition({ TestObject testObj ->
            return WebUI.getText(testObj).contains(expectedText);
        }, { boolean success, TestObject testObj ->
            if (!success) {
                WebUI.waitForElementNotPresent(testObj, 1, FailureHandling.OPTIONAL);

                WebUI.waitForElementPresent(testObj, timeOut);
            }
        },
        to,
        timeOut,
        failureHandling);
    }
}

public class BaseSelectionStrategy {
	protected final String input;

	public BaseSelectionStrategy(String input) {
		this.input = input;
	}

	public void doSelect(TestObject dropdownOption) {
		this.waitForDropdownOption(dropdownOption);
		WebUI.click(dropdownOption);
	}

	public boolean waitForDropdownOption(TestObject dropdownOption, FailureHandling failureHandling = FailureHandling.STOP_ON_FAILURE) {
		return GeneralWebUIUtils.WaitForElementHasText(dropdownOption, input, this.getWaitTime(), failureHandling);
	}

	public int getWaitTime() {
		return 5;
	}
}

// this is only here to allow time-logging of an action
public final class TimeLoggerUtil { 
	public static boolean LogAction(Closure<Boolean> onAction, String elementDesc, String expectationDesc) throws StepFailedException {
		final long startTime = System.currentTimeMillis();

		try {
			if (onAction()) {
				KeywordUtil.logInfo("${elementDesc} took ${(System.currentTimeMillis() - startTime) / 1000} seconds to ${expectationDesc}");
				return true;
			} else {
				KeywordUtil.markWarning("${elementDesc} didn't ${expectationDesc} after ${(System.currentTimeMillis() - startTime) / 1000} seconds");
			}
		} catch (StepFailedException ex) {
			KeywordUtil.markFailedAndStop("${elementDesc} didn't ${expectationDesc} after ${(System.currentTimeMillis() - startTime) / 1000} seconds...\n${ex.getMessage()}");
			throw ex;
		}

		return false;
	}
}

Then, to use it, simply be like:

GeneralWebUIUtils.HandleAutoComplete(
	findTestObject("Page/My Autocomplete/input field"),
	"some text", // or whatever
	findTestObject("Page/My Autocomplete/loader"),
	findTestObject("Page/My Autocomplete/first dropdown option"),
);

I will admit, this is a lot more code, but is way more of a robust solution as it will not hang your test case up!

What, your autocomplete is fetch-on-scroll? No problem!

Simply create a BaseScrollableSelectionStrategy (and an ActionHandler):

@InheritConstructors
// feel free to extend this class as you see fit
public class BaseScrollableSelectionStrategy extends BaseSelectionStrategy {

	@Override
	public void doSelect(TestObject dropdownOption) {
		final long startTime = System.currentTimeMillis();

		ActionHandler.Handle({
			this.waitForDropdownOption(dropdownOption);
		}, { boolean success, _ ->
			KeywordUtil.logInfo("${dropdownOption.getObjectId()} ${this.getActionStatus(success, startTime)}")
			if (!success) {
				final TestObject lastAvailableDropdownItem = new TestObject("Last available dropdown item")
						.addProperty("xpath",
						ConditionType.EQUALS,
						"//lyte-drop-box[not(contains(concat(' ', @class, ' '), ' lyteDropdownHidden '))]//lyte-drop-item[last()]"); // TODO: change this to the XPath for the last dropdown option on your auto complete drop-down

				TimeLoggerUtil.LogAction({
					if (!WebUI.waitForElementPresent(lastAvailableDropdownItem, 2))
						return false;

					GeneralWebUIUtils.ScrollDropdownOptionIntoView(lastAvailableDropdownItem);

					return true;
				}, lastAvailableDropdownItem.getObjectId(),
				"scroll into view");
			}
		}, 15)
		GeneralWebUIUtils.ScrollDropdownOptionIntoView(dropdownOption);
		WebUI.click(dropdownOption);
	}

	protected String getActionStatus(boolean success, long startTime) {
		if (success)
			return "took ${(System.currentTimeMillis() - startTime) / 1000} seconds to show up";
		return "didn't show up after ${(System.currentTimeMillis() - startTime) / 1000} seconds";
	}

	@Override
	public int getWaitTime() {
		return 2;
	}
}

public class ActionHandler {
    public static void Handle(Closure onAction, Closure onDone, long timeOut) {
        long startTime = System.currentTimeSeconds();
        while (System.currentTimeSeconds() < startTime + timeOut) {
            try {
                onDone(true, onAction());
                return;
            } catch (Exception ex) {
                onDone(false, ex);
            }
        }
    }
}

add a keyword for scrolling the dropdown option into view:

public static void ScrollDropdownOptionIntoView(TestObject to) {
    WebUI.executeJavaScript("arguments[0].scrollIntoView({block: 'center'})", [WebUiCommonHelper.findWebElement(to, 3)])

    WebUI.waitForElementVisible(to, 2)
}

and then your use case should look like:

final String textFieldInput = "some text" // or whatever

GeneralWebUIUtils.HandleAutoComplete(
	findTestObject("Page/My Autocomplete/input field"),
	textFieldInput,
	findTestObject("Page/My Autocomplete/loader"),
	findTestObject("Page/My Autocomplete/first dropdown option"),
	new BaseScrollableSelectionStrategy(textFieldInput), // or whatever your derived scrollable selection strategy is
);

New Table Row

This is yet another unjustified use case that I had to deal with, this one much earlier on in the project. It was something like this:

Bad way
WebUI.click(findTestObject("Page/Add Button"));

WebUI.delay(3);

WebUI.click(findTestObject("Page/Second Row Dropdown button"));
Better way
WebUI.click(findTestObject("Page/Add Button"));

final TestObject secondRowDropdownBtn = findTestObject("Page/Second Row Dropdown button");

WebUI.waitForElementPresent(secondRowDropdownBtn, 3);

WebUI.scrollToElement(secondRowDropdownBtn, 2);

WebUI.click(secondRowDropdownBtn);

…or better yet…

WebUI.click(findTestObject("Page/Add Button"));

final TestObject secondRowDropdown = findTestObject("Page/Second Row Dropdown button");

WebUI.waitForElementPresent(secondRowDropdown, 3);

WebUI.scrollToElement(secondRowDropdown, 2);

WebUI.click(findTestObject("Page/Second Row Dropdown button"));

Saving a model to be created, and then doing/checking something on the next page

This is a common one. You are about to click “Submit”, to create some record (item/rate card/discount/member/…)…

You wait for the button to disable…

…then you try that WebUI.waitForPageLoad(15); between waiting for the button to disappear, and waiting for the next page to fully load…

Test case fails! WHY.png

Bad way

You pull your hair out in frustration, and then give into that temptation to instead do WebUI.delay(15)

It works, but it’s slow?!

You now have to run that test case to create 10+ items/rate cards/discounts/members/… per a data file…

AnyMinuteNow.gif

Better way

This way handles both the save button and the URL change. It will require more custom keywords to our GeneralWebUIUtils:

public final class GeneralWebUIUtils { 
    private static boolean WaitForURLCondition(Closure<Boolean> onCheckCondition, int timeOut, Closure<String> onErrorMessage, FailureHandling failureHandling = FailureHandling.STOP_ON_FAILURE) {
        final long startTime = System.currentTimeMillis()
        boolean isConditionSatisfied = false;
        while ((System.currentTimeMillis() < startTime + timeOut * 1000) && (!isConditionSatisfied)) {
            isConditionSatisfied = onCheckCondition(WebUI.getUrl())
        }
        if ((!isConditionSatisfied) && (failureHandling.equals(FailureHandling.STOP_ON_FAILURE))) {
            KeywordUtil.markFailedAndStop("${onErrorMessage(WebUI.getUrl())} after ${(System.currentTimeMillis() - startTime) / 1000} seconds");
        }
        return isConditionSatisfied;
    }

    public static boolean WaitForURLNotEquals(String url, int timeOut, FailureHandling failureHandling = FailureHandling.STOP_ON_FAILURE) {
        return this.WaitForURLCondition({ String browserURL ->
            return !(browserURL =~ SMDStringUtils.GetURLPattern(url)).matches();
        },
        timeOut, { String browserURL ->
            "URL '${browserURL}' matches unexpected '${url}'"
        },
        failureHandling)
    }

    public static boolean WaitForURLEquals(String url, int timeOut, FailureHandling failureHandling = FailureHandling.STOP_ON_FAILURE) {
        return this.WaitForURLCondition({ String browserURL ->
            return (browserURL =~ SMDStringUtils.GetURLPattern(url)).matches();
        },
        timeOut, { String browserURL ->
            "URL '${browserURL}' does not match expected '${url}'"
        },
        failureHandling)
    }

public static void HandleSaveButton(TestObject saveButton) throws StepFailedException {
        this.HandleSaveButton(saveButton, true);
    }

    public static void HandleSaveButton(TestObject saveButton, boolean shouldSuccessfullySave) throws StepFailedException {
        WebUI.scrollToElement(saveButton, 3)

        TimeLoggerUtil.LogAction({
            return WebUI.waitForElementClickable(saveButton, 3, FailureHandling.STOP_ON_FAILURE);
        },
        "Save button",
        "become clickable");

        WebUI.click(saveButton)

        TimeLoggerUtil.LogAction({
            return WebUI.waitForElementHasAttribute(saveButton, GeneralWebUIUtils.DISABLED, 5, FailureHandling.STOP_ON_FAILURE);
        },
        "Save button",
        "disable");

        if (shouldSuccessfullySave) {
            TimeLoggerUtil.LogAction({
                return WebUI.waitForElementNotPresent(saveButton, 5, FailureHandling.STOP_ON_FAILURE);
            },
            "Save button",
            "disappear from the DOM");

            TimeLoggerUtil.LogAction({
                WebUI.waitForPageLoad(5);
                return true;
            },
            "Page",
            "load")
        }
    }
}

public final class SMDStringUtils { 
	public static String GetURLPattern(String url) {
		return (/^(http(s)?\:\/\/)?${url}(\/)?$/);
	}
}

In the test case, we just do:

// submit the changes
GeneralWebUIUtils.HandleSaveButton(findTestObject('Page/Submit button'))

// verify that the save happened successfully
GeneralWebUIUtils.WaitForURLNotEquals(creationPageURL, 15) // creationPageURL is the URL of the creation page you were just on

or better yet, if you had to click some “Add” button on some initial page, and submit button is supposed to end up sending you back to that initial page:

// submit the changes
GeneralWebUIUtils.HandleSaveButton(findTestObject('Page/Submit button'))

// verify that the save happened successfully
GeneralWebUIUtils.WaitForURLEquals(initPageURL, 15) // initPageURL is the URL of the initial page you were just on

Thoughts? Concerns?

Let me know in the comments section below!

Just one. :sunglasses:

My concern involves your strategy to wait for (and track) visibility/invisibility of an element across (effectively) two lines of code. This approach, when successful, can lead you to think you’ve nailed the transient nature of an element allowing your test code (your “robot”) to “walk beside” the AUT as it progresses. This is unnecessary and is prone to flakiness.

The problem arises when…

  1. The network your AUT is running on changes (gets faster, in this case). Your code misses the appearance (because the underlying XHR/Promise is satisfied super quickly) and fails the test.

  2. Any evolution of the AUT that modifies the perceived behavior leaves your “robot” confused as to what’s going on, since your code told him to wait when there’s nothing to wait for.

For any transition from one page state to another, it is usually (always?) better to wait for outcomes. And if you think about it, that’s exactly what real human users do: they see the change they’re looking for (e.g. the list populates) and then respond with their next action.

While we both agree that a fixed delay is a poor TC design strategy (for all the reasons you highlighted), it is more robust than a flaky approach which simply hasn’t revealed itself yet.

It certainly sounds like you’ve developed a pretty comprehensive library of utilities. You might want to refine it further and incorporate the outcomes approach where you can.

therefore, use a random delay.
or simulate a ‘patience level’ as an acceptance criteria.
in the end, this is what a real user will do, wait for the element to be available.
some are patient, some are not
i doubt any real user will open the devtools, check if a certain resource is loaded before to click somewhere else and so on.
the user don’t care about code smell. the user wait for a while, if the page is not loaded it may hit refresh or it may close the page and forget about.
the user dont care about networking issues, how drunk was the developper or what outages are in progress
the user need the content to be available in a decent amount of time.
eventualy, the user mai wait more than usual if he is informed that a certain expensive operation is in progress but not for very long time… (and such can be captured if implemented)
expensive operations should happen async on the AUT side.
period!

Yea, that would be great for performance/reliability testing, but how many testing teams have the capacity for that type of testing, let alone do it any more?

The aim of most test cases, from my limited experience, seem to be positive/negative regression testing…

Well … been there in the past.
But I find some time and I opened some discussions with the PM/PO/BA to decide what will be an acceptable time for a certain page to load.
At that time, I was in charge also with API’s load testing, so my voice had some weight.
(and I was the only one allowed to do it with as many threads as possible because I was capable to restore the servers in case they crash :smiley: )

I understand the pressure of testing with limited resources (like in human resources) but everything should have a start.
The more relevant are the tests, the better will be the AUT quality.
Extending the scope of testing will bring just benefits.
Your proposed solutions are nice, from coding elegance point of view, but are not all the time mimiking user expectations.

Whenever possible, I will design a certain poll/retry function with an expiration time.
If the resource is loaded faster, fine, if the expire time is reached, the test should fail.
The acceptance criteria has to be agreed with BA and development team.

e.g this one seems to be nice, if i understand the code correctly:

while ((System.currentTimeMillis() < startTime + timeOut * 1000) && (!isConditionSatisfied))