False positives when test is run in a suite or automatically

I have a test that’s in my testing suite that has been passing when it’s run with the rest of the suite, but when I go to look at the original webpage the error is still present. Currently I am using Task Scheduler to launch the testing suite in the morning.

The test fails (as it should) when it’s run manually, whether I run the suite manually or the test separately by itself.

Is there something I need to adjust on my end, or is this a Katalon issue?


can you share your test (at least the important part) with us? How do you fail the test?

import com.kms.katalon.core.checkpoint.Checkpoint as Checkpoint
import com.kms.katalon.core.cucumber.keyword.CucumberBuiltinKeywords as CucumberKW
import com.kms.katalon.core.mobile.keyword.MobileBuiltInKeywords as Mobile
import com.kms.katalon.core.model.FailureHandling as FailureHandling
import com.kms.katalon.core.testcase.TestCase as TestCase
import com.kms.katalon.core.testdata.TestData as TestData
import com.kms.katalon.core.testobject.TestObject as TestObject
import com.kms.katalon.core.webservice.keyword.WSBuiltInKeywords as WS
import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUI
import internal.GlobalVariable as GlobalVariable
import static com.kms.katalon.core.checkpoint.CheckpointFactory.findCheckpoint
import static com.kms.katalon.core.testcase.TestCaseFactory.findTestCase
import static com.kms.katalon.core.testdata.TestDataFactory.findTestData
import static com.kms.katalon.core.testobject.ObjectRepository.findTestObject
import com.kms.katalon.core.logging.KeywordLogger as KeywordLogger
import org.openqa.selenium.WebElement as WebElement
import com.kms.katalon.core.webui.common.WebUiCommonHelper as WebUiCommonHelper
import com.kms.katalon.core.util.KeywordUtil as KeywordUtil





List<WebElement> links1 = WebUiCommonHelper.findWebElements(links, 30)

List<WebElement> linkTexts = new ArrayList<String>()

for (WebElement link : links1) {

Set<String> uniqueLinkTexts = new HashSet<String>()

boolean isUnique

boolean passes = true

for (String s : linkTexts) {
    isUnique = uniqueLinkTexts.add(s)

    if (!(isUnique)) {
        passes = false

        KeywordUtil.markWarning('Duplicate found. Link: ' + s)

assert passes

Basically, the test is supposed to fail right now, and the console should display the links that are duplicated.

When launched automatically, it passes.

That’s weird. Just a guess, but try to change assert statement like this:

if(passes) {
} else {
	KeywordUtil.markFailed("Not OK")
1 Like

Strangely, the same thing happened…it passed (falsely) on an automatic launch but failed (as it should have) when I launched it manually.

And is there markPassed() message in the test log?