Accessibility Testing in Katalon Studio with axe-core – More Issues Detected After Initial Fixes

Product Forums Katalon Studio
, ,
1

Hi everyone,

I’ve been using Katalon Studio to perform web accessibility testing, specifically leveraging the axe-core library for scanning and identifying WCAG-related issues.

During the testing process, I noticed something puzzling: after identifying and fixing the initial set of accessibility issues reported by axe-core, subsequent scans revealed additional issues that weren’t flagged in the first run. It seems like once some elements were fixed or remediated, other hidden or deeper-level issues started surfacing in the reports.

This raises a couple of questions:

Is this behavior expected from axe-core, where only a subset of issues is shown initially?

Could it be that certain accessibility violations are only detectable after specific layers of the DOM are corrected?

Most importantly, is there a way to configure the axe-core integration in Katalon so that it detects all possible accessibility issues in a single pass, rather than uncovering them incrementally?

I’m trying to streamline the accessibility testing process as much as possible and would like to avoid the back-and-forth of multiple test cycles just to reveal the full scope of violations.

If anyone has faced this behavior or has recommendations on configuration options, best practices, or even workarounds within Katalon Studio or axe-core, I’d really appreciate your insights.

Thanks in advance!

1 Like
2

The behavior you’re observing with axe-core in Katalon Studio is a common aspect of automated accessibility testing. Here’s a structured breakdown of why this happens and how to address it:

1. Why New Issues Appear After Fixes

  • DOM Dependency: Some accessibility issues are only detectable once parent/child elements are corrected. For example:
    • Fixing semantic HTML (e.g., adding role="navigation") may expose issues in nested elements.
    • Resolving focus management might reveal keyboard traps in previously inaccessible sections.
  • Dynamic Content: If fixes involve interactive elements (e.g., opening modals), subsequent scans may analyze newly visible content.
  • Contextual Rules: Certain axe-core rules (e.g., aria-hidden-focus) only trigger when specific conditions are met (e.g., after ARIA attributes are fixed).

2. Configuring axe-core for Comprehensive Scans

To maximize issue detection in a single pass:

a. Enable All Rules

By default, axe-core runs only “best-practice” rules. Override this in Katalon to include all WCAG standards:

import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUI
import com.deque.html.axecore.selenium.AxeBuilder

AxeBuilder axe = new AxeBuilder()
// Configure to run ALL rules (including WCAG 2.1 AA/AAA)
axe.configure().disableIframeTesting().withRules("all").analyze(WebUI.getDriver())

b. Scan Full Page Context

Ensure the scan includes entire pages, iframes, and shadow DOM:

AxeBuilder axe = new AxeBuilder()
axe.include("body") // Scan entire page
axe.analyze(WebUI.getDriver())

c. Handle Dynamic Content

Wait for dynamic elements (e.g., lazy-loaded content) before scanning:

WebUI.waitForElementPresent(locator, 30)
axe.analyze(WebUI.getDriver())

3. Best Practices in Katalon Studio

  • Test Multiple States: Run scans after key user interactions (e.g., form submissions, modal openings):
WebUI.click(loginButton)
axe.analyze(WebUI.getDriver()) // Scan post-login state
  • Leverage @AfterTest Methods: Automate scans after each test case to catch state-specific issues.
  • Use axe-core Options:
    • Exclude non-relevant elements (e.g., third-party widgets):
axe.exclude("#external-widget")
  • Limit rules to WCAG 2.1 AA:
axe.withTags("wcag2a", "wcag2aa")

4. Why Incremental Discovery is Normal

  • Progressive Enhancement: Surface-level fixes (e.g., color contrast) often expose deeper structural issues (e.g., heading hierarchy).
  • Tool Limitations: axe-core detects ~57% of issues automatically; manual testing is still required for full coverage (per WebAIM research).
  • Prioritization: axe-core prioritizes severe issues first (e.g., missing labels over color contrast).

5. Workflow Optimization

  1. Baseline Scan: Run an initial scan with all rules enabled.
  2. Fix and Rescan: Address reported issues, then re-scan with the same configuration.
  3. Manual Testing: Supplement with manual checks for issues axe-core can’t detect (e.g., logical tab order).

Key Takeaways

  • No Silver Bullet: Automated tools like axe-core cannot catch all issues in one pass due to dependencies and dynamic content.
  • Configuration Matters: Adjust axe-core’s rules and context to match your app’s complexity.
  • Iterative Process: Accessibility testing is inherently iterative—plan for multiple test-fix cycles.

This approach balances automation with the reality of accessibility testing, ensuring you catch the majority of issues efficiently.

3

Bumping the post to let more experts join to support this post.

Happy Tom And Jerry GIF

4

Thanks for the detailed explanation — it really helped clarify a lot of things.

Just to share some updates from my side based on further testing:

  • My application includes an iframe, and I’ve noticed that when using disableIframeTesting(), axe-core completely skips issues inside the iframe. So for my case, I’ve avoided disabling it — but still, some issues within the iframe don’t seem to be picked up even when it’s enabled.
  • We’re already running scans with all rules enabled, so adding .withRules("all") or specific tags like wcag2aa didn’t make a noticeable difference in the number of issues reported.
  • I also tried scanning the full page content using axe.include("body"), but oddly enough, this is returning fewer issues than expected — especially compared to more targeted scans.
  • Even after including rules comprehensively and testing various states of the page, it still feels like axe-core doesn’t catch everything in a single pass. Some issues only surface after initial fixes are made — which supports the point you made about progressive discovery and DOM dependencies.

So at this point, it looks like I’ll need to continue with iterative scans post-fix, and possibly combine axe-core with some manual validation to close the gap.

Appreciate the insights again.