There are many test suites in my Test Suite Collection. When I’m executing this Suite Collection on Test Cloud all individual test suites are getting executed but few of them are not included in the Test Summary report.
As a result of this I’m not able to identify which exact test cases have failed and having a incorrect status reporting.
Anyone else facing the same issue?
1 Like
The issue where test cases execute successfully but are missing from the Test Summary report in Katalon Studio is typically caused by configuration or reporting setup issues. Here’s how to resolve it:
1. Verify Test Suite Collection Configuration
- Check Test Suite Inclusion:
- Open your Test Suite Collection.
- Ensure all individual Test Suites are explicitly added to the collection.
- Avoid manually editing the generated script (e.g., in
testCaseCollections/.../Scripts/groovy); use Katalon’s GUI to configure inclusions.
- Example:
groovy
// WRONG: Manually writing scripts may skip reporting
// CORRECT: Use "Add" button in Katalon GUI to include Test Suites
2. Configure Report Settings
- Enable Report Generation:
- Go to Project Settings > Report > Enable Test Execution Report.
- Ensure Export Test Execution Report is checked (for HTML/PDF formats).
- Output Folder Permissions:
- Check if Katalon has write access to the
Reportsfolder (e.g., no admin restrictions or read-only mode).
- Check if Katalon has write access to the
3. Avoid Premature Termination
- If tests finish too quickly (e.g., due to
System.exitorRuntime.getRuntime().halt), the report may not generate. Ensure graceful termination:
groovy
// Use Katalon's built-in keywords to mark test status
if (testFailed) {
KeywordUtil.markFailed("Test failed reason")
}
4. Check for Corrupted Executions
- Rebuild the Project:
- Go to File > Clean Cache and Restart.
- Delete Old Reports:
- Manually delete the
Reportsfolder to force Katalon to regenerate it.
- Manually delete the
5. Test Case Structure Issues
- Avoid Nested Test Suites:
- Ensure Test Suites in the collection are flat (not nested within each other).
- Explicitly Mark Test Results:
- Use
KeywordUtil.markPassed()orKeywordUtil.markFailed()in scripts to ensure results are recorded.
- Use
6. Katalon Version Compatibility
- Update Katalon:
- Older versions (e.g., 7.x) may have reporting bugs. Upgrade to the latest version (e.g., 9.x).
- Check Plugins:
- Conflicting plugins (e.g., custom report generators) might interfere. Disable plugins temporarily.
7. Analyze Logs for Errors
- Console Logs:
- Look for errors like
Failed to generate reportin the Execution Log.
- Look for errors like
- Internal Logs:
- Check
.metadata/.logsin your Katalon project folder for exceptions during report generation.
- Check
8. Use Unique Report Names
- Avoid overwriting reports by appending timestamps:
groovy
import com.kms.katalon.core.configuration.RunConfiguration
// Set unique report folder
RunConfiguration.setReportFolderName("Report_${new Date().format('yyyyMMdd_HHmmss')}")
Common Fixes (Prioritized)
- Reconfigure Test Suite Collection via Katalon GUI (not scripts).
- Enable/Reinstall Report Settings in Project Settings.
- Update Katalon Studio to the latest version.
- Avoid manual termination of tests; let Katalon handle teardown.
I forgot to mention that this issue is while executing the test cases on Test Cloud. Updating the same in main thread
Hi @akshay_sartabe,
Thank you for sharing your issue. I will ask my team about your case and back here soon.
Hi @akshay_sartabe,
Are you running the test with TestCloud on TestOps? Please give us the test run ID and your email for further investigation. Thank you.
@Elly_Tran @ha.tpham
I have created a ticket for this and they have said it is a bug in Katalon.
@ha.tpham Yes I’m running test cases with TestCloud on TestOps
@akshay_sartabe Oh, If so, please follow the ticket and let me know if you need further support. Thank you.
Yeah, I do need support to fast-track to get the bug fixed as it has around 2 months now and no resolution on it.
Hi @akshay_sartabe
Absolutely. I’m initiating an internal cross-functional check on your ticket to expedite the investigation. Apologies for any inconvenience.
@ha.tpham @Elly_Tran
I need you help here. The team have closed the ticket saying it is fixed in the 10.2.1-rc5 but I not able to see this anywhere while scheduling the test run.
Below is the screenshot of reply on my ticket and now that it is closed there is no way for me to communicate there.
Hi @akshay_sartabe,
Please try to clear cache and recheck since I can see the version 10.2.1-rc5 on my account.
Besides, ‘latest’ tag and ‘latest 10.x’ are targeting 10.2.1-rc5 as well.
1 Like