Looking for tools to measure code coverage

Can anyone recommend a tool that will measure the code coverage of my Katalon tests? Can a tool like Clover be used with Katalon?

Appreciate any suggestions.
Thanks!
D

Code coverage make sense with unit testing.
Katalon is designed for functional testing, e.g ‘blacbox test’, trying to be abstract wrt AUT code

I’m looking at it from the QA side of things. We use Katalon extensively to test our web application - all user scenario based test cases. The executive team always questions the code coverage of our tests… As a Manager, I would love to be able to provide that for them let alone have a way to truly evaluate our automated test suite(s) not so much the application.

well … the executive team is just executive, they use concepts with no sense from functional testing point of view :slight_smile:

Coverability, with functional testing, is given by how many test scenario’s you have for a certain feature/module: positive workflows (happy path), negative scenarios, boundary tests, data driven tests and so on.
Also, by how many suite types: new feature/module test, smoke, regression, integration tests, end-to-end etc.

code coverage is the measurement of how many line of code the unit test covered (e.g, how many method involved in the unit test code). And the test that Katalon executed is called E2E testing that test on the system (the black box) itself, not the code. So there is no way that you can calculate the coverage percent in traditional way.

But you might, in the worst case that the executive team insist on having the coverage number, you can define new way to calculate the coverage by how many module that’s been tested by your automation suites.

1 Like

How about coverage from Katalon itself to make sure when we run the full regression we don’t have dormant test code that is not being executed … this will help in cleaning the test project.

@nhussein

i think the test report generated after one execution it is showing the total number of executed testcases. compare this with the amount of existing ones.

on the other side, even with unit testing, there is no way to tell if some code is dormant or not.
the coverage report will only tell how many line of codes are covered by the written tests, but not how many lines of codes are actually used in the running app (e.g you can have classes in certain packages, which have unit tests giving a 100 % coverage … but those classes are not imported in the main code)

to detect dormant code, usually it is a job for linting, but that it depends on many factors: what is the app language, what is the linter in use, if will detect unused classes or only imported but not used ones, and so on.
I don’t think Katalon have such feature.

L.E. In fact i faced a similar problem (for python but does not matter, should work for groovy/java too): how can i tell if a certain class it is in use by some test scripts.
so i simply used grep to search for the class name in all scripts, if it shows nothing, then it is safe to mark it as deprecated/remove it. But, if the class it is in use, will not tell you how many methods from the given class are actually used in the scripts.
here you may expand the method to collect also all defined methods names and search for them, but may not be accurate (you can have more than one class with methods sharing same name, so you have to detect also how the class is imported)
Also, will not tell if the package containing the class it is used in other projects … so take care about the workflow of your entire team (e.g. some people may use shared packages for custom keywords)
And again, the class may be imported but no method belonging to it used … here it is only a matter of good practice when designing testcases. Regular use of Ctrl_Sfift_O will get rid of the unused imports (for python we use flake8 in a bamboo job, if an import is unused, the linter will “scream” and mark the verification build as failed when the code is pushed to repository, so the PR cannot be merged until the issue is fixed)

So, read again the topic from the beginning and find another way to define the coverability, to suit better to your needs … or have a brainstorming with your bosses and the dev/devops teams to search for alternate ways to ensure the code is clean.

@bionel what I meant is having something like this:
https://www.jetbrains.com/help/idea/running-test-with-coverage.html

@nhussein

as already explained, this is not applicable for katalon, but for unit tests (tests written in junit, testng etc, which are ‘glued’ with the application code).
see: https://www.jetbrains.com/help/idea/testing.html

and here some intro into the terms: https://dev.to/chrisvasqm/introduction-to-unit-testing-with-java-2544

Katalon is a tool for blackbox testing (End-to-End)

If you wish, you can write unit tests for your katalon project if you have a lot of custom keyword classes … i think there are few discussions on the forum on ‘how to use junit with katalon’ … but what will be the relevance of such report? Will only double the work of the QA team with no added value for the AUT.

Functional tests should be kept as simple as possible. The ‘coverage’ term as in ‘code coverage’ is not applicable here, the main indicator for this type of testing is given by how many test scenarios you have for a single functionality (e.g. a ‘Login’ page, or an API endpoint): positive tests, negative tests, boundary if applicable and so on …

if you are looking for api coverage, here is one tool that could suffice.
https://support.smartbear.com/readyapi/docs/testing/coverage/index.html

Is there a Katalon equivalent?