TestOps Simple Report Required

Am I the only person who finds TestOps confusing to use and not very helpful? I’ve really tried to like it, I promise.
As an example, my manager asked if I can provide a very basic statistic… how many times has each test case been executed. I was certain that this must be possible through TestOps, but if it IS there, I’m not navigating my way through the labyrinth of menus correctly.
Related to this (and perhaps one of the reasons I’m so confused here) is that we use a mixture of floating and seat licenses, and these (for some reason) have to be in separate organizations, so floating licenses are set up under our main organization, but the couple of seat licenses we have are under an “admin” organization. At first it wasn’t even obvious how to switch from one to another in TestOps, until I discovered that clicking on the TestOps log in the top left then allows me to select which Organization I want to view.
I just kicked off a test suite execution using an Azure DevOps pipeline, which runs my test suite remotely (on an ADO “Agent” machine) with the KRE. I don’t see any details of this execution in TestOps under the main organization or the admin organization. Do I need to do anything special (perhaps in the KRE command line) to tell it to upload results? In the command line output it says “Report has been sent to Katalon TestOps”… but I think it’s lying! Any words of wisdom please?

Hi, I am also looking for an answer to this same question. Anybody have any tips? All of our licenses are node-locked, for what its worth, but I am unable to get reports from tests run through Azure DevOps to show up in TestOps. Integration has been turned on and it works fine when run through katalon studio.

1 Like

I know this is ancient, but I too have been trying to figure this out. When you generate the command line in Katalon you can choose the organization and that will add an -orgID to the command line which I am using. I run the pipeline in ADO and nothing ever shows in TestOps.

1 Like

I must admit, I ended up doing everything I needed in Azure DevOps eventually, and gave up on TestOps. I wrote some code to output a more comprehensive junit xml report, which ADO reads and displays a correct number of passes and failures, then created a dashboard in ADO to show a history of test executions.
I also created code to output the results I needed into a summary email that gets sent to whoever kicks off the test pipeline (remote Katalon execution), with links to screenshots in ADO. Took a few days of fiddling, but got what I wanted in the end.

That’s great. I suppose that your achievement would interest many others who work with Katalon on ADO. I hope you to share your code to help them. Preferably you would create a public GitHub repository and host a runnable project to demostrate your precious code. Then, please, share the URL of your GitHub repository here in the forum so that people can link to it by a single click.

Unfortunately our company doesn’t allow us to share code written here, otherwise I’d be happy to. Also the code relies on other components of our home-grown framework, so it would require a lot of tweaking to work for anyone else. I’m always happy to answer questions about the general approach though!

I absolutely cannot believe how hard it has been to try and get my ADO results into TestOps. Just blows me away. Should not be that difficult.

I agree with you. Maybe the results are in TestOps somewhere and we cannot find them! I don’t think that’s the case though.

While I can’t share code written for the company, I would like to at least share the basics to help anyone that would like to create a report in ADO.

The first step is to have your Katalon tests output a JUnit report in XML format. This can be done anyway that you are comfortable with, but I wrote one function that creates the XML file (which I call “JUnit_Output.xml”), with the test suite name, start date/time, duration and machine name, and then another function that each testcase calls to insert into it the testcase’s name, duration, and (in the case of a failure) failure message. Here’s an example to follow…

<testsuites>
  <testsuite name="My Test Suite" timestamp="2024-02-19 17:43:34" time="114.00" hostname="My Machine Name">
    <testcase name="TestCase 1" time="4.00">
    </testcase>
    <testcase name="TestCase 2" time="60.00">
      <failure message="Something went wrong here"></failure>
    </testcase>
    <testcase name="TestCase 3" time="50.00">
      <failure message="The wheels fell off"></failure>
    </testcase>
  </testsuite>
</testsuites>

The next step is to edit your test pipeline (YML) in ADO, by adding the following task to read the Junit file and publish the results…

   - task: PublishTestResults@2
     condition: always()
     inputs:
       testResultsFormat: 'JUnit'
       testResultsFiles: '**/JUnit_Output.xml'
       searchFolder: '$(Agent.BuildDirectory)'
       failTaskOnFailedTests: false
       testRunTitle: '$(testsuitepath)'

Hope this helps someone!

1 Like