Integration with Azure DevOps Test Plans (Pre-release)


This is a companion discussion topic for the original entry at https://docs.katalon.com/katalon-studio/docs/azure-devops-test-plans.html

Hi,

First, congratulations to the team because of this new feature. That will be very helpful for us.

However, I implemented the steps according to the documentation and when I run the tests, in my Event Log appear this error:

Katalon TestOps: Unexpected response, URL: https://analytics.katalon.com/api/v1/organizations/setting?organizationId=194758, Status: 403, Response: {“error”:“access_denied”,“error_description”:“Either this resource does not exist, or the current user does not have permissions to access this resource”}

And the results will not be sent to Test Plan (in Azure). I guessed about the license, but, I have an Enterprise license active.

Important: I’m not using Katalon TestOps.

Can you help me?

Hi Sandro,

Thank you for your interest about our new feature.
To answer your problem, could you please check your Azure Devops Token is full scope access or not. Then send us your Execution Log if there is till error.

Have a nice day,

Hi, Huyen!

Same error! :frowning:

This behavior is strange because I can configure all settings (about my project in Azure), link the test cases between differents platforms (Azure and Katalon), but, the results aren’t sent by Katalon and I don’t know why.

Also, the PAT token is full access.

Important: I disabled the devops integration and the error persists. I think that my screenshot isn’t true.

So, @huyennguyen, to execute Katalon with Azure DevOps is necessary extra configuration than Integration with Azure DevOps Test Plans | Katalon Docs?

Thanks.

Hi Sandro,

Would you mind copying all Event Log and paste them here for me?

Thank you for your time.

2 Likes

Hi,

Here is it: https://pastebin.com/4jMcL7jW

Thank you!

Hi Sandro,

Your log reveals that there is no indication of Azure DevOps.
You can follow this video for instructions.
If your result still cannot send to Azure DevOps. Can you capture your Azure settings or it would be better if you can give me a video of how you configured and ran it.

Thank you.

2 Likes

Hi, Huyen!

Is working fine now. The error occurred because of myself… I was running the tests to test this feature separately, but, this feature works only in test suite collections. :smiley:

Thank you so much for your help!

2 Likes

Hi Team

I have the issue working after resolving the issue with fetching projects (see discussion: link). Big thank you Huyen for helping me out with that issue.

I have a question on how we could report on the automated test coverage. In Katalon we link a Katalon test case to a DevOps test case in the integration tab:
image

However if I look at that linked test case in Azure DevOps, I cannot find the relationship back to the Katalon test case. I.e. the tab with “Associated Automation” is blank


or the “Automation Status” remains unchanged.

Are their plans to make it visible in DevOps if a manual test case is asscociated to an automated test case?

Regards

Peter

Hi,

I have a use case where I need to run the same test multiple times. So to achieve this I added the test to a test suite and used a data file with 3 rows of different data. The first 2 rows have correct data but the third row causes a failure. So Katalon correctly displays the test results as 2 passed and 1 failed and the same results are propagated to TestOps as well. However on Azure I see that it shows all 3 tests failed and from the screenshot below it is clear that it is overwriting all test runs with the last one which failed and the execution times are also exactly same:

For comparison here is screenshot from TestOps

And from Katalon Studio

Can you please tell if this workflow is supported? And if it is then what setting am I missing?
I did select “Submit test results for multiple test points…” setting in Azure Integration settings.

Here are contents of Event Log from Katalon Studio:

Blockquote
13:25:58.762 [Worker-43: Launching the test suite…] DEBUG com.kms.katalon.core.testdata.TestDataFactory - Finding test data with id ‘Data Files/PeerPointData’
13:25:58.763 [Worker-43: Launching the test suite…] DEBUG com.kms.katalon.core.testdata.TestDataFactory - Test data is excel file, reading excel file
13:25:58.763 [Worker-43: Launching the test suite…] DEBUG com.kms.katalon.core.testdata.TestDataFactory - Reading excel file with source file ‘/Users/XXXXXXXXXXXX/Desktop/XXXXXXXXXXXX/XXXXXXXXXXXX/XXXXXXXXXXXX/Data Files/XXXXXXXXXXXX.xlsx’ and sheet name ‘TestData’
chromedriver is located at default location: /Applications/Katalon Studio.app/Contents/Eclipse/configuration/resources/drivers/chromedriver_mac/chromedriver. In case your browser is updated to a newer version, please use this command to update chromdriver: --config -webui.autoUpdateDrivers=true
Launcher status after execution process completed: Running
Launcher status after execution process completed: Running
Start writing execution.uuid file to folder: /Users/XXXXXXXXXXXX/Desktop/XXXXXXXXXXXX/XXXXXXXXXXXX/XXXXXXXXXXXX/Reports/20210624_132558/SmokeTest/20210624_132559
Start writing execution.uuid file to folder: /Users/XXXXXXXXXXXX/Desktop/XXXXXXXXXXXX/XXXXXXXXXXXX/XXXXXXXXXXXX/Reports/20210624_132558/SmokeTest/20210624_132559
Start generating JUnit report folder at: /Users/XXXXXXXXXXXX/Desktop/XXXXXXXXXXXX/XXXXXXXXXXXX/XXXXXXXXXXXX/Reports/20210624_132558/SmokeTest/20210624_132559…
Start generating JUnit report folder at: /Users/XXXXXXXXXXXX/Desktop/XXXXXXXXXXXX/XXXXXXXXXXXX/XXXXXXXXXXXX/Reports/20210624_132558/SmokeTest/20210624_132559…
JUnit report were generated at folder: /Users/XXXXXXXXXXXX/Desktop/XXXXXXXXXXXX/XXXXXXXXXXXX/XXXXXXXXXXXX/Reports/20210624_132558/SmokeTest/20210624_132559
JUnit report were generated at folder: /Users/XXXXXXXXXXXX/Desktop/XXXXXXXXXXXX/XXXXXXXXXXXX/XXXXXXXXXXXX/Reports/20210624_132558/SmokeTest/20210624_132559
For real-time monitoring and better reporting capabilities please integrate this project with Katalon TestOps (more details at https://docs.katalon.com/katalon-analytics/docs/integration-with-katalon-studio.html#enable-integration)
Start sending test result to Katalon TestOps
Uploading log files of test suite
Sending file: /Users/XXXXXXXXXXXX/Desktop/XXXXXXXXXXXX/XXXXXXXXXXXX/XXXXXXXXXXXX/Reports/20210624_132558/SmokeTest/20210624_132559/execution0.log
Sending file: /Users/XXXXXXXXXXXX/Desktop/XXXXXXXXXXXX/XXXXXXXXXXXX/XXXXXXXXXXXX/Reports/20210624_132558/SmokeTest/20210624_132559/tsc_id.txt
Sending file: /Users/XXXXXXXXXXXX/Desktop/XXXXXXXXXXXX/XXXXXXXXXXXX/XXXXXXXXXXXX/Reports/20210624_132558/SmokeTest/20210624_132559/20210624_132559.html
Sending file: /Users/XXXXXXXXXXXX/Desktop/XXXXXXXXXXXX/XXXXXXXXXXXX/XXXXXXXXXXXX/Reports/20210624_132558/SmokeTest/20210624_132559/testCaseBinding
Sending file: /Users/XXXXXXXXXXXX/Desktop/XXXXXXXXXXXX/XXXXXXXXXXXX/XXXXXXXXXXXX/Reports/20210624_132558/SmokeTest/20210624_132559/JUnit_Report.xml
Sending file: /Users/XXXXXXXXXXXX/Desktop/XXXXXXXXXXXX/XXXXXXXXXXXX/XXXXXXXXXXXX/Reports/20210624_132558/SmokeTest/20210624_132559/20210624_132559.csv
Sending file: /Users/XXXXXXXXXXXX/Desktop/XXXXXXXXXXXX/XXXXXXXXXXXX/XXXXXXXXXXXX/Reports/20210624_132558/SmokeTest/20210624_132559/execution.properties
Sending file: /Users/XXXXXXXXXXXX/Desktop/XXXXXXXXXXXX/XXXXXXXXXXXX/XXXXXXXXXXXX/Reports/20210624_132558/SmokeTest/20210624_132559/execution.uuid
Katalon TestOps: Start uploading report to Katalon TestOps server: https://analytics.katalon.com
Katalon TestOps: Start uploading report to Katalon TestOps server: https://analytics.katalon.com
Katalon TestOps - Execution URL: https://analytics.katalon.com/from-ks/execution-result?teamId=51962&projectId=64374&id=14509981
End sending to Katalon TestOps
Azure: Creating test run…
Azure: Creating test run…
Azure: Test run 100 created. URL=https://dev.azure.com//XXXXXXXXXXXX/XXXXXXXXXXXX/XXXXXXXXXXXX/_apis/test/Runs/100
Azure: Test run 100 created. URL=https://dev.azure.com//XXXXXXXXXXXX/XXXXXXXXXXXX/XXXXXXXXXXXX/_apis/test/Runs/100
Azure: Retrieving test case results…
Azure: Retrieving test case results…
Azure: Updating test case results…
Azure: Updating test case results…
Azure: Uploading file: 20210624_132559.html…
Azure: Uploading file: 20210624_132559.html…
Azure: Completing test run…
Azure: Completing test run…
Azure: Test run Run by Katalon Studio (ID: 100) with Test Pland Id 17 completed. URL=https://dev.azure.com//XXXXXXXXXXXX/XXXXXXXXXXXX/XXXXXXXXXXXX/_apis/test/Runs/100
Azure: Test run Run by Katalon Studio (ID: 100) with Test Pland Id 17 completed. URL=https://dev.azure.com//XXXXXXXXXXXX/XXXXXXXXXXXX/XXXXXXXXXXXX/_apis/test/Runs/100

Does the Azure Devops Integration also work with the Katalon Runtime Engine?

@o.huibers,

Yes, it does. Don’t forget to keep [project folder]/settings/external/com.kms.katalon.integration.azure.properties that includes Azure DevOps integration information in your repository.

1 Like

Hi …I have integrated my project and execution is successful .My only concern is it doesn’t attach test result ( html report) in the history of test case execution .

image
image

1 Like

Did you find an answer to this query?

Without this Association we are unable to set the state as Automated. This way are unable to create useful Reporting for our Functional Automation Testing Coverage from ADO.