Integration with Azure DevOps Test Plans (Pre-release)

Feedback & Reviews Documentation
1
This is a companion discussion topic for the original entry at https://docs.katalon.com/katalon-studio/docs/azure-devops-test-plans.html
2

Hi,

First, congratulations to the team because of this new feature. That will be very helpful for us.

However, I implemented the steps according to the documentation and when I run the tests, in my Event Log appear this error:

Katalon TestOps: Unexpected response, URL: https://analytics.katalon.com/api/v1/organizations/setting?organizationId=194758, Status: 403, Response: {ā€œerrorā€:ā€œaccess_deniedā€,ā€œerror_descriptionā€:ā€œEither this resource does not exist, or the current user does not have permissions to access this resourceā€}

And the results will not be sent to Test Plan (in Azure). I guessed about the license, but, I have an Enterprise license active.

Important: Iā€™m not using Katalon TestOps.

Can you help me?

3

Hi Sandro,

Thank you for your interest about our new feature.
To answer your problem, could you please check your Azure Devops Token is full scope access or not. Then send us your Execution Log if there is till error.

Have a nice day,

4

Hi, Huyen!

Same error! :frowning:

image
image1398Ɨ309 29.7 KB

This behavior is strange because I can configure all settings (about my project in Azure), link the test cases between differents platforms (Azure and Katalon), but, the results arenā€™t sent by Katalon and I donā€™t know why.

Also, the PAT token is full access.

5

Important: I disabled the devops integration and the error persists. I think that my screenshot isnā€™t true.

So, @huyennguyen, to execute Katalon with Azure DevOps is necessary extra configuration than Integration with Azure DevOps Test Plans | Katalon Docs?

Thanks.

6

Hi Sandro,

Would you mind copying all Event Log and paste them here for me?

Thank you for your time.

2 Likes
7

Hi,

Here is it: https://pastebin.com/4jMcL7jW

Thank you!

8

Hi Sandro,

Your log reveals that there is no indication of Azure DevOps.
You can follow this video for instructions.
If your result still cannot send to Azure DevOps. Can you capture your Azure settings or it would be better if you can give me a video of how you configured and ran it.

Thank you.

2 Likes
9

Hi, Huyen!

Is working fine now. The error occurred because of myselfā€¦ I was running the tests to test this feature separately, but, this feature works only in test suite collections. :smiley:

Thank you so much for your help!

2 Likes
10

Hi Team

I have the issue working after resolving the issue with fetching projects (see discussion: link). Big thank you Huyen for helping me out with that issue.

I have a question on how we could report on the automated test coverage. In Katalon we link a Katalon test case to a DevOps test case in the integration tab:
image

However if I look at that linked test case in Azure DevOps, I cannot find the relationship back to the Katalon test case. I.e. the tab with ā€œAssociated Automationā€ is blank

image
image805Ɨ450 16 KB

or the ā€œAutomation Statusā€ remains unchanged.
image
image802Ɨ475 21.9 KB

Are their plans to make it visible in DevOps if a manual test case is asscociated to an automated test case?

Regards

Peter

11

Hi,

I have a use case where I need to run the same test multiple times. So to achieve this I added the test to a test suite and used a data file with 3 rows of different data. The first 2 rows have correct data but the third row causes a failure. So Katalon correctly displays the test results as 2 passed and 1 failed and the same results are propagated to TestOps as well. However on Azure I see that it shows all 3 tests failed and from the screenshot below it is clear that it is overwriting all test runs with the last one which failed and the execution times are also exactly same:

Azure
Azure2830Ɨ556 89.4 KB

For comparison here is screenshot from TestOps

TestOps
TestOps3352Ɨ670 198 KB

And from Katalon Studio

KatalonStudio
KatalonStudio2462Ɨ1146 279 KB

Can you please tell if this workflow is supported? And if it is then what setting am I missing?
I did select ā€œSubmit test results for multiple test pointsā€¦ā€ setting in Azure Integration settings.

Here are contents of Event Log from Katalon Studio:

Blockquote
13:25:58.762 [Worker-43: Launching the test suiteā€¦] DEBUG com.kms.katalon.core.testdata.TestDataFactory - Finding test data with id ā€˜Data Files/PeerPointDataā€™
13:25:58.763 [Worker-43: Launching the test suiteā€¦] DEBUG com.kms.katalon.core.testdata.TestDataFactory - Test data is excel file, reading excel file
13:25:58.763 [Worker-43: Launching the test suiteā€¦] DEBUG com.kms.katalon.core.testdata.TestDataFactory - Reading excel file with source file ā€˜/Users/XXXXXXXXXXXX/Desktop/XXXXXXXXXXXX/XXXXXXXXXXXX/XXXXXXXXXXXX/Data Files/XXXXXXXXXXXX.xlsxā€™ and sheet name ā€˜TestDataā€™
chromedriver is located at default location: /Applications/Katalon Studio.app/Contents/Eclipse/configuration/resources/drivers/chromedriver_mac/chromedriver. In case your browser is updated to a newer version, please use this command to update chromdriver: --config -webui.autoUpdateDrivers=true
Launcher status after execution process completed: Running
Launcher status after execution process completed: Running
Start writing execution.uuid file to folder: /Users/XXXXXXXXXXXX/Desktop/XXXXXXXXXXXX/XXXXXXXXXXXX/XXXXXXXXXXXX/Reports/20210624_132558/SmokeTest/20210624_132559
Start writing execution.uuid file to folder: /Users/XXXXXXXXXXXX/Desktop/XXXXXXXXXXXX/XXXXXXXXXXXX/XXXXXXXXXXXX/Reports/20210624_132558/SmokeTest/20210624_132559
Start generating JUnit report folder at: /Users/XXXXXXXXXXXX/Desktop/XXXXXXXXXXXX/XXXXXXXXXXXX/XXXXXXXXXXXX/Reports/20210624_132558/SmokeTest/20210624_132559ā€¦
Start generating JUnit report folder at: /Users/XXXXXXXXXXXX/Desktop/XXXXXXXXXXXX/XXXXXXXXXXXX/XXXXXXXXXXXX/Reports/20210624_132558/SmokeTest/20210624_132559ā€¦
JUnit report were generated at folder: /Users/XXXXXXXXXXXX/Desktop/XXXXXXXXXXXX/XXXXXXXXXXXX/XXXXXXXXXXXX/Reports/20210624_132558/SmokeTest/20210624_132559
JUnit report were generated at folder: /Users/XXXXXXXXXXXX/Desktop/XXXXXXXXXXXX/XXXXXXXXXXXX/XXXXXXXXXXXX/Reports/20210624_132558/SmokeTest/20210624_132559
For real-time monitoring and better reporting capabilities please integrate this project with Katalon TestOps (more details at https://docs.katalon.com/katalon-analytics/docs/integration-with-katalon-studio.html#enable-integration)
Start sending test result to Katalon TestOps
Uploading log files of test suite
Sending file: /Users/XXXXXXXXXXXX/Desktop/XXXXXXXXXXXX/XXXXXXXXXXXX/XXXXXXXXXXXX/Reports/20210624_132558/SmokeTest/20210624_132559/execution0.log
Sending file: /Users/XXXXXXXXXXXX/Desktop/XXXXXXXXXXXX/XXXXXXXXXXXX/XXXXXXXXXXXX/Reports/20210624_132558/SmokeTest/20210624_132559/tsc_id.txt
Sending file: /Users/XXXXXXXXXXXX/Desktop/XXXXXXXXXXXX/XXXXXXXXXXXX/XXXXXXXXXXXX/Reports/20210624_132558/SmokeTest/20210624_132559/20210624_132559.html
Sending file: /Users/XXXXXXXXXXXX/Desktop/XXXXXXXXXXXX/XXXXXXXXXXXX/XXXXXXXXXXXX/Reports/20210624_132558/SmokeTest/20210624_132559/testCaseBinding
Sending file: /Users/XXXXXXXXXXXX/Desktop/XXXXXXXXXXXX/XXXXXXXXXXXX/XXXXXXXXXXXX/Reports/20210624_132558/SmokeTest/20210624_132559/JUnit_Report.xml
Sending file: /Users/XXXXXXXXXXXX/Desktop/XXXXXXXXXXXX/XXXXXXXXXXXX/XXXXXXXXXXXX/Reports/20210624_132558/SmokeTest/20210624_132559/20210624_132559.csv
Sending file: /Users/XXXXXXXXXXXX/Desktop/XXXXXXXXXXXX/XXXXXXXXXXXX/XXXXXXXXXXXX/Reports/20210624_132558/SmokeTest/20210624_132559/execution.properties
Sending file: /Users/XXXXXXXXXXXX/Desktop/XXXXXXXXXXXX/XXXXXXXXXXXX/XXXXXXXXXXXX/Reports/20210624_132558/SmokeTest/20210624_132559/execution.uuid
Katalon TestOps: Start uploading report to Katalon TestOps server: https://analytics.katalon.com
Katalon TestOps: Start uploading report to Katalon TestOps server: https://analytics.katalon.com
Katalon TestOps - Execution URL: Katalon TestOps
End sending to Katalon TestOps
Azure: Creating test runā€¦
Azure: Creating test runā€¦
Azure: Test run 100 created. URL=https://dev.azure.com//XXXXXXXXXXXX/XXXXXXXXXXXX/XXXXXXXXXXXX/_apis/test/Runs/100
Azure: Test run 100 created. URL=https://dev.azure.com//XXXXXXXXXXXX/XXXXXXXXXXXX/XXXXXXXXXXXX/_apis/test/Runs/100
Azure: Retrieving test case resultsā€¦
Azure: Retrieving test case resultsā€¦
Azure: Updating test case resultsā€¦
Azure: Updating test case resultsā€¦
Azure: Uploading file: 20210624_132559.htmlā€¦
Azure: Uploading file: 20210624_132559.htmlā€¦
Azure: Completing test runā€¦
Azure: Completing test runā€¦
Azure: Test run Run by Katalon Studio (ID: 100) with Test Pland Id 17 completed. URL=https://dev.azure.com//XXXXXXXXXXXX/XXXXXXXXXXXX/XXXXXXXXXXXX/_apis/test/Runs/100
Azure: Test run Run by Katalon Studio (ID: 100) with Test Pland Id 17 completed. URL=https://dev.azure.com//XXXXXXXXXXXX/XXXXXXXXXXXX/XXXXXXXXXXXX/_apis/test/Runs/100

12

Does the Azure Devops Integration also work with the Katalon Runtime Engine?

13

@o.huibers,

Yes, it does. Donā€™t forget to keep [project folder]/settings/external/com.kms.katalon.integration.azure.properties that includes Azure DevOps integration information in your repository.

1 Like
14

Hi ā€¦I have integrated my project and execution is successful .My only concern is it doesnā€™t attach test result ( html report) in the history of test case execution .

image
image

1 Like
15

Did you find an answer to this query?

Without this Association we are unable to set the state as Automated. This way are unable to create useful Reporting for our Functional Automation Testing Coverage from ADO.

16

Hi @duyluong
is there an answer for @novino_dragonā€™s post above?
They asked about integration with ADO where a Katalon test case is run as a data driven test. In this case, a single ADO test case maps to multiple test results (eg a test of mine runs 10 times, using different data). What is best way to present this information please?

17

Please use Azure DevOps restapi, we used in our project , it worked successfully

You could use REST API to do this Work Items - Update:

PATCH https://dev.azure.com/{organization}/{project}/_apis/wit/workitems/{id}?api-version=6.0

Content-Type: application/json-patch+json

Body:

[
  {
    "op": "add",
    "path": "/fields/Microsoft.VSTS.TCM.AutomatedTestName",
    "value": "[namespace.classname.methodname (e.g. UnitTestProject1.UnitTest1.TestMethod2)]"
  },
  {
    "op": "add",
    "path": "/fields/Microsoft.VSTS.TCM.AutomatedTestStorage",
    "value": "[assembly name(e.g. unittestproject1.dll)"
  },
  {
    "op": "add",
    "path": "/fields/Microsoft.VSTS.TCM.AutomatedTestId",
    "value": "[guid id]"
  },
  {
    "op": "add",
    "path": "/fields/Microsoft.VSTS.TCM.AutomatedTestType",
    "value": "Unit Test"
  },
   {
    "op": "add",
    "path": "/fields/Microsoft.VSTS.TCM.AutomationStatus",
    "value": "Automated"
  }
]