How to update azuretestplan status when katalon tests run from pipeline with agent?

testql · June 17, 2025, 2:23am

Hello need to know how to make execution status updated when I use my ADO pipeline. This works when I run from my PC, but today nobody use a PC to run automation script. Can you explain step by step how to make this set up please?

dineshh · June 17, 2025, 7:29am

To update Azure Test Plan statuses when running Katalon tests via an Azure DevOps (ADO) pipeline using a build/release agent:

Prerequisites

Azure Test Plan Setup:

Test cases are created in Azure Test Plans.
Each test case has a work item ID (e.g., 12345).

Katalon Project Setup:

Use the TestCase tag in Katalon to map tests to Azure Test Plan IDs (e.g., @TestCase:12345).
Configure Katalon to generate JUnit or TRX test reports (needed for Azure integration).

Pipeline Permissions:

The ADO pipeline service account has permissions to Update Test Plans.
Use a PAT token with Test Management scope or an Azure Service Connection with sufficient access.

Step 1: Link Katalon Tests to Azure Test Cases

In your Katalon test script, add the TestCase tag with the Azure Test Plan work item ID:

groovy

@Test
@TestCase("12345") // Replace with your Azure Test Case ID
def "Verify Login Functionality"() {
  // Test steps
}

Step 2: Configure Katalon to Generate Test Reports

In your Katalon project’s build.gradle, ensure JUnit/TRX reporting is enabled:

text

test {
  useJUnit {
  }
  reports {
    junitXml.enabled = true
    html.enabled = true
  }
}

Step 3: Set Up the Azure Pipeline

YAML Pipeline Example

text

trigger:
- main

pool:
  vmImage: 'windows-latest' # or 'ubuntu-latest'

variables:
  katalonVersion: '8.3.5' # Match your Katalon CLI version
  katalonProjectPath: '$(System.DefaultWorkingDirectory)/YourKatalonProject'

steps:
- task: CmdLine@2
  displayName: 'Run Katalon Tests'
  inputs:
    script: |
      cd ./Katalon_Studio_Engine_Linux_64-${katalonVersion} # For Linux agents
      ./katalonc -noSplash -runMode=console \
        -projectPath="${katalonProjectPath}" \
        -retry=0 \
        -testSuitePath="Test Suites/YourTestSuite" \
        -browserType="Chrome" \
        -apiKey="YOUR_KATALON_API_KEY" \
        -reportFolder="$(Agent.TempDirectory)/Reports" \
        -reportFileName="TestResults"

# Publish test results to Azure Test Plans
- task: PublishTestResults@2
  inputs:
    testResultsFormat: 'JUnit' # or 'VSTest' for TRX
    testResultsFiles: '**/TEST-*.xml' # Path to JUnit reports
    mergeTestResults: true
    testRunTitle: 'Katalon Test Run'

# Optional: Update Test Plan status via REST API (if needed)
- powershell: |
    $url = "https://dev.azure.com/{org}/{project}/_apis/testplan/Plans/{planId}/Suites/{suiteId}/TestPoint?api-version=7.1-preview.2"
    $body = @{
      outcome = "Passed" # or "Failed"
    } | ConvertTo-Json
    Invoke-RestMethod -Uri $url -Method Patch -Body $body -Headers @{
      Authorization = "Bearer $(System.AccessToken)"
      "Content-Type" = "application/json"
    }
  displayName: 'Update Test Plan Status (Optional)'

Step 4: Map Test Results to Azure Test Cases

In Azure Test Plans, ensure the test case IDs match the tags in Katalon.
Use the PublishTestResults@2 task to upload results. Azure will auto-link results to test cases if IDs match.

Step 5: Authentication

If using PAT:
- Add a secret variable AZURE_TOKEN in the pipeline and use it in REST API calls.
If using System.AccessToken:
- Enable the Allow scripts to access the OAuth token option in the pipeline job.

Troubleshooting

Test results not linked:

Ensure the @TestCase tag matches the Azure Test Case ID exactly.
Verify the JUnit/TRX report contains the correct IDs.

Permission errors:

Grant the Project Collection Build Service account Contributor rights in Test Plans.

Report path issues:

Use $(Agent.TempDirectory) for report paths to avoid permission conflicts.

Alternate Approach: Use the Katalon Azure DevOps Plugin

Install the Katalon Studio extension from the Azure Marketplace.
Use the Katalon Studio task in your pipeline:

text

- task: KatalonStudio@2
  inputs:
    version: '8.3.5'
    command: '-runMode=console -projectPath="$(System.DefaultWorkingDirectory)/Project" -testSuitePath="Test Suites/YourSuite" -reportFolder="$(Agent.TempDirectory)"'

By following these steps, your Azure Test Plan statuses will update automatically when Katalon tests run via the pipeline.

testql · June 17, 2025, 8:14am

Thanks Dineshh,
I use the extension Katalon for azuredevops and I have already linked azure testcase ID in every katalon testcase using Integration tab.

Katalon is installed in a VM and run by self hosted agent.
Here how I run Katalon in the actual pipeline :

task: katalon-llc.katalon.katalonTask.katalonTask@1

displayName: ‘Execute Katalon Studio project’

inputs:

location: ‘C:\Delivery\Applications\KATALON\Katalon_Studio_Engine_Windows_64-9.1.0’

executeArgs: 'katalonc -noSplash -runMode=console -retry=0 -testSuitePath=“Test Suites/DSK” -executionProfile=“UAT” -browserType=“Chrome” --config -proxy.auth.option=USE_SYSTEM -proxy.system.option=USE_SYSTEM -proxy.system.applyToDesiredCapabilities=true -webui.autoUpdateDrivers=true

Can you tell me how to update this with the task KatalonStudio@2 you gave me ?

dineshh · June 18, 2025, 5:36am

[quote=“testql, post:3, topic:176284”]
katalon-llc.katalon.katalonTask.katalonTask@1
[/quote] couldn’t access the link

dineshh · June 18, 2025, 5:45am

Step-by-Step Pipeline Update

Replace your current katalonTask@1 with the following YAML configuration for KatalonStudio@2:

- task: KatalonStudio@2
  displayName: 'Execute Katalon Studio Project'
  inputs:
    katalonRuntime: 'CUSTOM'  # Required for self-hosted installations
    customKatalonLocation: 'C:\Delivery\Applications\KATALON\Katalon_Studio_Engine_Windows_64-9.1.0'  # Path to your Katalon installation
    projectPath: '$(System.DefaultWorkingDirectory)\YourKatalonProjectFolder'  # Path to the Katalon project (e.g., repo root)
    command: '-noSplash -runMode=console -retry=0 -testSuitePath="Test Suites/DSK" -executionProfile="UAT" -browserType="Chrome" -proxy.auth.option=USE_SYSTEM -proxy.system.option=USE_SYSTEM -proxy.system.applyToDesiredCapabilities=true -webui.autoUpdateDrivers=true'

Key Changes Explained

Task Version:

Switched from katalonTask@1 to KatalonStudio@2 (modern task with better integration).

Custom Katalon Installation:

katalonRuntime: 'CUSTOM' tells Azure to use your pre-installed Katalon Engine.
customKatalonLocation: Explicit path to your Katalon Engine installation (matches your VM setup).

Project Path:

projectPath: Ensure this points to the root folder of your Katalon project (e.g., where prj and Test Suites folders reside). If your repo is cloned to $(System.DefaultWorkingDirectory), adjust the path accordingly (e.g., $(System.DefaultWorkingDirectory)\YourProjectName).

Command Arguments:

Retained all your existing arguments (test suite, profile, browser, proxy settings).
Fix: Removed --config (not needed unless explicitly referencing a config file).
Use straight quotes (") instead of curly quotes (“/”) to avoid YAML parsing issues.

Additional Configuration

Proxy Settings (If Required)

If your self-hosted agent needs proxy authentication, add these environment variables to the pipeline agent or task:

env:
  HTTP_PROXY: 'http://proxy-server:port'
  HTTPS_PROXY: 'http://proxy-server:port'
  NO_PROXY: 'localhost,127.0.0.1'

Publish Test Results to Azure Test Plans

Add this step after the Katalon task to auto-update Test Plan statuses:

- task: PublishTestResults@2
  inputs:
    testResultsFormat: 'JUnit'
    testResultsFiles: '**/TEST-*.xml'  # Default JUnit report path
    testRunTitle: 'Katalon Test Run'

Troubleshooting Tips

Path Validation:

Ensure customKatalonLocation points to the root folder of Katalon Studio Engine (not the .exe file).
Example valid path:

C:\Delivery\Applications\KATALON\Katalon_Studio_Engine_Windows_64-9.1.0

Katalon CLI Compatibility:

Confirm your Katalon version (9.1.0) matches the engine installed on the VM.

Self-Hosted Agent Permissions:

The agent must have read/write access to the Katalon installation directory and project folder.

Logs:

Check pipeline logs under the Execute Katalon Studio Project step for errors like missing dependencies or invalid paths.

Final Pipeline Example

steps:
- checkout: self  # Check out your repo

- task: KatalonStudio@2
  displayName: 'Execute Katalon Studio Project'
  inputs:
    katalonRuntime: 'CUSTOM'
    customKatalonLocation: 'C:\Delivery\Applications\KATALON\Katalon_Studio_Engine_Windows_64-9.1.0'
    projectPath: '$(System.DefaultWorkingDirectory)\YourKatalonProject'
    command: '-noSplash -runMode=console -retry=0 -testSuitePath="Test Suites/DSK" -executionProfile="UAT" -browserType="Chrome" -proxy.auth.option=USE_SYSTEM -proxy.system.option=USE_SYSTEM -proxy.system.applyToDesiredCapabilities=true -webui.autoUpdateDrivers=true'

- task: PublishTestResults@2
  displayName: 'Publish Test Results to Azure'
  inputs:
    testResultsFormat: 'JUnit'
    testResultsFiles: '**/TEST-*.xml'
    testRunTitle: 'Katalon Test Run'

testql · June 25, 2025, 8:49am

Thanks for your answer. I’ve been trying to find KatalonStudio@2 in tasks but nothing. Do I have to do something special to see this task ready to use in pipeline assistant ? the only one I have for katalon is “Execute Katalon Studio Tests”

dineshh · June 26, 2025, 6:28am

Verify Extension Installation
Go to your Azure DevOps organization → Manage extensions → Check if “Katalon for Azure DevOps” is installed. If not:

Visit Katalon for Azure DevOps extension page
Click “Get it free” and install it for your organization.

Using the Correct Task Name
The task you see (“Execute Katalon Studio Tests”) is the correct one for KatalonStudio@2. There’s no separate “KatalonStudio@2” in the UI - this is the YAML identifier for the visual task.
Configure Pipeline Task
In your YAML pipeline, use this structure:

text

steps:
- task: KatalonStudio@2
  displayName: 'Execute Katalon Studio Tests'
  inputs:
    katalonRuntime: 'CUSTOM'
    customKatalonLocation: 'C:\Delivery\Applications\KATALON\Katalon_Studio_Engine_Windows_64-9.1.0'
    projectPath: '$(System.DefaultWorkingDirectory)\YourProjectFolder'
    command: '-noSplash -runMode=console -retry=0 -testSuitePath="Test Suites/DSK" -executionProfile="UAT" -browserType="Chrome" -proxy.auth.option=USE_SYSTEM -proxy.system.option=USE_SYSTEM -proxy.system.applyToDesiredCapabilities=true -webui.autoUpdateDrivers=true'

Replace YourProjectFolder with your Katalon project directory name.
4. Publish Test Results (Critical for Azure Test Plans integration)
Add this task after Katalon execution:

text

- task: PublishTestResults@2
  inputs:
    testResultsFormat: 'JUnit'
    testResultsFiles: '**/TEST-*.xml'
    testRunTitle: 'Katalon Test Results'
    mergeTestResults: true

Key Configuration Points

Parameter	Value	Notes
`katalonRuntime`	`CUSTOM`	Required for self-hosted installation
`customKatalonLocation`	Path to Katalon Engine	Must be exact installation path on VM
`projectPath`	`$(System.DefaultWorkingDirectory)\...`	Path to Katalon project root (contains `prj` folder)
`command`	Your CLI arguments	Use straight quotes `"` instead of curly quotes

Troubleshooting Tips

Agent Permission Issues
Ensure the agent service account has:

Full control of Katalon installation directory (C:\Delivery\Applications\KATALON)
Modify rights to the project directory

Path Validation
Add a PowerShell step to verify paths exist:

text

- powershell: |
    Test-Path "$(env:customKatalonLocation)\katalonc.exe"
    Test-Path "$(projectPath)"
  displayName: 'Validate Paths'

Check Katalon Version Compatibility
Ensure your Katalon Engine version (9.1.0) matches the CLI syntax. Update Katalon if needed.
Log Collection
Add this to your command to get detailed logs:

text

command: '... -consoleLog -logLevel=INFO -reportFolder="$(System.DefaultWorkingDirectory)\Results"'

Full Pipeline Example

text

steps:
- checkout: self  # Required to get your test scripts

- task: KatalonStudio@2
  displayName: 'Execute Katalon Tests'
  inputs:
    katalonRuntime: 'CUSTOM'
    customKatalonLocation: 'C:\Delivery\Applications\KATALON\Katalon_Studio_Engine_Windows_64-9.1.0'
    projectPath: '$(System.DefaultWorkingDirectory)\KatalonProject'
    command: '-noSplash -runMode=console -retry=0 -testSuitePath="Test Suites/DSK" -executionProfile="UAT" -browserType="Chrome" -proxy.auth.option=USE_SYSTEM -proxy.system.option=USE_SYSTEM -proxy.system.applyToDesiredCapabilities=true -webui.autoUpdateDrivers=true'

- task: PublishTestResults@2
  displayName: 'Publish to Azure Test Plans'
  inputs:
    testResultsFormat: 'JUnit'
    testResultsFiles: '**/TEST-*.xml'
    testRunTitle: 'Katalon Regression Suite'

Post-Execution

After a successful run:

Go to Azure Test Plans → Test Runs
Find “Katalon Regression Suite” run
You should see:

Test cases linked via ID
Status updated automatically (Passed/Failed)
Stack traces for failures

If test cases still don’t update:

Verify Azure Test Case IDs are in this format: AB123 (no extra characters)
Check test case IDs are mapped in Katalon: Project → Settings → Integration → Azure DevOps
Ensure the Katalon project has API access to Azure DevOps

Topic		Replies	Views
Why isn't my ADO test plan being updated when running tests from ADO pipeline while it does work when running tests locally from Katalon Studio? Katalon TestCloud azure , katalon-testcloud	1	514	January 19, 2024
Reports location on test run on Agents Katalon Studio katalon-studio	0	330	October 4, 2019
Issue getting test case results to update in Azure Katalon Studio integrations , katalon-studio	1	875	January 16, 2025
Can you configure a katalon test case through a ADO Pipeline Katalon Studio integrations , azure , katalon-studio	0	962	May 18, 2022
Can an Azure Agent (pipeline) execution update TestOps for reporting? Product Forums azure , katalon-testops , git-interface	1	918	June 22, 2022

How to update azuretestplan status when katalon tests run from pipeline with agent?

Prerequisites

Step 1: Link Katalon Tests to Azure Test Cases

Step 2: Configure Katalon to Generate Test Reports

Step 3: Set Up the Azure Pipeline

YAML Pipeline Example

Step 4: Map Test Results to Azure Test Cases

Step 5: Authentication

Troubleshooting

Alternate Approach: Use the Katalon Azure DevOps Plugin

Step-by-Step Pipeline Update

Key Changes Explained

Additional Configuration

Proxy Settings (If Required)

Publish Test Results to Azure Test Plans

Troubleshooting Tips

Final Pipeline Example

Key Configuration Points

Troubleshooting Tips

Full Pipeline Example

Post-Execution

Related topics