Hi, I’m executing the automated test using python and PyTest Framework. So far the execution run smoothly, but when i check the TestResult, logs show empty data, then I check several information that i think it’s quite important for the Test Report, it’s also not show, my question are, how to configure these information from the python script so these important information can be readable from TestOps Report:
I doubt TestOps supports pytest or any other framework.
It is designed for Katalon use, and I cannot find any API docs of it.
You should abandon your idea, with pytest use any other TestCase management tool which supports uploading results from standard junit report files (junit.xml) or have at least a decent API properly documented.
e.g testrail, this have both
@didit ah, ok.
well, please show us the full pytest command.
do you produce an junit report? perhaps testops need such to further parse it to update the results?
just guessing
Hi Bionel,
I’m not sure those options will help as the first example will generate JUnit XML Report Files which I still need to upload manually and I think is not effective, we are also still not sure the report that showed will be the same report from “testops_pytest.listener” or not
Have you tried?
For my understanding, TestOps is a CI wanabbe tool.
Decent CI tools (e.g Bamboo) have the option to parse certain reports in standard formats.
Since you have the option to configure the report folder in the job config, I expect testops to do the same, but you have to specify with your pytest command where to save it …
Or there is no option to specify that? If like that, this ‘integration’ is useless.
However, notice that I never used this tool. So i am just guessing. @albert.vu any idea who can bring some light into how this option is supposed to work?
looks like you have to specify a certain plugin in your conftest.py file.
Do you have such? Not sure what plugin should be there, but i assume it is the one mentioned in your screenshot with pip install.
I assume this plugin will activate some pytest hooks which should collect the test results and upload them automatically to TestOps (brilliant idea to blur exactly the plugin needed in the screenshots)
LE: or may be pytest_testops.listener
Also, looks like the testops-config.json file should exist also in your pytest collection, apparently those fields on testops UI only generates the sample code needed.
@Elly_Tran Can please bring this into attention of tech writting teams?
I understand to censor sensitive data, but also the name of the plugin needed?
Note also, the link to TestOps Pytest sample in that doc goes to a repository have anything else than something usefull for pytest (just some configs for other frameworks)
Aside that, this doc is totally ambiguous.
Yes I’d specify a plugin based on the docs mentioned here
Already following all of the steps there, the blurred code inside the docs is pytest_testops.listener so I assume that maybe there are some other configuration that can bring testops result more meaningful
The sample usage for Integrate Pytest with TestOps can be found here GitHub - katalon-studio-samples/testops-pytest-sample , but I cannot find any specific document that configures anything inside TestOps reports
Wow, are you Albert Vu who often comes to my email inbox for Announcement and updates?
I’m glad to see your comment here, thanks @bionel for tagging @albert.vu
or you may have some conecitvity issues from the environment where the tests are runing to TestOps.
Are you running them locally, or in a testops agent?
Hi Bionel,
It has very little chance this problem happen because of connectivity, aside from that I’ve tried to compare testops results between python script and studio using the same machine, and the result the problem are not happen from the execution from katalon studio(of course, because testops designed for studio)
here’s the log when using Python pytest after script finishes testing
well, i am out of idea right now.
provided i can find the source code of the plugin, i will look into it to see what it intends to do.
not sure if i can use testops with a free licence, if yes i may attempt to do a draft pytest script to see how it behave, but i cannot promise (it depends on my available time)
in the mean time let’s wait for @tri.tle input
Hi Bionel,
It would be great if you could see the plugin code to see what happened there, please let me know when you find something
AFAIK it allows free license but is limited to certain numbers of test run , you doesn’t need to do a draft of pytest script, just clone it from GitHub - katalon-studio-samples/testops-pytest-sample then change testops-config.json file with json configuration that you can copy from Project Integration
and then just run it on terminal using pytest or python pytest it will execute sample testcase provided on the script and also upload the report to testops
Grabbed the sample project. At first attempt, looks like the plugin is broken at least for python 3.11, got some errors:
LE: nvm, was due to the "projectId": field in the config.json file …
@didit what is the python version used by you?
I am using 3.11.something … anyway, shouldn’t matter right now.
The plugin indeed seems to implement some hooks, even if I run with -collect-only it attempts to publish something to testops.
(.venv) [ibus@ibus-t14 testops-pytest-sample]$ pytest tests/test_calculator.py --collect-only
========================================================================================== test session starts ==========================================================================================
platform linux -- Python 3.11.5, pytest-7.4.2, pluggy-1.3.0
rootdir: /home/ibus/Work/katalon/testops-pytest-sample
collected 4 items
<Module tests/test_calculator.py>
<Function test_add_pass>
<Function test_exception>
<Function test_add_fail>
<Function test_add_skip>
=========================================================================================== warnings summary ============================================================================================
.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:1095
/home/ibus/Work/katalon/testops-pytest-sample/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:1095: InsecureRequestWarning: Unverified HTTPS request is being made to host 'testops.katalon.io'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings
warnings.warn(
.venv/lib64/python3.11/site-packages/testops_api/rest.py:36
/home/ibus/Work/katalon/testops-pytest-sample/.venv/lib64/python3.11/site-packages/testops_api/rest.py:36: DeprecationWarning: HTTPResponse.getheaders() is deprecated and will be removed in urllib3 v2.1.0. Instead access HTTPResponse.headers directly.
return self.urllib3_response.getheaders()
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
====================================================================================== 4 tests collected in 0.60s =======================================================================================
Processing test result...
TestSuite started: tests/test_calculator.py
Uploading report to TestOps...
TestOps Configuration:
{
"api_key": "*",
"baseline_collection_id": -1,
"build_label": null,
"build_url": null,
"project_id": 5,
"proxy_information": {
"host": null,
"password": "",
"port": -1,
"protocol": "http",
"username": ""
},
"report_folder": "testops-report",
"server_url": "https://testops.katalon.io"
}
An error has occurred in testops-pytest plugin.
The errors I got now are due to the missing api-key and so on.
Will try to find some time to setup also the integration and see what is actually happening, but not right now …
Also, is there anything generated for you into the testops-report folder? should be some json files, I think this is the data the plugin will use to upload the results …
Please check if they have some valid content …
So, I can conclude, this integration works (somehow).
No idea how I can see the actual results of the tests in TestOps (of if something like this is provided) but it worked at least for my case.
So the only idea I have right now for @didit is … make sure your API key and Project ID are properly configured in your testops-config.json file and that the json files under your configured report folder do actually have valid data after you run the tests.
also, make sure that your pytest command (i asked from the early begining of this thread about the full invocation) actually run any test (use collect-only for debug)
Actually, I don’t have a problem with the integration as I explained before, I can run the test case smoothly and get the result inside testops reports, you can also refer to my post attachment, my problem is how to configure some information from the python script so important information can be readable from TestOps Report, eg :
Right now we can see the result, but only for the test suites(file inside test folder with “test” prefix), and the test cases itself, but we didn’t see assertion results or logs even though I already implemented logging
