This output is formated in the “Markdown” format.
You would need a viewer for *.md file.
Katalon Studio does not support rendering Markdown files.
I would recommend to use Visual Studio Code with Markdown plugin.
With VSCode, you can preview any Markdown documents, like this
I have ever developed a more comprehensive solution for myself.
This enabled me to organize a mass of downloaded web resources (screenshot images, html, json, xml, etc) in a custom domain-specific database on the local file system. It can compare 2 sets of web resources (2 sets downloaded from a single host at different timing, or 2 sets downloaded from different hosts, such as the development environment and the productional environment). It can compile a diff report with rich meta data.
The following screenshot shows a sample image of the report.
There is no point into diffing json files, IMHO (and with any other structured data files, e.g. xml)
Altough is doable as @kazurayam already shown, I will keep this method usage for files ‘not easy to parse’.
I will better use json schema validation at first, to check if the main structure of the response is fine, followed by a certain data validation (parse the json, iterate over it, compare with whatever input data)
Why am I saying this? Well, let’s take a look at the following example:
File A
{
key1: value1,
key2: value2
}
File B
{
key2: value2,
key1: value1
}
If you diff them you will get a failed test.
However, both have the same schema, and from the data point of view, both of them are valid.
may be nice for you but not for me, particularily with json.
objecs are not sorted, and depending on the serialization method used under the hood, with each call you may get a slightly different file.
i know plenty api’s working like this, e. g Jira api.
so using diff with json will only get false negative
you call this a defect? or lazy programing? it is not.
the code is optimized to use async processing, so this can occur particularily with large data
the data is valid, so the code is robust. if you use the data wrong, the issue is on the tester/consumer side
one more challenge for you, with json.
what if you have a paged endpoint?
good luck to diff such to validate the response
Comoon, @kazurayam …
Are you able to distinguish between a json array and a json object?
i will expect more from your side …
The example provided by you is not valid, nor by schema, nor by data.
So it will fail from the schema validation at a minimum
as you didn’t read my explanations.
json is a structured data. make no sense to diff it.
you will get only fake validation even if the sample used for refference is malformed
bye, and good luck with testing.
why to attempt to validate a malformed json? only you may know the answer.
if it is malformed, the AUT is already screwed. period!
so it is waste of time to proceed further with data validation
If it is written (and tested) by monkeys . Or, perhaps, by ChatGPT.
Are you aware that, you can check also with an API response the type of the data content received?
(which should be the previous check, prior to schema validation and further)
or should i go even further and say that, before any check, you should check the response code?
so you are suggesting that, the tester should spend time and effort to write voodoo ‘validation’ code because the developper of the AUT is lazy or dumb and is not using proper serialization / de-serialization libraries from the early beginning?
think twice…
i dont understand your point bionel.
There are many times I have visually validated a response from an API. It has 30 fields. I want to check when the API returns something different. as this could mean there is a new bug.
In my case, the order of the JSON always comes in the same. Kazarayems code seems to be very useful. I am excited to try. Kaz, do you use Visual studio code to program tests in Katalon? I would love to do this!
ofcourse you don’t.
otherwise you wouldn’t rise this topic.
looks like i am the only one on this comunity understanding data structures and how to properly handle it, from the early begining of the development to validation.
good luck with your approaches and have fun when you will have to deal with serious AUTs consuming really large datasets.
(30 fields is a joke)
so kindly mark kazu suggestion as a solution to close this topic
LE: don’t take this as a passive/aggressive answer, despite it looks like.
it is just an advice from my side.
you can ignore it.
We can not develop the codes (Test Case, Keyword) of a Katalon project in VSCode. We can execute Test Cases only inside Katalon Studio (or Katalon Runtime Engine). Katalon Test Cases never run inside VSCode.
However, VSCode is usefull to do various operations that Katalon Studio does not support. For example
VSCode lets me open the Terminal UI. KS doesn’t. In the Terminal, I operate the git command. I operate the gradle command quite often.
VSCode lets me open a *.html file as raw text and let me edit it.
Katalon Studio doesn’t let me open a *.html file as raw text. When I double-click a index.html file stored in the project somewhere, Katalon always open it inside the browser. Therefore in Katalon Studio, I am not able to edit a file of which name ends with .html. I am not joking.
VSCode can render the preview of text files in Markdown format and in Asciidoc format. VSCode can render the diagrams created by the PlantUML language. I use these formats for authoring documentations.
Text editors for *.html, *.js, *css, *.json, *.xml … of VSCode are superb. Color themes, indentation, auto-formatting, code suggestions, spell-checking — all in VSCode are just comfortable. Katalon’s editor is (as you know) as dum as Notepad.
So I employ VSCode quite often to supplement Katalon Studio.
I have a dream that one day in the VSCode window, I will be able to develop the source code of my web app and my Katalon tests together as one united project.
Just my 2cents, but I tend to agree with @bionel here. The solution to do a diff might be valid for some users in some use cases (e.g. static API responses), but if you are serious on verifying complex json responses, diffs are not the best solution.
Besides the points already mentioned, most of the time, API responses will have items like GUUID’s or dateTime stamps, which of course will always be different each time you generate it. Proper json deserialization (JsonSlurper e.g.) can then help to assert in a more robust way. Katalon does have some “Verification” code snippets, but this is all very basic in my opinion. I use a dedicated API testing tool to be more efficient on this and we use Katalon as more dedicated Front End testing tool.
I won’t name the other tool, we are at Katalon forum, but here is just a screen to show how an api JSON response can be visualized with quick click actions to do property content assertions & jsonPath matches (might help Katalon devs to see how current webservice tests can be improved).
These types of assertions need to be used in combo with the HTTP response code check, the schema validation, SLA reponse times checks etc. like Bionel mentioned. The assertion can also be expanded with more scripting, like in the use case where the sorting in the response is not always guaranteed.
My post is in no way any mean of invalidating anyone else his/her opinion. This is just my view from years of experience on API testing, so please don’t take any offense on this as it is not intended this way…
Nobody should take it as an offense … I hope.
We are engineers here, with different points of view.
And we came from different fields.
As long as the topics are not taken personally, we are good
Debates … well, will always be, and this is the purpose of the comunity.
To exchange experience and visions. And mistakes.
I would like to propose my solution to @bionel’s point. See the following
I will quote a relevant section from this article.
ex43 pretty print JSON then diff
Let me assume I have 2 JSON instances:
FileA:
{
"key1":"data1",
"key2":"data2"
}
FileB:
{
"key2":"data2",
"key1":"data1"
}
These 2 JSON instances are semantically identical.
If I apply take textual diff of the FileA and FileB, I will get the following diff report.
This diff report is arguable. This report tells me that there are textual differences. However, if I do not like to be disturbed by the differences which are semantically insignificant, the above report is too noisy.
Now I would apply the methodology “ex42 pretty print JSON while ordering Map entities by keys”. I would convert the 2 input JSON and then take the diff.
The following report shows the result.
The following Test Case creates these 2 diff reports.
In the latter diff report, both JSON are converted by the com.kazurayam.ks.JsonPrettyPrinter class to have the same order of Map entries by keys. Eventually the report clearly shows that FileA and FileB are similar.
The original post by @philippe.novikov was a bit ambiguous. He wrote 2 different requirements mixed up.
He wants to verify a single complex JSON response.
He wants to compare 2 JSON instances to find the difference between the two
These 2 requirements deserve solutions for each.
@bionel and @joost.degeyndt suggested a solution for the 1st requirement. They proposed validating a JSON instance against a JSON schema. This article introduces the networknk json validator. Katalon Studio v8.6.8 bundles its jar. It is ready to use in KS.
I suggested a solution for the 2nd requirement.
Both solutions are possible in Katalon Studio. Now it’s the turn of the original poster to decide which way to take, or even both.
