This is a companion discussion topic for the original entry at https://docs.katalon.com/katalon-studio/docs/schema-validation.html
This is a companion discussion topic for the original entry at https://docs.katalon.com/katalon-studio/docs/schema-validation.html
2 Likes
âSchema Validationâ ---- is it a correct English expression? I doubt it.
This is the beginning of a great feature!
What is the underlying validation library/engine being used? Are there any known limitations on which aspects of the standard which are not supported?
Will this feature evolve further to validate responses versus the API spec rather than schema snippets? This seems to be a very hot topic among QA practitioners.
In a .classpath file generated by KS v8.3.1-alpha I found a line
<classpathentry kind="lib" path="/Applications/Katalon Studio.app/Contents/Eclipse/plugins/com.networknt.json-schema-validator_1.0.66.jar"/>
This would suggest KS is using this library:
The above âREAMDEâ document has a section titled âKnown issueâ.
1 Like
A JSON Schema describes a set of format rules for a json document; no more. Still it is useful enough.
I donât think so.
If you want to validate your JSON document against some predefined âSemantic Constraintsâ (what you would call âThe API specâ), I think you would need something like Schematron for XML, for example
What I mean is to validate a service response against the service definition/OAS file. The method proposed on this page requires taking the definition and manually breaking it down into schema snippets that can be used for validation.
As it is the feature is useful but would much more compelling if it was further elaborated to validate the response against the service definition. It could be leveraged for provider or consumer contract testing.
I do not know what âOASâ means. Google search for âOASâ gave me some links unrelated to IT. Could you give us any references about âOASâ you mean?
@rburgos I donât think what do you ask will be ever supported ⌠since nobody here knows what OAS means.
I suppose it is something what your developers are using internally to define the specs.
So ⌠that will be chaotic to implement since each developers may use something else.
Some may use yaml files, some may use json DRAFT schema, some may use whatever the analyst dream over night.
DRAFT specs are IETF standards and libraries implementing validation against it already exist and are pretty well maintained.
With a bit of practice, you will be able to write quite complex schemes from large specs in ~10 mins.
If you need a custom solution to validate against whatever specs, you will have to code your own solution (or pay somebody to do it for you)
OAS is common terminology when working on services but I should have spelled out for clarity. âOASâ refers to the OpenAPI Specification, typically version 3x. Version 2 was also known as Swagger â but both terms continue to be used interchangeably.
Also some elaboration on the question might help. Katalon supports creating test objects (object repository) from the OAS spec automatically. It then allows us to quickly put together tests based on âweb service requestsâ. These tests should align with functionality and acceptance criteria, etc.
When following a âdesign-firstâ pattern the OAS spec is created early, quick mocks are available to the team to interact, and the team uses the design as the guide for the rest of implementation. In this scenario, it is very useful to provide early feedback and perhaps discover issues (fail-fast/adapt-fast). It also helps to detect early if there is any drift between what the spec calls (expectation) and what the developers are creating.
This is where âtesting to the specâ or âprovider contract testingâ comes into picture and it is simple: 1) you make the call to the service, 2) the service responds, 3) you validate the response conforms with the service definition as defined by itâs OpenAPI specification âcontractâ.
An additional advantage of âtesting to the specâ for QA team is that they do not need to create tests for things that are already expressed on the contract (data types, lengths, formats, ranges, enums, etc). Thus saving time and avoiding repetitive boiler plate type tests.
An alternative being considered is switching to use other products such as ReadyAPI (example: https://support.smartbear.com/readyapi/docs/testing/assertions/reference/compliance/swagger.html) but it would be much preferable to continue to use a single tool for testing services as well as UI and other areas within Katalon.
The feature as created requires that we take the OAS definition, manually break it apart, re-format to be standalone, save it separately, and then use on validation.
Because the design process is highly collaborative and iterative in nature, repeating these steps could become an issue an errors could be introduced.
May be common for you but not for me.
As long as DuckDuckGo does not offer me this in the very first results, I am not bothering to do more research.
As for OpenAPI ⌠ok, now we are talking engineering and letâs read the docs of the underlying library used (benefit to @kazurayam which had patience to look for relevant hints into the implementation):
quoting:
OpenAPI Support
The OpenAPI 3.0 specification is using JSON schema to validate the request/response, but there are some differences. With a configuration file, you can enable the library to work with OpenAPI 3.0 validation.
So, for now, I donât think it is considered yet for implementation, however, feel free to raise a feature request.
At this moment, this feature it is still in POC but nothing is stopping you to explore and try to tweak it or develop your own similar solution.
Thanks for the quick reply â the feature on its current state would need significant additional elaboration to achieve the stated goals, but it is usable with the mentioned manual workarounds.
When I started this thread I wasnât aware I was requesting a new feature, I thought I didnât know how to locate it as it is now available in other software offerings and I was hoping for someone to give me quick pointer 
I appreciate everyoneâs input. Cheers!
For now letâs see what we will get when a âstable versionâ is released.
I will support you to have also such additional feature if you raise a request, since I think it will be very helpful.
We are commenting now into the documentation section, which is not considered usually for the dev-team roadmap, however will need also a lot of love.
And ⌠let the guys do their job âŚ
One step at a time âŚ