How to validate the JSON response with expected JSON schema

So, here’s my approach.

First, I add some .jar’s into my project:

To grab them … you can manually search them into Maven central … or do a small trick:
- Create a temporary custom keyword and put only this code into it (I name it Grab):

package utilimport groovy.grape.Grape@Grab(group='org.everit.json', module='org.everit.json.schema', version='1.5.1')public class Grab {}

- Once you hit ‘save’ katalon may looks like is freezing … but fear not. After a while, depending on your internet connection it will be back to normal. Now, navigate into C:\Users\your.user\.groovy folder and you will find there a grapes folder. Collect any .jar downloaded and bring them into your project. Now you can delete this temporary package.

Second, I made a custom keyword like this:

package com.apiimport org.everit.json.schema.Schemaimport org.everit.json.schema.ValidationExceptionimport org.everit.json.schema.loader.SchemaLoaderimport org.json.JSONObjectimport org.json.JSONTokenerimport com.kms.katalon.core.annotation.Keywordimport com.kms.katalon.core.util.KeywordUtilclass EveritValidator {	/**	 * Send request and verify status code	 * @param stringJson Json to validate	 * @param stringSchema validation schema	 */	@Keyword	def verifyJsonSchema(String stringJson, String stringSchema) {		JSONObject rawSchema = new JSONObject(new JSONTokener(stringSchema))		Schema schema = SchemaLoader.load(rawSchema)		try {			schema.validate(new JSONObject(stringJson))			KeywordUtil.markPassed("Valid schema")		} catch (ValidationException e) {			StringBuffer outmessage = new StringBuffer()			outmessage << e.getMessage() << "\n"			e.getAllMessages().each { msg -> outmessage << "$msg \n"}			KeywordUtil.markFailed(outmessage as String)		}	}}

Now, the test case looks as follows (I provide the validation schema from an external file located in ./resources/mySchema.json and I use also an parametrized REST test objectnamed** GETRequest.ContentJSON(url,token)** to send the API request, adapt to your needs):

import static com.kms.katalon.core.testobject.ObjectRepository.findTestObjectimport com.kms.katalon.core.webservice.keyword.WSBuiltInKeywords as WSimport internal.GlobalVariable as GlobalVariable'Given'WS.comment("Setup")reqUrl = "http://your/request/url"schema = new File('./resources/mySchema.json').textrequest = findTestObject('GETRequest.ContentJSON(url,token)', [('url') : reqUrl, ('token') : GlobalVariable.myToken])'When'WS.comment("Sending GET request: ${request.getRestUrl()}")response = WS.sendRequest(request)'Then'WS.comment("Check response status")WS.verifyResponseStatusCode(response, 200)WS.comment("Check response contentType")assert response.isJsonContentType()WS.comment("Validate response schema")responseJson = response.getResponseBodyContent()CustomKeywords.'com.api.EveritValidator.verifyJsonSchema'(responseJson, schema)

If any validation fails, the keyword will collect from the response message all failures and show them into the execution log (also marking the test failed), if passed will just log “valid schema”

Hope it helped!

libs.png

1 Like

Ibus, Thank you so much. I really appreciate your time for the detailed response. We are validating API testing tools. This helps. We are comparing vRest with Katalon.

hi
can someone 'll give me a valid json exemple for the schema ??

i work with local file for schema and
i alwways got an issue :

java.lang.NoClassDefFoundError: com/damnhandy/uri/template/MalformedUriTemplateException

thk

@Joel_ANGEVELLE

take a look at the DRAFT 04 (or 06 / 07 depending on your needs) specs:

Also, some good working examples can be found in the doc of the everit-json library, see:

Some other samples can be found in the jackson doc’s (go on a given example and look for a ‘here’ link with sample codes):
http://java-json-tools.github.io/json-schema-validator/2.2.x/index.html?com/github/fge/jsonschema/examples/package-summary.html

@Ibus I have had some luck with the everit solution you provided above but most of my validations are failing. Can you (or someone) please take a look at what I have done?

Thanks,

Matt

When calling like this I get the following error: def responseJson = group_response.getResponseBodyContent().replace("[", “”).replace("]", “”)

com.kms.katalon.core.exception.StepFailedException: #: expected type: JSONArray, found: JSONObject
#: expected type: JSONArray, found: JSONObject

When calling like this I get the following error: def responseJson = group_response.getResponseBodyContent()

org.json.JSONException: A JSONObject text must begin with ‘{’ at 1 [character 2 line 1]

Here is my code, responses, etc.:

Test Code:

//schema file
def schema = new File('./resources/languageGroupsSchema.json').text
KeywordUtil.logInfo("Expected Schema: " + schema)

//send request
def group_response = WS.sendRequest(findTestObject('Object Repository/API/TransAPI/translations_groups'))

//validate schema
def responseJson = group_response.getResponseBodyContent().replace("[", "").replace("]", "")
CustomKeywords.'com.ws.EveritValidator.verifyJsonSchema'(responseJson, schema)

Keyword:

class EveritValidator {

	/*** Send request and verify status code
	 * * @param stringJson Json to validate	 
	 * * @param stringSchema validation schema	 
	 * */

	@Keyword
	def verifyJsonSchema(String stringJson, String stringSchema) {
		JSONObject rawSchema = new JSONObject(new JSONTokener(stringSchema))
		Schema schema = SchemaLoader.load(rawSchema)
		try {
			schema.validate(new JSONObject(stringJson))
			KeywordUtil.markPassed("Schema is valid - PASSED")
		} catch (ValidationException e) {
			StringBuffer outmessage = new StringBuffer()
			outmessage << e.getMessage() << "\n"
			e.getAllMessages().each { msg -> outmessage << "$msg \n" }
			KeywordUtil.markFailed(outmessage as String)
		}
	}
}

My JSON Response:

[
    {
        "ID": 1,
        "Name": "Warning"
    },
    {
        "ID": 2,
        "Name": “TextArea”
    },
    {
        "ID": 3,
        "Name": "LastActions"
    },
    {
        "ID": 4,
        "Name": "DayOfWeek"
    },
    {
        "ID": 5,
        "Name": "Icons"
    },
    {
        "ID": 6,
        "Name": "Direction"
    },
    {
        "ID": 7,
        "Name": "Tendency"
    },
    {
        "ID": 8,
        "Name": "AirTypes"
    },
    {
        "ID": 9,
        "Name": "UCCText"
    },
    {
        "ID": 10,
        "Name": "Statements"
    }
]

My Schema:

{
    "$schema": "http://json-schema.org/draft-06/schema#",
    "type": "array",
    "items": {
        "$ref": "#/definitions/WelcomeElement"
    },
    "definitions": {
        "WelcomeElement": {
            "type": "object",
            "additionalProperties": false,
            "properties": {
                "ID": {
                    "type": "integer"
                },
                "Name": {
                    "type": "string"
                }
            },
            "required": [
                "ID",
                "Name"
            ],
            "title": "WelcomeElement"
        }
    }
}

I made a demo project on Github: https://github.com/kazurayam/MattThurmanJSONSchema


You shoud use org.json.JSONArray to parse your JSON Response, rather than org.json.JSONObject.

package com.ws

import org.everit.json.schema.Schema
import org.everit.json.schema.loader.SchemaLoader
import org.everit.json.schema.ValidationException
import org.json.JSONArray
import org.json.JSONObject
import org.json.JSONTokener
import com.kms.katalon.core.util.KeywordUtil
import com.kms.katalon.core.annotation.Keyword

class EveritValidator {
	@Keyword
	def verifyJsonSchema(String stringJson, String stringSchema) {
		JSONObject rawSchema = new JSONObject(new JSONTokener(stringSchema))
		Schema schema = SchemaLoader.load(rawSchema)
		try {
			
			//schema.validate(new JSONObject(stringJson))
			schema.validate(new JSONArray(stringJson))
			
			KeywordUtil.markPassed("Schema is valid - PASSED")
		} catch (ValidationException e) {
			StringBuffer outmessage = new StringBuffer()
			outmessage << e.getMessage() << "\n"
			e.getAllMessages().each { msg -> outmessage << "$msg \n" }
			KeywordUtil.markFailed(outmessage as String)
		}
	}
}

this is wrong. you are replacing [] with nothing, building an invalid json.
If the intention was to convert it it to an object, you should replace them with {} (and re-write the schema)
Use json.JSONArray as @kazurayam suggested and discard that replace.

schema.validate(new JSONArray(stringJson))

this is the only line needed to change in the keyword

I had this issue too once and I just made two keywords, one for validating objects and one for validating arrays and using them by-case … but i was too lazy to update the solution i posted :slight_smile:

@kazurayam @Ibus

Thanks!!! I will try these solutions out today!

@kazurayam @Ibus works great…thanks again!

I have a document swagger JSON schema
{
“swagger”: “2.0”,
“info”: {
“description”: “API Testing Practice”,
“version”: “1.0.0”,
“title”: “API Training”,
“termsOfService”: “http://swagger.io/terms/”,
“contact”: {
“email”: "apiteam@swagger.io"
},
“license”: {
“name”: “Apache 2.0”,
“url”: “http://www.apache.org/licenses/LICENSE-2.0.html
}
},
“host”: “reqres.in”,
“basePath”: “/api”,
“tags”: [
{
“name”: “user”,
“description”: “Operations about user”
}
],
“schemes”: [
“https”
],
“paths”: {
“/users”: {
“get”: {
“tags”: [
“user”
],
“parameters”: [
{
“name”: “page”,
“in”: “query”,
“description”: “page number”,
“required”: true,
“type”: “number”
}
],
“summary”: “List users”,
“operationId”: “getAllUsers”,
“produces”: [
“application/json”
],
“responses”: {
“default”: {
“description”: “successful operation”,
“schema”: {
“$ref”: “#/definitions/Users”
}
}
}
}
}
},
“definitions”: {
“Users”: {
“type”: “object”,
“properties”: {
“page”: {
“type”: “number”
},
“per_page”: {
“type”: “number”
},
“total”: {
“type”: “number”
},
“total_pages”: {
“type”: “number”
},
“data”: {
“type”: “array”,
“items”: {
“type”: “object”,
“required”: [
“id”
],
“properties”: {
“id”: {
“type”: “number”
},
“first_name”: {
“type”: “string”
},
“last_name”: {
“type”: “string”
},
“email”: {
“type”: “string”
}
}
}
}
}
}
},
“externalDocs”: {
“description”: “Find out more about Swagger”,
“url”: “http://swagger.io
}
}

I send the request and it returns the data like that

{“page”:2,“per_page”:3,“total”:12,“total_pages”:4,“data”:[{“id”:4,“email”:“eve.holt@reqres.in”,“first_name”:“Eve”,“last_name”:“Holt”,“avatar”:“https://s3.amazonaws.com/uifaces/faces/twitter/marcoramires/128.jpg"},{“id”:5,“email”:“charles.morris@reqres.in”,“first_name”:“Charles”,“last_name”:“Morris”,“avatar”:“https://s3.amazonaws.com/uifaces/faces/twitter/stephenmoon/128.jpg”},{“id”:6,“email”:“tracey.ramos@reqres.in”,“first_name”:“Tracey”,“last_name”:“Ramos”,“avatar”:"https://s3.amazonaws.com/uifaces/faces/twitter/bigmancho/128.jpg”}]}

I tried to use the custom keywords
@Keyword
def verifyJsonSchema(String stringJson, String stringSchema) {
JSONObject rawSchema = new JSONObject(new JSONTokener(stringSchema))
Schema schema = SchemaLoader.load(rawSchema)
try {
schema.validate(new JSONObject(stringJson))
KeywordUtil.markPassed(“Schema is valid - PASSED”)
}
catch (ValidationException e) {
StringBuffer outmessage = new StringBuffer()
outmessage << e.getMessage() << “\n”
e.getAllMessages().each { msg -> outmessage << “$msg \n” }
KeywordUtil.markFailed(outmessage as String)
}
}

The results always run PASS although I change the data type of the JSON key. (Ex: {
“page”: {
“type”: “string”
},…)
Could you please help me check it @kazurayam. Thanks

for the keyword to work, validation schema should follow the IETF spec, e…g for DRAFT04 (default in everit validator) see:
https://tools.ietf.org/html/draft-zyp-json-schema-04

you have also few working examples in the everit doc:

so, on short, you have to translate your swagger schema to a valid DRAFTXX schema

I do not understand what you did. Could you please provided a set of codes which enables me to reporoduce your problem? Please describe what you expect and what actually happens.

Hi @kazarayam. You can see the image that I attached below for more details. Thanks

Hang_Bui,

You wrote

The data type of page is “number” but I tried to change it to “string” and run the test case again. It is PASS.

It will pass. It is not a problem at all.

In terms of JSON Schema, “string” type practically means “anything”. “string” includes any “number”. Therefore your test case will pass.

Hi @Ibus. Could you please tell me more details. How to change the swagger to valid draftxx?. Is there any tool support to convert from swagger file to draftxx

It means that currently I couldn’t compare the data type in the swagger file with the data type from the response request. Is it right? Is there any way to verify it? Thanks

nope. if the value is quoted is string. if not, number or boolean is expected, acc. to draft04 at least.

@Hang_Bui kindly post the content of your schema file. and pleaaase, use code formating

Thanks @kazurayam for your answer

OK, I was not aware how the JSON Schema spec draft04 denotes. But I seem to remember that I have ever encountered some cases where “string” type seemed to mean “anything” — just as @Rams_Palani saw. I would guess that the Everit JSON Schema validator might be a bit buggy. Of course we need to verify it.

i was using everit with exact same scenario, changing from number to string will fail. i have the feeling the OP schema is not complete or does not have the right structure … in some cases, if the schema is not matching with the response (but is a valid json) will just pass everything. e.g use an empty object:

{}

this schema is valid. will pass all

LE: i allways avoided to use refs in the schema. can create confusions, and is easy to do mistakes. i know, is convenient to use it when you have a nice swagger … but i tend to write schema’s from scratch …

LLE: from the pics posted i cannot see what is in the root field. there should be an exact sintax so everit can switch to draft04, 07 etc. if is nothing will default to 04. if is something else … not sure what will happend but it may treat is as empty schema