Repository files navigation

JDocs (JSON Documents) is a JSON manipulation library.It completely eliminates the need to have model / POJO classes and instead works directlyon the JSON document. Once you use this library, you may neverwant to return to using Java model classes or using JSON schema for JSON document validation.

Once you have taken a look, if you like what you see, we would very much appreciate a like for the project - it keeps usmotivated knowing that ourwork is getting traction and helping people in the community.

And while you are here, may we mention another of our offerings which could be of interest:

Unify-flowret - A lightweight Java based orchestration engine

https://github.com/americanexpress/unify-flowret

JDocs is available as a jar file in Maven central with the following latest Maven coordinates.

<groupId>com.americanexpress.unify.jdocs</groupId><artifactId>unify-jdocs</artifactId><version>1.9.1</version>

When JSON is used in applications (either to store data or as a means to exchange data betweenapplications / systems), the JSON text document is converted into language specific data structures.For the Java programming language, this consists of Java classes and objects.

For the following JSON:

{"first_name":"Deepak","last_name":"Arora","phones": [     {"type":"home","number":"0134578965"     },     {"type":"mobile","number":"04455678965"     }   ] }

the following Java classes would be created:

publicclassPerson {Stringfirst_name;Stringlast_name;Phone[]phones;}publicclassPhone {Stringtype;Stringnumber;}

These Java classes created are referred to as model classes. The JSON document is parsed and converted intoJava objects which are then used in the program. The process of converting JSON documents intolanguage specific objects is known as ‘unmarshalling" and of converting the language specific objectsback into JSON documents as "marshalling" i.e.

Marshalling -> Java object to JSON document

Unmarshalling -> JSON document to Java object

In an application program, model classes, marshalling and unmarshalling is extensively used to convertJSON into Java objects and vice versa. This approach has the following challenges associated with it:

1. The JSON to Java mapping and vice versa has to be created and maintained. Any time the JSON documentstructure changes, these classes must also be updated as also the programs using these classes.Having model objects creates an additional layer which always needs to be kept in sync with the JSON textdocument structure. The complexity is compounded by the fact that JSON documents can be arbitrary levelsdeep in which case keeping them in sync becomes ever more challenging. This also tightly couples the JSONdocument, model classes and the business logic code making the application difficult to change
2. The problem of code bloat. Typically applications deal with multiple JSON document types.Each JSON document type may map to multiple Java classes. This situation leads to a plethora ofJava classes and wrapper functions written to access fields in these classes. Over timethis leads to code bloat consisting of numerous Java classes with limited value except for reading and writing JSONelements. Also, accessing nested fields / arrays may requires multiple lines of code as traversal offields needs to be done across levels and null values / absent entries dealt with

The consequences over time of the above are:

1. Inability to carry out a fast, accurate and exhaustive impact analysis. For example, where all in the code base isthis json path used?
2. Changing JSON document structure becomes extremely difficult, tedious and error prone. In large code bases,it becomes next to impossible
3. There is an adverse impact on performance leading to higher usage of system resources
4. The code comprehension and readability suffers leading to maintainability issues
5. Finally deterioration in quality, longer turnaround time for changes, higher efforts andultimately higher costs and risks

JDocs has been designed to completely do away with model classes and provide advanced features formanipulating JSON documents directly through JSON paths (with a slight home grown variation).

Doing away with model classes has the following benefits:

1. Reduces the amount of code by ~90% which in turns means significantly faster implementations,reduced effort, improved quality and faster time to market
2. Helps developers concentrate on implementing business logic rather than spending time and effort onmanipulating model classes to access data
3. Simplifies the way code is written and makes it easily comprehensible.To know which elements are being accessed, developers no longer need to go through lines and lines ofcode that only deal with traversing model classes and have very little to do with business logic
4. Allows for fast, accurate and exhaustive impact analysis across the code base. Developers can in a matter ofseconds locate all usages of a json path across the code base
5. By always referring to data as JSON paths, it allows developers to gain a much better understanding ofthe business domain, data and its usage

Lets start with a sample JSON document as below:

{"first_name":"Deepak","last_name":"Arora","is_married":true,"number_of_children":2,"home_address": {"line_1":"XYZ, Greenway Pkwy, #ABC","zip":"85254"  }}

The first step is to get aDocument by so:

Stringjson ="<contents of the example json file>";Documentd =newJDocument(json);

Reading and writing can be done using get and set methods of the API:

Strings =d.getString("$.first_name");// will return DeepakBooleanb =d.getBoolean("$.is_married");// will return trueIntegeri =d.getInteger("$.number_of_children");// will return 2s =d.getString("$.home_address.line_1");// will return "XYZ, Greenway Pkwy, #ABC"

Similarly, set methods of the API are used to set values directly in the JSON document.Let’s say you execute the following commands:

d.setString("$.first_name","John");d.setString("$.last_name","Ryan");d.setString("$.middle_name","Smith");d.setBoolean("$.is_married",false);d.setInteger("$.number_of_children",0);d.setString("$.home_address.zip","85032");Strings =d.getPrettyPrintJson();

The value ofs will now be:

{"first_name":"John","middle_name":"Smith","last_name":"Ryan","is_married":false,"number_of_children":0,"home_address": {"line_1":"XYZ, Greenway Pkwy, #ABC","zip":"85032"  }}

One could also use the following method to get a compressed JSON document:

Strings =d.getJson();

See that the data element "middle_name" which was not existing earlier has been created.The elements which already existed at the path specified have been updated.You can create any arbitrary path in the document by specifying that path in the API method.This can be used to create complex objects or objects inside objects.Consider an empty JSON document on which the following commands are run:

Documentd =newJDocument();d.setString("$.person.first_name","John");d.setString("$.person.last_name","Ryan");d.setString("$.person.middle_name","Smith");d.setString("$.person.address.city","Phoenix");Strings =d.getPrettyPrintJson();

The value ofs will be:

{"person": {"first_name":"John","middle_name":"Smith","last_name":"Ryan","address": {"city":"Phoenix"    }  }}

From the above, note that complex objects person and person.address have automatically been created.

So far so good and you may ask whats so special about this? There are libraries available thatprovide reading and writing of elements using JSON paths. Well, now let's start to make things interesting.

Consider the following JSON document. Lets refer to it as snippet 1:

{"first_name":"Deepak","phones": [    {"type":"Home","number":"123456"    }  ]}

Documentd =newJDocument(json);// assuming json is a string containing snippet 1Strings =d.getString("$.phones[0].type");// will return Homes =d.getString("$.phones[0].number");// will return 123456

Lets make things more interesting! You could refer to the array index by specifying a selection criteria. Aselection criteria is a simplefield=value specification inside of the square brackets. It tells JDocs tolook for an element in thephones array which has a field by the name of type and whose value is "Home":

d.getString("$.phones[type=Home].type");// will return Homed.getString("$.phones[type=Home].number");// will return 123456

A similar construct could be used to set the contents of an array. Consider the following statements:

Documentd =newJDocument(json);// assuming json is a string containing snippet 1d.setString("$.phones[0].number","222222");d.setString("$.phones[0].country","USA");d.setString("$.phones[1].type","Cell");d.setString("$.phones[1].number","333333");d.setString("$.phones[1].country","USA");Strings =d.getPrettyPrintJson();

Would result in the following value ofs:

{"first_name":"Deepak","phones": [    {"type":"Home","number":"222222","country":"USA"    },    {"type":"Cell","number":"333333","country":"USA"    }  ]}

Note that a new element has been created in the array. The same effect could also have been achieved by the following:

Documentd =newJDocument(json);// assuming json is a string containing snippet 1d.setString("$.phones[type=Home].number","222222")d.setString("$.phones[type=Home].country","USA")d.setString("$.phones[type=Cell].number","333333")d.setString("$.phones[type=Cell].country","USA")Strings =d.getPrettyPrintJson();

Note that JDocs when it did not find an array element withtype=Cell, it went ahead and created one.By default, since the value of the fieldtype is not being set explicitly, it assumed the field to be ofStringtype.If you had not wanted that, you could very well have used something like below:

Documentd =newJDocument(json);// assuming json is a string containing snippet 1d.setString("$.phones[type=Home].number","222222")d.setString("$.phones[type=Home].country","USA")d.setInteger("$.phones[type=0].type",0)d.setString("$.phones[type=0].number","333333")d.setString("$.phones[type=0].country","USA")Strings =d.getPrettyPrintJson();

The above would result in the following value ofs:

{"first_name":"Deepak","phones": [    {"type":"Home","number":"222222","country":"USA"    },    {"type":0,"number":"333333","country":"USA"    }  ]}

Now for some interesting scenarios. You may ask what if I do the following?

Documentd =newJDocument(json);// assuming json is a string containing snippet 1d.setInteger("$.phones[type=Home].type",0);Strings =d.getPrettyPrintJson();

Well, JDocs will try and search for the element with field type having a value Home and will implicitlyhandle different data types both for searching and writing. In the case of the above,even though the field was stored asString type, JDocs converted it to a number.The following will be the value ins:

{"first_name":"Deepak","phones": [    {"type":0,"number":"123456"    }  ]}

Now what if you were to do the following?

Documentd =newJDocument();// creating an empty documentd.setInteger("$.phones[1].type",0);// specifying array index as 1 without the element at 0 being presentStrings =d.getPrettyPrintJson();

In this case, JDocs would throw an out of bounds exception as below:

com.americanexpress.unify.jdocs.UnifyException:Arrayindexoutofbounds->phones

JDocs realized that an element was being attempted to be added at an index which did not have an elementat the previous index and hence disallowed that by throwing an exception. In case an array index is specified,JDocs will only add an array element if the previous element exists or if the element index has value 0.If however, a selection criteria of the form oftype=value was specified, and if no element existedwhich had a field named type with the given value, it would then have created an element at the end of the array.

You could use this same notation to create new arrays, create complex objects within arrays,create arrays within arrays etc. An example of this is below:

Documentd=newJDocument(json);// assuming json is a string containing snippet 1d.setString("$.addresses[0].type","Home")d.setString("$.addresses[0].line_1","Greenway Pkwy")Strings=d.getPrettyPrintJson();

Nows will have the following value:

{"first_name":"Deepak","phones": [    {"type":"Home","number":"123456"    }  ],"addresses": [    {"type":"Home","line_1":"Greenway Pkwy"    }  ]}

Important points to keep in mind about array creation:

In case array indexes are used:

1. In case the index exists, the element in that index will be updated
2. In case the index does not exist and the index value is 0, an array element will be created
3. In case the index does not exist and the index value is greater than 0, an elementwill be created only if the index is one greater than the maximum index in the existing array

In case an array selection criteria is used:

1. If an element exists which contains the field and value as specified in the selection criteria,that element will be updated
2. If no element has a field as specified in the selection criteria, a new array element willbe created with a field set to the value specified in the criteria

Now, specifying array indexes hard coded as above is fine. But in the real world, you do not know the numberof elements that an array may have and so you need a way to find out the number of elements in thearray and to be able to traverse over all the elements, read them and them and maybe set fields in themas you go along.

Consider the following JSON sample. Lets call it snippet 2.

{"first_name":"Deepak","phones": [    {"type":"Home","number":"222222","country":"USA"    },    {"type":"Cell","number":"333333","country":"USA"    }  ]}

You can find the size of an array as below:

Documentd =newJDocument(json);// assuming json is a string containing snippet 2intsize =d.getArraySize("$.phones[]");// will contain the value 2

Note that in the above, empty square brackets was specified with phones. This is required so asto inform JDocs that phones is an array in the JSON document.

Now traversing over the array is trivial. But note the technique used closely. You would not want to deviatefrom this technique for the following reasons specified below.

CAUTION!!! deviating from this technique will negate one of the most important benefit of JDocswhich is the ability to carry out an exhaustive impact analysis of any JSON path used in the codebase.Lets see the traversal first. Continuing from the above Java snippet:

Documentd =newJDocument(json);// assuming json is a string containing snippet 2intsize =d.getArraySize("$.phones[]");// will contain the value 2for (inti =0;i <size;i++) {Stringindex =i +"";Stringtype =d.getString("$.phones[%].type",index);Stringnumber =d.getString("$.phones[%].number",index);Stringcountry =d.getString("$.phones[%].country",index);System.out.println(type);System.out.println(number);System.out.println(country);}

will print:

Home222222USACell333333USA

Setting the elements while traversing is on exactly similar lines as below:

Documentd =newJDocument(json);// assuming json is a string containing snippet 2intsize =d.getArraySize("$.phones[]");// will contain the value 2for (inti =0;i <size;i++) {Stringindex =i +"";d.setString("$.phones[%].type","Cell",index);d.setString("$.phones[%].number","111111",index);d.setString("$.phones[%].country","USA",index);}for (i =0;i <size;i++) {Stringindex =i +"";Stringtype =d.getString("$.phones[%].type",index);Stringnumber =d.getString("$.phones[%].number",index);Stringcountry =d.getString("$.phones[%].country",index);System.out.println(type);System.out.println(number);System.out.println(country);}

will print:

Cell111111USACell111111USA

Now lets discuss why deviating from the above technique is not be a good idea. See the code below:

Documentd =newJDocument(json);// assuming json is a string containing snippet 2intsize =d.getArraySize("$.phones[]");// will contain the value 2for (inti =0;i <size;i++) {Stringindex =i +"";Stringtype =d.getString("$.phones[" +index +"].type");System.out.println(type);}

Note that while the above code will work, notice two things:

1. The line containing the JSON path has been split into 2 lines
2. The construction of the JSON path is now split using +

What either of the above points mean is that it is no longer possible in an IDE to be able to useregular expression searching for a JSON path in the codebase. Earlier, a regularexpression"\$\.phones\[.*\].type" could be used to search for all occurrences of this JSON path acrossthe code base but now that ability is lost. To be able to retain this ability is absolutely of crucialimportance as that is what one needs when something is changed in the JSON structure. One needs toknow where all it is used across the codebase.

Further more is also reduces code comprehension as the the JSON path is no longer visibleas a running continuous string. Unfortunately, this is one thing which cannot be controlled usingautomation and the message just has to be drilled across and reviews conducted on code.

Many of the API methods which take JSON path as input use variable lenght string arguments so as tobe able to specify the JSON path as a running continuous string. The % symbol is used between squarebrackets which is replaced in sequence by the variable arguments.

The API provides a method to delete paths in the document.The path may be a leaf node or may point to a complex object or an array or an array element.

Consider the following JSON snippet 3:

{"first_name":"Deepak","national_ids": {"ssn":"2345678","dl_number":"DL1234"  },"phones": [    {"type":"Home","number":"222222","country":"USA"    },    {"type":"Cell","number":"333333","country":"USA"    }  ]}

The method is straightforward as below:

Documentd =newJDocument(json);// assuming json is a string containing snippet 3d.deletePath("$.first_name");// will delete the first_name fieldd.deletePath("$.national_ids");// will delete the whole complex object national_idsd.deletePath("$.phones[]");// will delete the whole phones arrayd.deletePath("$.phones[0]");// will delete the first element of the arrayd.deletePath("$.phones[type=Home]");// will delete that element of the array which has a type field equal to Home value

The method also takes variable length argument so as to be able to use it for arrays like so:

Documentd =newJDocument(json);// assuming json is a string containing snippet 3d.deletePath("$.phones[type=%]","Home");// will delete that element of the array which has a type field equal to Home valued.deletePath("$.phones[%]",0 +"");// will delete the first element

Till now we talked of operations which can be done on a free form JSON document meaning that it ispossible to read and write pretty much any path we feel like. But what if we wanted to lock down the structure of a JSONdocument? Typically, we would use something like JSON schema. We thought that there was a far simpler andmore intuitive way to do the same thing. This is where we now start to talk of Base and Typed documents.

In JDocs, there can be two types of documents. A Base document which we have worked with so farabove and a Typed document. A Base document is one that has a free structure i.e.any element can be written at any path using the API.A Typed document is one that has a defined structure that needs to be adhered to.We define this structure in something called as a Model document.In other words, a Model document locks the structure of a JSON document to a specific one.Further, for each leaf element, a Model also defines the constraints in terms of field type, format etc.Every typed document is associated with a model document.

A model documents is also a JSON document that contains all valid paths for a data document.Against each leaf node path, it specifies the constraints applicable for that element.

In case of arrays, the model document contains only one element and constraints definedin this one element are applicable for all elements of the array in the data document. Whileit is true that JSON documents can have different data types for the same field across elements, butif you really think about it, for the purpose of defining a structure, each element reallyneeds to be of the same type.

Lets understand this using an example. Consider the snippet 4 below:

{"first_name":"Deepak","start_date":"01-28-2020 21:18:10 America/Phoenix","id":12345,"phones": [    {"type":"Home","number":111111    },    {"type":"Cell","number":222222    }  ]}

The model for the above JSON document would be defined as below:

{"first_name":"{\"type\":\"string\"}","start_date":"{\"type\":\"date\",\"format\":\"MM-dd-uuuu HH:mm:ss VV\"}","id":"{\"type\":\"integer\",\"regex\":\".*\"}","phones": [      {"type":"{\"type\":\"string\"}","number":"{\"type\":\"integer\",\"regex\":\".*\"}"      }   ]}

Loading model documents

It is mandatory for a model to be loaded before a typed document of that type is created.Model documents can be loaded using the following:

JDocument.loadDocumentTypes(type,json);

String type specifies the type of the documentString json specifies the content of the model document

Creating typed documents

Documentd =newJDocument("model",null);

The above creates an empty document which is tied to a model document named asmodel(expected to be already loaded as described in the above section).

Strings ="...";// s contains the contents of a JSON documentDocumentd =newJDocument("model",s);

The above creates a typedJDocument from an existing JSON document stored in the strings.JDocs, while loading this document, will run the validations on the document against the modeland if the structure / constraints do not match, the appropriate exception will be thrown.

Also, when writes to the document are done using setXXX methods,the structure and constraints will be validated against the model.For example, for snippet 4 above, the following calls will succeed as the paths and theconstraints on the elements are all valid:

Documentd =newJDocument("model",null);d.setString("$.first_name","Deepak1");d.setString("$.phones[0].type","Home");d.setInteger("$.phones[0].number",333333);

Whereas the following will fail:

d.setString("$.first_name1","Deepak1")// incorrect pathd.setString("$.start_date","15-Apr-2019")// incorrect date formatd.setString("$.phones[0].number","111111")// incorrect data type

As regards constraints, the validation is done against the specified regular expression. If a match returns true, thevalidation is assumed to succeed else an exception is thrown.

The following attributes can be specified as part of a constraint on a path:

Below is the model for the above specifications:

{"type":"{\"type\":\"string\"}","null_allowed":"{\"type\":\"boolean\"}","regex":"{\"type\":\"string\"}","ignore_regex_if_empty":"{\"type\":\"boolean\"}","format":"{\"type\":\"string\"}","empty_date_allowed":"{\"type\":\"boolean\"}","min_length":"{\"type\":\"integer\"}","max_length":"{\"type\":\"integer\"}","min_value":"{\"type\":\"integer|long|decimal\"}","max_value":"{\"type\":\"integer|long|decimal\"}","min_date":"{\"type\":\"string\"}","max_date":"{\"type\":\"string\"}"}

Validating typed documents

Typed documents may be validated when:

1. They are constructed
2. At the time of specifying the type of a base document (setType method)
3. When calling thevalidateAllPaths orvalidateModelPaths methods
4. When calling thegetXXX orsetXXX methods on the document e.g.getString /setString

There are three types of validations mechanism that can be specified:

CONSTS_JDOCS.VALIDATION_TYPE.ALL_DATA_PATHS

All paths found in the document will be validated against the model document.

CONSTS_JDOCS.VALIDATION_TYPE.ONLY_MODEL_PATHS

Only those data paths for which model paths exists will be validated against the model document.

CONSTS_JDOCS.VALIDATION_TYPE.ONLY_AT_READ_WRITE

Validations will be performed only when a data path is read or written by the use ofgetXXX andsetXXX methods.

The validation typesCONSTS_JDOCS.VALIDATION_TYPE.ONLY_MODEL_PATHSandCONSTS_JDOCS.VALIDATION_TYPE.ONLY_AT_READ_WRITEare used in scenarios where we may be getting extra fields / paths from an external entity (think an API response) andthese paths are not part of our model document. In such cases, we can look to validate the paths which are found in themodel document or validate them only at read and write of paths.

This is a typical scenario in the use of APIs which can return extra blocks and paths which may not be present in themodel document and which we may not be interested in. If we were to validate at the time of creating the document, themodel validations would fail unless we kept the model document in sync with all the changes happening on the API side.Most times, this is not possible as the teams are separate and there is no reason that adding fields to theresponse which we are not interested in should cause a failure.

The validation mechanism can be set at the whole of JDocs level or at a per-document level. To set the validationmechanism at a JDocs level, initialize JDocs by specifying the validation type in theinit method as below. Thisvalidation type will come into effect for all typed documents unless a validation type has been specified explicitly fora document:

public static void init(CONSTS_JDOCS.VALIDATION_TYPE validationType)

If theinit method is called without any parameters, thenCONSTS_JDOCS.VALIDATION_TYPE.ALL_DATA_PATHS is assumeddefault.

We can also specify the validation type independently at the document level. Assuming that we have called the defaultinit method (with no parameters), lets say that for only one document in a particular instance, we wanted to useCONSTS_JDOCS.VALIDATION_TYPE.ONLY_MODEL_PATHS validation mechanism. We could override JDocs default validationmechanism by specifying the validation while constructing the document as below:

Document d = new JDocument(type, json, CONSTS_JDOCS.VALIDATION_TYPE.ONLY_MODEL_PATHS);

or while setting the type of a base document as below:

d.setType(type, CONSTS_JDOCS.VALIDATION_TYPE.ONLY_MODEL_PATHS);

Validating documents against different models

Both base and typed documents can be validated against different models. This scenario come in handy when there maybe different validations required for data originating from different use cases. For example, if we were to collectcustomer information for name or address, there may be a need to have different validations for data collected fordifferent countries. The special characters that could be included for US may be different from what could be includedfor France. In order to accomplish this, we would define multiple model documents that may only be used for validation.The following validation methods allow us to do such validations.

public void validateAllPaths(String type);

public void validateModelPaths(String type);

Given a document, it is possible to extract content from it as a new document. Consider the snippet below:

{"id":"id","family": {"number_of_members":2,"members": [         {"name":"Deepak Arora","phones": [               {"type":"Home","number":"1111111111"               },               {"type":"Cell","number":"2222222222"               }            ]         },         {"name":"Nitika Kaushal","phones": [               {"type":"Home","number":"3333333333"               },               {"type":"Cell","number":"4444444444"               }            ]         }      ]   }}

We can extract content like below:

Documentd=newJDocument(json);// assuming json is a string containing above snippetDocumentd1=d.getContent("$.family.members[1].phones[0]",false,true);Strings=d1.getPrettyPrintJson();

The value ofs will contain:

{"family" : {"members": [      {"phones": [          {"type":"Home","number":"3333333333"          }        ]      }    ]  }}

Note that the third parameter (includeFullPath) is specified astrue. This means that return the full path in thedocument. Had we setthat tofalse, the following would have been returned:

{"type":"Home","number":"3333333333"}

The path specified in this method has to point to either:

• a complex object
• an array element (which also needs to be a complex object)
• an array (likephones[]). In this case, theincludeFullPath parameter has to be false else the APIwill throw an exception

In case the document we are extracting content from is aTypedDocument, then we have the option of returningaBaseDocument or aTypedDocument. This is specified using the second parameterreturnTypedDocument. If thisparameter is set totrue and if the document from which content is being extracted is aTypedDocument, thenthe return document will also be aTypedDocument of the same type. In case the document we are extracting content fromis not aTypedDocument, this parameter is ignored.

Of course, if aTypedDocument is being returned, its needs to conform to the structure of the model document. Inthis situation, if we specifyincludeFullPath = false, it is possible that the returned document when constructedwill not conform to the model document in which case the API will throw an exception.

As with other methods in the API, the path can contain% and the value specified in the last variable argumentsparameter.

Given two Jdoc documents, it is possible to copy content from one document to another.Using this functionality, one can copy complex objects from one document to another.A complex object means any object in a document that is not a leaf node i.e.it itself contains objects. A complex node may also be an array or an element of an array.For reading and writing leaf elements, the get and set API is meant to be used.

Copy contents is also applicable on typed documents, in which case, all "from" paths and their values arevalidated against the model and constraints and exceptions thrown in case of errors encountered.

Consider the following as snippet 5:

{"first_name":"Deepak","phones": [    {"type":"Home","number":111111    }  ]}

and the following as snippet 6:

{"first_name":"Deepak","addresses": [    {"type":"Home","number":111111    }  ]}

The following is am example of copying content:

Documentto =newJDocument(json);// assuming json is a string containing snippet 5Documentfrom =newJDocument(json1);// assuming json1 is a string containing snippet 6to.setContent(from,"$.addresses[]","$.addresses[]");Strings =to.getPrettyPrintJson();

The strings will contain the following:

{"first_name":"Deepak","phones": [    {"type":"Home","number":111111    }  ],"addresses": [    {"type":"Home","number":111111    }  ]}

For typed documents, JDocs implements a unique concept of array keys.An array key is a field in the array element which is unique across all elements in the array.Think of it as an array having this field as its primary key.Also, this key field is expected to be present in all elements in the array.

Consider the following snippet 6:

{"first_name":"Deepak","phones": [    {"phone_type":"Home","number":"222222","country":"USA"    },    {"phone_type":"Cell","number":"333333","country":"USA"    }  ]}

Here, the type field in each of the elements of the phones array can be used as an array key.This field is defined in the model document using the reserved keyword "jdocs_arr_pk" as below:

{"first_name":"{\"type\":\"string\"}","phones": [    {"jdocs_arr_pk":"{\"field\":\"phone_type\"}","phone_type":"{\"type\":\"string\"}","number":"{\"type\":\"string\"}","country":"{\"type\":\"string\"}"    }  ]}

Hence, for elements in arrays in typed documents, "jdocs_arr_pk" becomes a reserved value and cannot be usedto define a field name.

The concept of array key fields is used while merging typed documents into one another asdescribed in the next section. This is one of the most powerful features of JDocs.

Using this feature, you can merge the contents of one typed document into another. For discussing theconcept of merging, a "from" document and a "to" document is used. Both "from" and "to"documents need to be typed documents having the same model.

Lets take snippet 6 as the starting point for understanding. Consider the following code:

Documentto =newJDocument("model",fromJson);// fromJson contains snippet 6 jsonDocumentfrom =newJDocument("model",null);// create an empty document of type modelfrom.setString("$.phones[0].phone_type","Office");from.setString("$.phones[0].number","888888");from.setString("$.phones[0].country","USA");to.merge(from,null);Strings =to.getPrettyPrintJson();

A couple of points to note here:

1. The second parameter ofmerge is a variable string argument and can be used to specify pathswhich need to be deleted in the to document before the merge is carried out
2. If array elements are being merged, JDocs expects the key field to be specified. This is requiredso that JDocs can look up the array element in the to document to merge the from contents into. If nosuch array element is found in the to document, a new array element with the specified key is created

JDocs will throw an exception in case of mismatches encountered during merge, for example,the data type of the "to" path may not match with that of the "from" pathor the path types may not match i.e. one path may be an array while the other may be anobject or a leaf node. In case any such case is encountered, an exception is thrown andthe operation aborted.

CAUTION -> In case of an exception encountered, whatever merges have been carried out into the from document willremain so and the document will not be reverted back. Ideally if both are typed documents, this should not occur. Eventhen if the application wants the document to be reverted, it should first create a copy of the to document, try themerge into that first and then if it succeeds, carry it out in the first document.

This feature of merge is used extensively in scenarios where the main document is given to clients as a read onlydocument and along with it, an empty document (called a fragment) is provided. Whatever the client has to update intothe main, it writes into the fragement and returns the fragment to the caller. The caller then merges the fragment intothe main document. This gives programs the ability to record incremental updates to a document and later build mechanismto play them back to see how the updates occurred.

Consider a scenario where there is a loan processing application document and its associated model. Assume thata service is called to the contents of the application document have to be passed in one of theJSON fields of the service request. In such a scenario, when a model document for the service request is created,the contents of the model document of the application document have to be embedded in it. Imagine if wehad many more such cases and kept embedding application document everywhere as required. Thiswould lead to unnecessary code bloat and making any change to the application model documentwould require changes at all places where the application model document was embedded. To avoidthis, the @here feature is used.

Consider the following application model document:

{"first_name":"{\"type\":\"string\"}","last_name":"{\"type\":\"string\"}","phones": [    {"jdocs_arr_pk":"{\"field\":\"phone_type\"}","phone_type":"{\"type\":\"string\"}","number":"{\"type\":\"string\"}","country":"{\"type\":\"string\"}"    }  ]}

Now consider that there is astoreApplication REST service which persists the contents of the documentto a database. This service has it own request JSON model document and the contents of the applicationare present in a field. Without the use of @here feature, the model document for tthe service requestwould look like:

{"request_id":"{\"type\":\"string\"}","application": {"first_name":"{\"type\":\"string\"}","last_name":"{\"type\":\"string\"}","phones": [      {"jdocs_arr_pk":"{\"field\":\"phone_type\"}","phone_type":"{\"type\":\"string\"}","number":"{\"type\":\"string\"}","country":"{\"type\":\"string\"}"      }    ]  }}

Note that the contents of the first json are embedded in the path "$.application".

Using @here, the model document for the service request can be written as:

{"request_id":"{\"type\":\"string\"}","application": {"@here":"/common/docs/models/application.json"   }}

The @here specifies a file path in the resources folder from where the file contents are read and inserted intothe main model document. This eliminates duplication and allows for making the change only at one place.

Searching for an array element

The below snippet will search the document for an array element in the phonesarray where the field type has the value home. Note that in order to use this API, the final elementneeds to be an array in which a selection criteria is specified. Also note that the selection criteriacan refer to any field in the element and the index of the match of the first occurrence will be returned. Incase it is required to read further, an iteration as described previously is recommended.

Documentd =newJDocument(s);// s contains a valid JSON stringinti =0;intindex =d.getArrayIndex("$.applicants[%].phones[type=home]",i +"");

Working with array values

JSON notation also supports array values as below:

{"valid_codes": ["V1","V2","V3"  ]}

JDocs provides a different set of read and write methods to work with such construct. These methodsare similar to getXXX and setXXX methods and are listed below:

BooleangetArrayValueBoolean(Stringpath,String...vargs);IntegergetArrayValueInteger(Stringpath,String...vargs);StringgetArrayValueString(Stringpath,String...vargs);LonggetArrayValueLong(Stringpath,String...vargs);BigDecimalgetArrayValueBigDecimal(Stringpath,String...vargs);voidsetArrayValueBoolean(Stringpath,booleanvalue,String...vargs);voidsetArrayValueInteger(Stringpath,intvalue,String...vargs);voidsetArrayValueLong(Stringpath,longvalue,String...vargs);voidsetArrayValueBigDecimal(Stringpath,BigDecimalvalue,String...vargs);voidsetArrayValueString(Stringpath,Stringvalue,String...vargs);

The model document for the above would be defined as:

{"valid_codes": ["{\"type\":\"string\"}"  ]}

Flattening a JSON document

Consider the following JSON snippet:

{"id":"id_1","family": {"members": [      {"sex":"male","first_name":"Deepak","last_name":"Arora","number_of_dependents":3      }    ]  }}

The flatten API of JDocs gives us a list of all the paths present in the document with or without values.The following gets the list of paths without values:

Documentd =newJDocument(json);// assuming json is a string containing above snippetList<String>paths =d.flatten();paths.stream.forEach(s ->System.out.println(s));

The above code will print the following:

$.id$.family.members[0].sex$.family.members[0].first_name$.family.members[0].last_name$.family.members[0].number_of_dependents

In case we wanted to get the flattened paths and also the values, we would use theflattenWithValues API as below:

Documentd =newJDocument(json);// assuming json is a string containing above snippetList<PathValue>paths =d.flatten();paths.stream.forEach(pv ->System.out.println(pv.getPath() +", " +pv.getValue() +", " +pv.getDataType()));

The above code will print the following:

$.id, id_1, string$.family.members[0].sex, male, string$.family.members[0].first_name, Deepak, string$.family.members[0].last_name, Arora, string$.family.members[0].number_of_dependents, 3, integer

Comparing two JSON documents

We can use thegetDifferences API to compare one JSON document with another. Consider the followingJSON documents:

JSON snippet 1:

{"id":"id_1","family": {"members": [      {"first_name":"Deepak"      }    ]  },"cars": [    {"make":"Honda","model":null    }  ],"vendors": ["v1","v2"  ]}

JSON snippet 2:

{"id":"id_2","family": {"members": [      {"first_name":"Deepak"      },      {"first_name":"Nitika"      }    ]  },"vendors": ["v1","v3"  ]}

The comparison can be done as below:

Documentld =newJDocument(jsonLeft);// assuming jsonLeft is a string containing above snippet 1Documentrd =newJDocument(jsonRight);// assuming jsonRight is a string containing above snippet 2List<DiffInfo>diList =ld.getDifferences(rd,true);Strings ="";for (DiffInfodi :diList) {Stringlpath = (di.getLeft() ==null) ?"null" :di.getLeft().getPath();Stringrpath = (di.getRight() ==null) ?"null" :di.getRight().getPath();s =s +di.getDiffResult() +", " +lpath +", " +rpath +"\n";}System.out.println(s);

The above would print:

DIFFERENT, $.id, $.idONLY_IN_LEFT, $.cars[0].make, nullDIFFERENT, $.vendors[1], $.vendors[1]ONLY_IN_RIGHT, null, $.family.members[1].first_name

Of course, if we wanted to also get the values, we could always usegetValue method ofPathValueobject returned from the calldi.getLeft() ordi.getRight. We could also use the methodgetDataType in case we wanted to get the data type of the value.

Please note the following regarding comparing JSON documents:

1. JDocs does a logical comparison of the documents. When fields with null values are encountered, theyare treated as being equivalent to the field not being present in the document. In the above example,in the left document,$.cars[0].model has null value while this path is not present in the right document.JDocs comparison assumes that these are equivalent. In other words a JSON path leaf node having a null valuesis assumed to be the same as the path not existing at all.
2. If any one of the documents being compared is a typed document, the data type of the path will be determinedfrom the model document

1. Ability to write data to JSON documents using JSON paths for fields AND complex objects AND arrays
2. A very simple and easy to use feature of binding the structure of a JSON document to a modelwhich can then be used to enforce the integrity of the structure during read and writes
3. Ability to define regex constraints on fields in the model which are used to validate the field values at the time ofwriting
4. Ability to merge documents into one another including merging complex objects and arrays nested any levels deep
5. Ability to reuse and embed models
6. Ability to copy content from one document to another using JSON paths
7. And most of all freedom from model classes and code bloat

The core API can be referred to in the fileDocument.java.

Go through the unit test cases in the source code. Unit test cases are available in the locationsrc/test

Provide us feedback. We would love to hear from you.

Deepak Arora, GitHub: @deepakarora3, Twitter: @DeepakAroraHi

We welcome Your interest in the American Express Open Source Community on Github. Any Contributor toany Open Source Project managed by the American Express Open Source Community must accept and signan Agreement indicating agreement to the terms below. Except for the rights granted in thisAgreement to American Express and to recipients of software distributed by American Express, Youreserve all right, title, and interest, if any, in and to Your Contributions. Pleasefill out the Agreement.

Any contributions made under this project will be governed by theApache License 2.0.

This project adheres to theAmerican Express Community Guidelines. Byparticipating, you are expected to honor these guidelines.