Repository files navigation

Salve (Schema-Aware Library for Validation and Edition) is a TypeScript librarywhich implements a validator able to validate an XML document on the basis of asubset of Relax NG (RNG). It is developed as part of the Buddhist TranslatorsWorkbench. It can be seen in action inwed.

Salve is used for validating XML with custom Relax NG schemas. We've alsovalidated files that use theTEI standard and theDocBook v5.0 schema. We want to support as much Relax NG as reasonably possible,but salve currently has the following limitations:

• XML Schema typesENTITY andENTITIES are treated as astring.

• None of the XML Schema types that deal with time allow theparametersminInclusive,minExclusive,maxInclusive andmaxExclusive.

• Salve does not verify that numerical values validated asfloat ordouble fit within the limits offloat ordouble. (This is a commonlimitation of validators. We tested withjing andxmllint --relaxngand found they do not raise errors if, for instance, a validation that expectsa float is given a value that cannot be represented with a float.)

If someone wishes to use salve but needs support for any of the features thatare missing, they may ask for the feature to be added. Submit an issue on GitHubfor it. If you do submit an issue to add a feature, please make a case forit. Even better, if someone wishes for a feature to be added, they cancontribute code to salve that will add the feature they want. A solidcontribution is more likely to result in the feature being speedily added tosalve than asking for us to add the feature, and waiting until we have time forit.

A full validation solution has the following components:

• A tokenizer: responsible for recognizing XML tokens, tag names, tagdelimiters, attribute names, attribute values, etc.

• A parser: responsible for converting tokens to validation events (see below).

• A well-formedness checker. Please check theEvents section for moreinformation about what this concretely means.

• A validator: responsible for checking that validation events are valid againsta schema, telling the parser what is possible at the current point invalidation, and telling the parser what is possible in general (e.g., whatnamespace uris are used in the schema).This is what salve offers, and onlythis!

A good example of this division of labor can be found inbin/parse.js and inthe test suite. In both cases the tokenizer function is performed bysaxes,and the parser function is performed by a parser object thatsaxes creates,customized to call salve'sWalker.fireEvent().

Salve has a sister library namedsalve-dom which uses theparsing facilities available in a browser to provide the tokenizer,well-formedness and parser components described above.

NOTE: If you are looking at the source tree of salve as cloned from GitHub, knowthat executables cannot be executed frombin. They can be executed after abuild, from thebuild/dist/bin directory.

If you are looking at the files installed bynpm when you install salve as apackage, the files inbinare those you want to execute.

A typical usage scenario would be as follows:

// Import the validation moduleconst salve = require("salve");// Source is a URL to the Relax NG schema to use. A ``file://`` URL// may be used to load from the local fs.const grammar = salve.convertRNGToPattern(source).pattern;// Get a walker on which to fire events.const walker = grammar.newWalker();

Then the code that parses the XML file to be validated should callfireEvent() onwalker. Remember to call theend() method on yourwalker at the end of validation to make sure that there are no unclosed tags,etc.

The filebin/parse.js (included in salve's source but not in the npm module)contains an example of a rudimentary parser runnable in Node.js::

$ node ....../parse.js [rng] [xml to validate]

The[rng] parameter is the Relax NG schema, recorded in the full XML formatused by Relax NG (not the compact form). The[xml to validate] parameter isthe XML file to validate against the schema.

As you can see above, a Relax NG schema, stored in XML, needs to be converted tosalve's internal format before salve can use it. Internally this happens:

• The XML file recording the Relax NG schema is converted to a tree of objectsrepresenting the XML.

• The XML tree is validated against the Relax NG schema.

• The XML tree is simplified as described in the Relax NG specification.

• Constraints specified by the Relax NG specification are checked.

• The XML tree is converted to a "pattern", which is a structure internal tosalve.

The simplest usage is like this::

const result = salve.convertRNGToPattern(source);

By default the conversion returns an object with a grammar stored on thepattern field, may reveal some simplification warnings on thewarningsfield, and provides a the simplified schema as an XML tree on thesimplifiedfield. In trivial use-case scenarios, onlypattern andwarnings areused.

In some cases, the code usingconvertRNGToPattern may want to serialize theresult of simplification for future use. To do this, it should usewriteTreeToJSON and pass the value of thesimplified field to serializethe simplified XML tree to JSON. The serialized JSON may then be read withreadTreeFromJSON to create a structure identical to the originalpattern. Consider the following code::

const result = salve.convertRNGToPattern(source);const json = salve.writeTreeToJSON(result.simplified);const x = salve.readTreeFromJSON(json);

After executing it,x contains a pattern which represents the same Relax NGschema asresult.pattern.

Do note that thatwriteTreeToJSON takesan XML tree and produces JSON,whereasreadTreeFromJSON reads a JSON and producesa pattern rather thanan XML tree. There is currently no use-case scenario that requires the twofunctions to mirror one-another.

Optionally, you may pass an options object as the 2nd argument ofconvertRNGToPattern withcreateManifest set totrue. If you do this,then you also get amanifest field which is an array of objects containingfile paths and their corresponding hashes. Manifests are useful to allow systemsthat use salve to know whether a pattern needs to be regenerated withconvertRNGToPattern. This is necessary if the system allows a schema tochange after use. Consider the following scenario. Alice uses an XML editor thatuses salve to perform on the fly validation.

1. Alice edits the filefoo.xml with the schemafoo.rng. It so happensthatfoo.rng importsmath.rng. The editor will useconvertRNGToPattern to convertfoo.rng andmath.rng into theformat salve needs. It also useswriteTreeToJSON to cache theresult. Alice sees a brief progress indicator while the editor converts theschema.

2. Over the span of a week, Alice continues editingfoo.xml. Each time sheopens the editor, the editor usesreadTreeFromJSON to load the tree fromcache instead of usingconvertRNGToPattern over and over. As far as Aliceis concerned, the editor starts immediately. There's no progress indicatorneeded becausereadTreeFromJSON is super fast.

3. Alice then changesmath.rng to add new elements. When Alice starts theXML editor to editfoo.xml again, the XML editor must be able to detectthat the schema needs to go throughconvertRNGToPatternagain becausemath.rng has changed.

The manifest is used to support the scenario above. If the XML editor stores theconverted schema, and the manifest into its cache, then it can detect if andwhen it needs to convert the schema anew.

Security note: It is up to you to decide what strength hash you need.Themanifest is not designed for the sake of providing security. So its hashes arenot designed to detect willful tampering but rather to quickly determine whethera schema was edited. In the vast majority of real world usage scenarios, using astronger hash would not provide better security because if an attacker canreplace a schema with their own file, they also can access the manifest andreplace the hash in the manifest.

Salve expects that the events it receives are those that would be emitted whenvalidating awell-formed document. That is, passing to salve the eventsemitted from a document that is malformed will cause salve to behave in anundefined manner. (It may crash. It may generate misleading errors. It may notreport any errors.) This situation is due to the fact that salve is currentlydeveloped in a context where the documents it validates cannot be malformed(because they are represented as DOM trees). So salve contains no functionalityto handle problems with well-formedness. Salvecan be used on malformeddocuments, provided you take care of reporting malformedness issues yourselfand strategize how you will pass events to salve.

Multiple strategies are possible for using salve in a context wherewell-formedness is not guaranteed. There is no one-size-fits-all solutionhere. A primitive parser could abort as soon as evidence surfaces that thedocument is malformed. A more sophisticated parser could process the problematicstructure so as to generate an error but give salve something well-formed. Forinstance if parsing<foo></baz>, such parser could emit an error onencountering</baz> and replace the event that would be emitted for</baz> with the event that would be emitted for</foo>, and salve willhappily validate it. The user will still get the error produced by the parser,and the parser will still be able to continue validating the document withsalve.

The parser is responsible for callingfireEvent() on the walker returned bythe tree created from the RNG. (See above.) The events currently supported byfireEvent() are defined below:

"enterStartTag", [uri, local-name]Emitted when encountering the beginning of a start tag (the string "<tag",where "tag" is the applicable tag name) or the equivalent. The qualifiedname should be resolved to its uri and local-name components.

"leaveStartTag", []Emitted when encountering the end of a start tag (the string ">") orequivalent.

"endTag", [uri, local-name]Emitted when encountering an end tag.

"attributeName", [uri, local-name]Emitted when encountering an attribute name.

"attributeValue", [value]Emitted when encountering an attribute value

"text", [value]Emitted when encountering text. This event must be fired for all instancesof text,including white space. Moreover, salve requires that you fireonetext event per consecutive sequence of text. For instance, if youhave the textfoo bar you may not fire one event forfoo andanother forbar. Or if you have a sequence of lines, you may not fire oneevent per line. You have to concatenate the lines and fire a singletextevent.

Do not generatetext events with an empty string as thevalue. (Conversely, a valid documentmust have anattributeValue forall attributes, even those that have empty text as a value.)

Salve support a couple of compact events that serve to pass as one event datathat would normally be passed as multiple events:

"attributeNameAndValue", [uri, local-name, value]Combines theattributeName andattributeValue events into one event.

"startTagAndAttributes", [uri, local-name, [attribute-data...]]Combines theenterStartTag,attributeNameAndValue andleaveStartTag events. Theattribute-data part of the event must be asequence ofuri, local-name, value as would be passed to withattributeNameAndValue.

For instance if an element namedfoo has the attributea with thevaluevalA, the event would be:"startTagAndAttributes", "", foo, "", "a", "valA".

.. note:: The compact events do not allow salve to be very precise withreporting errors. It is recommended to use them only when optimizingfor speed, at the expense of precision.

.. note:: When reporting possible events, salvenever returns compact eventsin the list.

The reason for the set of events supported is that salve is designed to handlenot only XML modeled as a DOM tree but also XML parsed as a text stringbeing dynamically edited. The best and closest example of this would be whatnxml-mode does in Emacs. If the user starts a new document and types onlythe following into their editing buffer::

<html

then what the parser has seen by the time it gets to the end of the buffer is anenterStartTag event with an empty uri and the local-name "html". The parserwill not see aleaveStartTag event until the user enters the greater-thansymbol ending the start tag.

You must callenterContext() orenterContextWithMapping each time youencounter a start tag that defines namespaces and callleaveContext() whenyou encounter its corresponding end tag. You must also calldefinePrefix(...) for each prefix defined by the element. Example::

<p xmlns="q" xmlns:foo="foons">...

would require calling::

enterContext()definePrefix("", "q")definePrefix("foo", "foons")

Presumably, after the above, your code would callresolveName("p") on yourwalker to determine what namespacep is in, which would yield the result"q". And then it would fire theenterStartTag event withq as thenamespace andp as the local name of the tag::

"enterStartTag", ["q", "p"]

Note the order of the events. The new context must start before salve sees theenterStartTag event because the way namespaces work, a start tag can declareits own namespace. So by the timeenterStartTag is issued, salve must knowwhat namespaces are declared by the tag. If the events were not issued this way,then the start tagp in the example would be interpreted to be in thedefault namespace in effectbefore it started, which could be other thanq. Similarly,leaveContext must be issued after the correspondingendTag event.

Note on performance: if you already have a simple JavaScript object thatmaps prefixes to URIs it is better to callenterContextWithMapping and passyour object to this method.enterContextWithMapping enters a new context andimmediately initializes it with the mapping you pass. This is faster thancallingenterContext and callingdefinePrefix a bunch of times.

For the lazy: it is possible to callenterContext() for each start tag andleaveContext() for each end tag irrespective of whether or not the start tagdeclares new namespaces. The test suite does it this way. Note, however, thatperformance will be affected somewhat because name resolution will have topotentially search a deeper stack of contexts than would be strictly necessary.

Calling thepossible() method on a walker will return the list of validEvent objects that could be fired on the walker, given what the walker hasseen so far. If the user is editing a document which contains only the text::

<html

and hits a function key which makes the editor callpossible(), then theeditor can tell the user what attributes would be possible to add to thiselement. In editing facilities likenxml-mode in Emacs this is calledcompletion. Similarly, once the start tag is ended by adding the greater-thansymbol::

and the user again asks for possibilities, callingpossible() will returnthe list ofEvent objects that could be fired. Note here that it is theresponsibility of the editor to translate what salve returns into something theuser can use. Thepossible() function returns onlyEvent objects.

Editors that would depend on salve for guided editing would most likely need touse theclone() method on the walker to record the state of parsing atstrategic points in the document being edited. This is to avoid needlessreparsing. How frequently this should happen depends on the structure of theeditor. Theclone() method and the code it depends on has been optimizedsince early versions of salve, but it is possible to call it too often,resulting in a slower validation speed than could be attained with lessaggressive cloning.

possible() may at times report possibilities that allow for a documentstructure that is ultimately invalid. This could happen, for instance, where theRelax NG schema usesdata to specify that the document should contain apositiveInteger between 1 and 10. Thepossible() method will report thata string matching the regular expression/^\+?\d+$/ is possible, when infact the number11 would match the expression but be invalid. The softwarethat uses salve should be prepared to handle such a situation.

.. note:: The symbolns used in this section corresponds tourielsewhere in this document andname corresponds tolocal-nameelsewhere. We find theuri,local-name pair to be clearer thanns,name. Isns meant to be a namespace prefix? A URI? Isname a qualified name, a local name, something else? So for thepurpose of documentation, we useuri,local-name wherever wecan. However, the Relax NG specification uses thens,namenomenclature, which salve also follows internally. The name classsupport is designed to be a close representation of what is describedin the Relax NG specification. Hence the choice of nomenclature inthis section.

The term "name class" is defined in the Relax NG specification, please refer tothe specification for details.

Support for Relax NG's name classes introduces a few peculiarities in howpossibilities are reported to clients using salve. The three events that acceptnames are affected:enterStartTag,endTag, andattributeName. Whensalve returns these events as possibilities, their lone parameter is an instanceofname_patterns.Base class. This object has a.match method that takesa namespace and a name and will returntrue if the namespace and name matchthe pattern, orfalse if not.

Client code that wants to provide a sophisticated analysis of what a name classdoes could use the.toObject() method to get a plain JavaScript object fromsuch an object. The returned object is essentially a syntax tree representingthe name class. Each pattern has a unique structure. The possible patterns are:

• Name, a pattern with fieldsns andname which respectively recordthe namespace URL and local name that this object matches. (Corresponds to the<name> element in the simplified Relax NG syntax.)

• NameChoice, a pattern with fieldsa andb which are two nameclasses. (Corresponds to a<choice> element appearing inside a name classin the simplified Relax NG syntax.)

• NsName, a pattern with the fieldns which is the namespace that thisobject would match. The object matches any name. It may have an optionalexcept field that contains a name class for patterns that it should notmatch. The lack ofname field distinguishes it fromName.(Corresponds to an<nsName> element in the simplified Relax NG syntax.)

• AnyName, a pattern. It has thepattern field set toAnyName. Weuse thispattern field becauseAnyName does not require any otherfields so{} would be its representation. This representation would tooeasily mask possible coding errors.AnyName matches any combination ofnamespace and name. May have an optionalexcept field that contains a nameclass for patterns it should not match. It corresponds to an<anyName>element in the simplified Relax NG syntax.

.. note:: We do not use thepattern field for all patterns above because theonly reason to do so would be to distinguish ambiguous structures. Forinstance, if Relax NG were to introduce a<superName> element thatalso needsns andname fields then it would look the same as<name> and we would not be able to distinguish one from theother. However, Relax NG is stable. In the unlikely event a newversion of Relax NG is released, we'll cross whatever bridge needs tobe crossed.

Note that the<except> element from Relax NG does not have a correspondingobject because the presence of<except> in a name class is recorded in theexcept field of the patterns above.

Here are a couple of examples. The name class for::

element (foo | bar | foo:foo) { ... }

would be recorded as (after partial beautification)::

{    a: {        a: {ns: "", name: "foo"},        b: {ns: "", name: "bar"}    },    b: {ns: "foo:foo", name: "foo"}}

The name class for::

element * - (foo:* - foo:a) { ... }

would be recorded as (after partial beautification)::

{    pattern: "AnyName",    except: {        ns: "foo:foo",        except: {ns: "foo:foo", name: "a"}    }}

Clients may want to call the.simple() method on a name pattern to determinewhether it is simple or not. A pattern is deemed "simple" if it is composed onlyofName andNameChoice objects. Such a pattern could be presented to auser as a finite list of possibilities. Otherwise, if the pattern is not simple,then either the number of choices is unbounded or it not a discrete list ofitems. In such a case, the client code may instead present to the user a fieldin which to enter the name of the element or attribute to be created andvalidate the name against the pattern. The method.toArray() can be used toreduce a pattern which is simple to an array ofName objects.

Note that the events returned bypossible() arenot identical to theevents thatfireEvent() expects. While most events returned are exactlythose that would be passed tofireEvent(), there are three exceptions: theenterStartTag,endTag andattributeName events returned bypossible() will have a single parameter after the event name which is anobject ofname_patterns.Base class. However, when passing a correspondingevent tofireEvent(), the same events take two string parameters after theevent name: a namespace URL and a local name. To spell it out, they are of thisform::

event_name, [uri, local-name]

whereevent_name is the string which is the name of the event to fire,uri is the namespace URI andlocal-name is the local name of the elementor attribute.

Error messages that report attribute or element names use thename_patterns.Name class to record names, even in cases wherepatterns.EName would do. This is for consistency purposes, because someerror messagesmust usename_patterns objects to report theirerrors. Rather than have some error messages useEName and some use theobject inname_patterns they all use the objects inname_patterns, withthe simple cases usingname_patterns.Name.

In most cases, in order to present the end user of your application with errormessages that make senseto the user, you will need to process errormessages. This is because error messages generated by salve provide in the errorobject(ns, local name) pairs. A user would most likely like to see anamespace prefix rather than URI (ns). However, since namespace prefixes area matter of user preference, and there may be many ways to decide how toassociate a namespace prefix with a URI, salve does not take a position in thismatter and lets the application that uses it decide how it wants to present URIsto users. The application also has to determine what strategy to use to presentcomplex (i.e., non-simple) name patterns to the user. Again, there is noone-size-fits-all solution.

A problem occurs when validating an XML document that contains an unexpectedelement. In such case, salve will issue an error but then what should it do withthe contents of the misplaced element? Salve handles this in two ways:

1. If the unexpected element is known in the schema and has only one definition,then salve will assume that the user meant to use the element defined in theschema and will validate it as such.

2. Otherwise, salve will turn off validation until the element is closed.

Consider the following case::

<p>Here we have a <name><first>John</first><last>Doe</last></name>because the <emph>person's name</emph> is not known.</p>

Ifname cannot appear inp butname has only one definition in theschema, then salve will emit an error upon encountering theenterStartTagevent forname, and then validatename as if it had been found in avalid place. If it turns out that the schema defines onename element whichcan appear inside aperson element and anothername element which canappear inside alocation element (which would be possible with Relax NG),then salve will emit an error but won't perform any validation insidename. Validation will resume after theendTag event forname. (Future versions of salve may implement logic to figure out ambiguouscases such as this one.) This latter scenario also occurs ifname is notdefined at all by the schema.

The code is documented usingtypedoc. The following command will generatethe documentation::

$ gulp doc

You may need to create agulp.local module to tellgulp where to getrst2html. (Defaults are such thatgulp will use yourPATH to locatesuch tools.) The formatted documentation will appear in thebuild/api/subdirectory, and theREADME.html in the root of the source tree.

NOTE: All the public interfaces of salve are available through thevalidate module. However,validate is a facade that exposes interfacesthat are implemented in separate modules likepatterns andformats.

Whenever you call on salve's functionalities to read a Relax NG schema, thefetch function must be available in the global space for salve to use. OnNode, this means you must load a polyfill to provide this function.

Running salve's testsadditionally requires that the developmentdependencies be installed. Please see thepackage.json file for detailsregarding these dependencies. Note thatgulp should be installed so that itsexecutable is in your path. Either this, or you will have to execute./node_modules/.bin/gulp

If you want to contribute to salve, your code will have to pass the checkslisted in.glerbl/repo_conf.py. So you either have to install glerbl to getthose checks done for you or run the checks through other means. SeeContributing.

The following lists the most prominent cases. It is not practical for us to keeptrack of every single feature that old browsers like IE11 don't support.

• fetch must be present.

• Promise must be present.

• Object.assign must be present.

• URL must be present.

• Symbol [andSymbol.iterator] must be present.

• The String methods introduced by ES6 (includes,endsWith, etc.)

• Array.prototype.includes

• OldSet andMap implementations like those in IE11 are either brokenor incomplete.

On old browsers, we recommend usingcore-js to take care of many of these inone fell swoop. You'll have to provide polyfills forfetch andURL fromother sources.

Note that we do not support old browsers. Notably, salve won't run on anyversion of IE.

Salve uses gulp. Salve's build setup gets the values for its configurationvariables from three sources:

• Internal default values.

• From an optionalgulp.local.js module that can override theinternal defaults.

• From command line options that can override everything above.

The variables that can be set are:

+-----------------------+------------------------------------------------------+|Name | Meaning |+=======================+======================================================+|doc_private | Whether to produce documentation for private || | entities. You can setdoc_private tofalse || | usingno_doc_private. |+-----------------------+------------------------------------------------------+|mocha_grep |--grep parameter for Mocha |+-----------------------+------------------------------------------------------+|rst2html |rst2html command to run |+-----------------------+------------------------------------------------------+

Note that when used on the command line, underscores become dashes, thus--mocha-grep and--doc-private.

Thegulp.local.js file is a module. You must export valueslike this::

exports.doc_private = true

Run::

$ gulp

This will create abuild/dist/ subdirectory in which the JavaScriptnecessary to validate XML files against a prepared Relax NG schema. You couldcopy what is inbuild/dist> to a server to serve these files to a clientthat would then perform validation.

When you install salve throughnpm, you get a package that contains:

• a hierarchy of CommonJS modules inlib,
• a minified UMD build assalve.min.js.

The UMD build can be loaded in a CommonJS environment, in a AMD environment oras "plain scripts" in a browser. If you use the latter, then salve will beaccessible as thesalve global.

Running the following command from the root of salve will run the tests::

$ gulp test

Runningmocha directly also works, but this may run the test against stalecode, whereasgulp test always runs a build first.

Contributions must pass the commit checks turned on inglerbl/repo_conf.py. Useglerbl install to install the hooks. Glerblitself can be found athttps://github.com/lddubeau/glerbl. It will eventuallymake its way to the Python package repository so thatpip install glerblwill work.

writeTreeToJSON converts a Relax NG file formatted in XML into a morecompact format used by salve at validation time. Salve supports version 3 ofthis file format. Versions 0 to 2 are now obsolete. The structure is::

{"v":<version>,"o":<options>,"d":[...]}

Thev field gives the version number of the data. Theo field is a bitfield of options indicating how the file was created. Right now the only thingit records is whether or not element paths are present in the generatedfile. Thed field contains the actual schema. Each item in it is of theform::

[, ...]

The first element,<array type>, determines how to interpret the array. Thearray type could indicate that the array should be interpreted as an actualarray or that it should be interpreted as an object of typeGroup orChoice, etc. If it is an array, then<array type> is discarded and therest of the array is the converted array. If it is another type of object thenagain the<array type> is discarded and an object is created with the restof the array as its constructor's parameters. All the array's elements after<array type> can be JSON primitive types, or arrays to be interpreted asactual arrays or as objects as described above.

Code completely original to salve is released under theMozilla Public Licenseversion 2.0. Copyright 2013-2016 MangalamResearch Center for Buddhist Languages, Berkeley, CA.

The RNG simplification files coded in XSL were adapted fromNicolas Debeissat'scode. Thesefiles were originally released under theCeCILLlicense. Nicolas inMarch2016then changed the license to the Apache License 2.0.

In the version of these files bundled with salve, multiple bugs have beencorrected, some minor and some major, and some changes have been made forsalve's own internal purposes. For the sake of simplicity, these changes arealso covered by the original licenses that apply to Nicolas' code.

Salve is designed and developed by Louis-Dominique Dubeau, Director ofSoftware Development for the Buddhist Translators Workbench project,Mangalam Research Center for Buddhist Languages.

Jesse Bethel has contributed to salve's documentation, and migrated salve'sbuild system from Make to Grunt.

This software has been made possible in part by a Level I Digital HumanitiesStart-up Grant and a Level II Digital Humanities Start-up Grant from theNational Endowment for the Humanities (grant numbers HD-51383-11 andHD-51772-13). Any views, findings, conclusions, or recommendations expressed inthis software do not necessarily represent those of the National Endowment forthe Humanities.