Repository files navigation

shex.js javascript implementation of Shape Expressions (try online)

npm install --save shex

There are two ways to run tests. You can run the default tests for whichever branch you have checked out (including main):

npm checkout shex-nextnpmtest

or you can clone shexSpec/shexTest next to your shex.js clone:

(cd ..&& git clone https://github.com/shexSpec/shexTest --branch extends)npmtest

Thetest harness first looks for a sibling shexTest repo and if it doesn't find it, usesnode_modules/shexTest.

test runsmocha -R dot (thedot reporter because there are around three thousand tests).

There are slower tests (command line interface, HTTP, etc) which you can run with theSLOW=<timeout in milliseconds> environment variable set. For the HTTP tests you will have to specifiy a git repository in$BRANCH, e.g.SLOW=10000 BRANCH=main TEST-cli=true'npm test

Theshex.js repo includes several branches for features that are in-flight in the ShEx Community Group. NPM@shexjs/* packages are published from theshex-next repo. Each of these repos depends on some branch of the test suite. The package.json file for each branch SHOULD have that corresponding shexTest branch à la:

"shex-test":"shexSpec/shexTest#extends"

If you are running tests from the automatically checked out shexTest module, you'll have tonpm install every time you change branches. If you are running from a sibling clone of shexTest, you'll have to cd to that sibling and checkout the branch which corresponds to the shex.js branch you have checked out.

There is a post-commit hook which will probably whine at you if they are misaligned, though it will simply fail to test some features if e.g. shexTest is on main while shex.js is on extends.

You can validate RDF data using thebin/validate executable or thelib/ShExValidation library described below.

Validate something in HTTP-land:

./node_modules/shex/bin/validate \    -x http://shex.io/examples/Issue.shex \    -d http://shex.io/examples/Issue1.ttl \    -s http://shex.io/examples/IssueShape \    -n http://shex.io/examples/Issue1

That validates nodehttp://shex.io/examples/Issue inhttp://shex.io/examples/Issue1.ttl against shapehttp://shex.io/examples/IssueShape inhttp://shex.io/examples/Issue.shex.The result is a JSON structure which tells you exactly how the data matched the schema.

{"type":"test","node":"http://shex.io/examples/Issue1","shape":"http://shex.io/examples/IssueShape","solution": {...  }}

Had we gotten anull, we'd know that the document was invalid with respect to the schema.See theShExJ primer for a description of ShEx validation and theShExJ specification for more details about the results format.

validate's -n and -s arguemtns are evaluated as IRIs relative to the (first) data and schema sources respectively.The above invocation validates the node<Issue1> inhttp://shex.io/examples/Issue1.ttl.This and the shape can be written as relative IRIs:

./node_modules/shex/bin/validate \    -x http://shex.io/examples/Issue.shex \    -d http://shex.io/examples/Issue1.ttl \    -s IssueShape \    -n Issue1

Parsing from the old interwebs involves a painful mix of asynchronous callbacks for getting the schema and the data and parsing the data (shorter path below):

varshexc="http://shex.io/examples/Issue.shex";varshape="http://shex.io/examples/IssueShape";vardata="http://shex.io/examples/Issue1.ttl";varnode="http://shex.io/examples/Issue1";varhttp=require("http");varshex=require("shex");varn3=require('n3');// generic async GET function.functionGET(url,then){http.request(url,function(resp){varbody="";resp.on('data',function(chunk){body+=chunk;});resp.on("end",function(){then(body);});}).end();}varSchema=null;// will be loaded and compiled asynchronouslyvarTriples=null;// will be loaded and parsed asynchronouslyfunctionvalidateWhenEverythingsLoaded(){if(Schema!==null&&Triples!==null){console.log(newshex.Validator(Schema).validate(Triples,node,shape));}}// loaded the schemaGET(shexc,function(b){// callback parses the schema and tries to validate.Schema=shex.Parser(shexc).parse(b)validateWhenEverythingsLoaded();});// load the dataGET(data,function(b){// callback parses the triples and tries to validate.vardb=n3.Store();n3.Parser({baseIRI:data,format:"text/turtle"}).parse(b,function(error,triple,prefixes){if(error){throwError("error parsing "+data+": "+error);}elseif(triple){db.addQuad(triple)}else{Triples=db;validateWhenEverythingsLoaded();}});});

See? That's all there was too it!

OK, that's miserable. Let's use the ShExLoader to wrap all that callback misery:

constshexc="http://shex.io/examples/IssueSchema";// schema locationconstdata="http://shex.io/examples/Issue1";// data locationconstnode="http://shex.io/examples/Issue1#Issue1";// node in that dataconstN3=require("n3");constShExLoader=require("@shexjs/loader")({// initialize with:fetch:require('node-fetch'),//   fetch implementationrdfjs:N3,//   RdfJs Turtle parser});const{ctor:RdfJsDb}=require('@shexjs/neighborhood-rdfjs');const{ShExValidator}=require("@shexjs/validator");ShExLoader.load({shexc:[shexc]},{turtle:[data]}).then(function(loaded){vardb=RdfJsDb(loaded.data);varvalidator=newShExValidator(loaded.schema,db,{results:"api"});constsmap=[// array of node/shape pairs{node:node,//   JSON-LD @id for nodeshape:ShExValidator.Start}//   schemas's start shape]varresult=validator.validateShapeMap(smap);// success if no "errors"console.log(JSON.stringify(result,null,2));});

Note that the shex loader takes an array of ShExC schemas, either file paths or URLs, and an array of JSON schemas (empty in this invocation) and an array of Turtle files.

As with validation (above), you can convert by either executable or library.

ShEx can be represented in the compact syntax

PREFIX ex: <http://ex.example/#><IssueShape> {                       # An <IssueShape> has:    ex:state (ex:unassigned            # state which is              ex:assigned),            #   unassigned or assigned.    ex:reportedBy @<UserShape>        # reported by a <UserShape>.}

or in JSON:

{"type":"schema","start":"http://shex.io/examples/IssueShape","shapes": {"http://shex.io/examples/IssueShape": {"type":"shape","expression": {"type":"eachOf","expressions": [          {"type":"tripleConstraint","predicate":"http://ex.example/#state","valueExpr": {"type":"valueClass","values": ["http://ex.example/#unassigned","http://ex.example/#assigned"          ] } },          {"type":"tripleConstraint","predicate":"http://ex.example/#reportedBy","valueExpr": {"type":"valueClass","reference":"http://shex.io/examples/UserShape" }          }] } } } }

You can convert between them with shex-to-json:

./node_modules/shex/bin/shex-to-json http://shex.io/examples/Issue.shex

and, less elegantly, back with json-to-shex.

As with validation, the ShExLoader wrapes callbacks and simplifies parsing the libraries:

varshexc="http://shex.io/examples/Issue.shex";varshex=require("shex");shex.Loader.load({shexc:[shexc]},null).then(function(loaded){console.log(JSON.stringify(loaded.schema,null,"  "));});

There's no actual conversion; the JSON representation is just the stringification of the parsed schema.

Command line arguments which don't start with http:// or https:// are assumed to be file paths.We can create a local JSON version of the Issues schema:

./node_modules/shex/bin/shex-to-json http://shex.io/examples/Issue.shex> Issue.json

and use it to validate the Issue1.ttl as we did above:

./node_modules/shex/bin/validate \    -j Issue.json \    -d http://shex.io/examples/Issue1.ttl \    -s http://shex.io/examples/IssueShape \    -n http://shex.io/examples/Issue1

Of course the data file can be local as well.

Happy validating!

Materialize is used to transform from a source schema to a target schema after validation is done.

The syntax is:

materialize`-t<target schema>`|-h [-j`<JSON Vars File>`] [-r`<RDF root IRI>`]

Materialize reads the output from the validate tool from STDIN and maps it to the specified target schema.

If supplied, a JSON vars file will be referenced to fill in constant values not specified from the source.This is useful in assigning default fields to the target when there is no equivalent value in the source schemaand source data.

Here is an example of a simple JSON vars file:

{"urn:local:Demographics:constSys":"System",}

If this vars file content is used, then any time a variable in the target file withvalue "urn:local:Demographics:constSys" is seen, the value "System will be substituted.

The RDF root IRI specifies the root node from which all nodes in the schema will descend.The default root if none is specified is:tag:eric@w3.org/2016/root

Here are some examples:

materialize -h

validate -x source_schema.shex -l data.jsonld -s ProblemShape| materialize -t target_schema.shex -j vars.json

cat problem.val| materialize -t target_schema.shex -j vars.json -r http://hl7.org/fhir/shape/problem

1. open a browser window (we'll callvalidator) withhttps://rawgit.com/shexSpec/shex.js/main/doc/shex-simple.html
2. open another browser window (we'll callviewer) withhttps://shex.io/shexTest/main/viewer?validation
3. wait 'tillviewer loads and look for "3circRefS1-IS2-IS3-IS3" (near the bottom)
4. drag the "#3circRefS1-IS2-IS3-IS3" cell (or the ✓ to the left of it) to the right of the QueryMap area ofvalidator
5. click on the long label under "Manifest:", then the long label under "Passing:" and validate.

It should validate, which involves the IMPORT of3circRefS2-IS3 and3circRefS3.3circRefS2-IS3 also IMPORTs3circRefS3 which shows that IMPORT is idempotent (has a side effect only the first time).

This repo useslerna to manage multiple NPM packages. These packages are located inpackages/*:

• shape-map -- aShapeMap parser
• @shexjs/parser -- parse ShExC into ShExJ
• @shexjs/writer -- serialize ShExK as ShExC
• @shexjs/term -- RDF terms uses in ShEx
• @shexjs/util -- some utilities for transforming schemas or validation output
• @shexjs/visitor -- avisitor for schemas
• @shexjs/validator -- validate nodes in an RDF graph against shapes in a schema!-@shexjs/eval-simple-1err -- eval-simple-1err
• @shexjs/eval-threaded-nerr -- eval-threaded-nerr
• @shexjs/loader -- an API for loading and using ShEx schemas
• @shexjs/node -- additional API functionality for a node environment
• @shexjs/cli -- a set of command line tools for transformaing and validating with schemas
• @shexjs/webapp -- the shex-simple WEBApp
• @shexjs/shape-path-query -- traverse ShEx schemas with a path language
• @shexjs/extension-test -- a small language for testing semantic actions in ShEx implementations (more)
• @shexjs/extension-map -- an extension for transforming data from one schema to another (more)
• @shexjs/extension-eval -- simple extension which evaluates Javascript semantic action code (more)

Here are some commands you'll need:

• building/testing
  • lerna bootstrap -- look for all packages inpackages/*
  • npm ci -- to install root node_modules per package-lock.json
  • lerna list -- in case you ever wonder what packages flashed past your eyes
• adding an NPM package (promise-worker) to one of our managed packages (webapp)
  • lerna add promise-worker --scope=@shexjs/webapp
• remove a package you aren't using it after all:
  • edit e.g. the package.json (e.g. packages/shex-webapp/package.json) to remove the dependency
  • lerna bootstrap --scope @shexjs/webapp --no-ci --force-local (why)
  • if it's in node_modules,npm remove promise-worker
• adding a dev package
  • shex.js followsthe advice of lerna docs to "hoist" all dev dependencies to the root package (hoist:true in lerna.config). Lerna moves devDeps required in more than one package (e.g. the webpack deps in the webapp and extension-map packages) to the root so they won't appear in e.g. packages/extension-map/node_modules.
  • in principle, this should work:lerna add -dev pseudo-worker --scope=@shexjs/webapp
  • but it doesn't seem to so instead:(cd packages/shex-webapp && npm install --save-dev pseudo-worker)