Repository files navigation

This repository provides Python components that implement key servicesfor the NIST Publishing Data Repository (PDR) platform. Python isused primarily for implementing the PDR's publishing services, andthis repository provides Version 2 (and higher) implementations builton Python 3.

This repository is one of the successors of the oar-pdr software,v1.4.7. The python parts of that software was built on Python 2.7.

This repository introduces a major revision to the python code withthe following goals:

• Migrate the code to run under Python 3
• Organize modules according to an updated architecture supportingmultiple publication channels
• Integrate with a new oar-pdr2 that combines the multi-languageimplementations into a single repository via language-basedsubmodules.

For deeper history of the python code, consult the git logs foroar-pdr v1.4.7.

python       --> Python source code for the metadata and preservation                  servicesscripts      --> Tools for running the services and running all testsoar-build    --> general oar build system support (do not customize)oar-metadata --> Python source code for supporting the NERDm (and                  related) metadata, provided as a submoduledocker/      --> Docker containers for building and running tests

The publishing services are built and run using Python 3 (supportingversions 3.6 through 3.7).

The oar-metadata package is a prerequisite which is configured as gitsub-module of this package. This means after you clone the oar-pdr gitrepository, you should usegit submodule to pull in the oar-metadatapackage into it:

git submodule update --init

See oar-metadata/README.md for a list of its prerequisites.

In addition to oar-metadata and its prerequisites, this package requiresthe following third-party packages:

• multibag-py v0.4 or later
• bagit v1.6.X
• fs v2.X.X

As an alternative to explicitly installing prerequisites to runthe tests, thedocker directory contains scripts for building aDocker container with these installed. Running thedocker/run.shscript will build the containers (caching them locally), start thecontainer, and put the user in a bash shell in the container. Fromthere, one can run the tests or use thejq andvalidate tools tointeract with metadata files.

This repository currently provides one specific software product:

• pdr-publish -- the publishing services

As a standard OAR repository, the software products can be built by simply viathemakedist script, assuming the prerequisites are installed:

  scripts/makedist

The built products will be written into thedist subdirectory(created by themakedist); each will be written into a zip-formattedfile with a name formed from the product name and a version string.

The individual products can be built separately by specifying theproduct name as arguments, e.g:

  scripts/makedist pdr-publish

Additional options are available; use the-h option to view thedetails:

  scripts/makedist -h

Assuming the prerequisites are installed, thetestall script can beused to execute all unit and integration tests:

  scripts/testall

Like withmakedist, you can run the tests for the different productsseparately by listing the desired product names as arguments totestall. Runningtestall -h will explain available command-lineoptions.

The Python build tool,setup.py, is used to build and test thesoftware. To build, type while in this directory:

  python setup.py build

This will create abuild subdirectory and compile and install thesoftware into it. To install it into an arbitrary location, type

  python setup.py --prefix=/oar/home/path install

where/oar/home/path is the path to the base directory where thesoftware should be installed.

Themakedist script (in../scripts) will package up aninstalled version of the software into a zip file, writing it out intothe../dist directory. Unpacking the zip file into a directory isequivalent to installing it there.

To run the unit tests, type:

  python setup.py test

Thetestall.python script (in../scripts) will runsome additional integration tests after running the unit tests. Inthe integration tests, the web service versions of the services arelaunched on local ports to test for proper responses via the webinterface.

Like all standard OAR repositories, this repository supports the useof Docker to build the software and run its tests. (This method isused at NIST in production operations.) The advantage of the Dockermethod is that it is not necessary to first install theprerequisites; this are installed automatically into Dockercontainers.

To build the software via a docker container, use themakedist.docker script:

  scripts/makedist.docker

Similarly,testall.docker runs the tests in a container:

  scripts/testall.docker

Like their non-docker counterparts, these scripts accept product namesas arguments.

docker-compose upexport PYTHONPATH=$PATH_TO_LIBexport MONGO_TESTDB_URL="mongodb://admin:admin@127.0.0.1:27017/midas?authSource=admin"python3 setup.py buildpython3 test/nistoar/midas/dbio

Thescripts directory containsWSGI applications scripts.

This software was developed by employees and contractors of theNational Institute of Standards and Technology (NIST), an agency ofthe Federal Government and is being made available as a publicservice. Pursuant to title 17 United States Code Section 105, works ofNIST employees are not subject to copyright protection in the UnitedStates. This software may be subject to foreign copyright.Permission in the United States and in foreign countries, to theextent that NIST may hold copyright, to use, copy, modify, createderivative works, and distribute this software and its documentationwithout fee is hereby granted on a non-exclusive basis, provided thatthis notice and disclaimer of warranty appears in all copies.

THE SOFTWARE IS PROVIDED 'AS IS' WITHOUT ANY WARRANTY OF ANY KIND,EITHER EXPRESSED, IMPLIED, OR STATUTORY, INCLUDING, BUT NOT LIMITEDTO, ANY WARRANTY THAT THE SOFTWARE WILL CONFORM TO SPECIFICATIONS, ANYIMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULARPURPOSE, AND FREEDOM FROM INFRINGEMENT, AND ANY WARRANTY THAT THEDOCUMENTATION WILL CONFORM TO THE SOFTWARE, OR ANY WARRANTY THAT THESOFTWARE WILL BE ERROR FREE. IN NO EVENT SHALL NIST BE LIABLE FOR ANYDAMAGES, INCLUDING, BUT NOT LIMITED TO, DIRECT, INDIRECT, SPECIAL ORCONSEQUENTIAL DAMAGES, ARISING OUT OF, RESULTING FROM, OR IN ANY WAYCONNECTED WITH THIS SOFTWARE, WHETHER OR NOT BASED UPON WARRANTY,CONTRACT, TORT, OR OTHERWISE, WHETHER OR NOT INJURY WAS SUSTAINED BYPERSONS OR PROPERTY OR OTHERWISE, AND WHETHER OR NOT LOSS WASSUSTAINED FROM, OR AROSE OUT OF THE RESULTS OF, OR USE OF, THESOFTWARE OR SERVICES PROVIDED HEREUNDER.