- Notifications
You must be signed in to change notification settings - Fork0
Integrated Pipeline Reports (IPR) API, the reporting API as part of the PORI platform
bcgsc/pori_ipr_api
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
IPR is part of theplatform for oncogenomic reporting and interpretation.
The Integrated pipeline reports API manages data access to the IPR database on iprdb01.bcgsc.ca.The API is responsible for providing all data for GSC genomic, probe reports, and Germline Small Mutation reports.
After cloning the repository, the application's dependencies need to be installed:
npm install
This process can take between 1-5 minutes depending on caches, and connection speed. Once NPM has finished installingall required dependencies, the server can be started with the following command:
NODE_ENV=[local|development|production|staging|test] npm start
If NPM is not available, the application server can be booted by executing through node directly:
NODE_ENV=[local|development|production] node bin/www
This repository contains configuration profiles for production, testing, and development environments. A profile will beselected based on the NODE_ENV environment setting, or by explicitly calling --env [production|development] wheninitializing the server. The configuration values are set with defaults inapp/config.js, whichcan be overridden with environment variables.
Environment Variables are expected to be prefixed withIPR_ and separated by underscores. For example,to override the database hostname you might use the following
export IPR_DATABASE_HOSTNAME=someTestServerNameThis would then be converted to
{"database": {"hostname":"someTestServerName"}}Database settings and credentials can also be given via command line arguments which follow the patternof--<section>.<setting>. For Example
app.js --database.hostname someTestServerName
Unit and Integration tests are run and written using Jest + Supertest with code coverage reports generated using Clover. Tests are configured to run using thetest environment variable - this currently cannot be overridden.
To run unit tests, cd into the project root directory and run the commandnpm run test_local. Once completed, it should generate and print summaries for the tests and their coverage. The database user credentials and API user credentials must be set before tests can be run.
npm runtest -- --database.password someDbPassword --testing.password someApiPasswordor with environment variables
export IPR_DATABASE_PASSWORD=someDbPasswordexport IPR_TESTING_PASSWORD=someApiPasswordnpm runtest
Regarding destructions of prop records after tests (e.g.: afterAll(async () => {await db.models.report.destroy({where: {id: report.id}, force: true});})), "force: true" will not pass locally but will pass on GitHub tests. This property ensures that prop records are deleted after the tests finish running.
While developing or running tests you might notice that changes that are done to the code are not being reflected in the API response. Or, you might notice that tests are returning not expected results. It might be worth checking if it's a cache issue.
In that case, you can comment out the cache code when testing manually. For unit tests, you can modify the query in order to avoid triggering the cache seeexample.
Developer documentation is generated using the JSDoc library. To generate a local copy of the documentation, cd into the root of the project directory and run the commandnpm run jsdoc. This should automatically create documentation within folder named 'jsdoc' that can be viewed in a web browser.
⚠️ WARNING: Releases which include database migrations should also include an update of the demo db dump. Seeinstructions on how to create and commit the new dump.
- Create a migration:
npx sequelize migration:create --name name_of_migration - Write up and down functions in your migration file
- According to your changes in migration file, change your Sequelize models manually
- Run:
npx sequelize-cli db:migrateor
export IPR_SERVICE_PASS=<PASSWORD>npx sequelize-cli db:migrate --url"postgres://ipr_service:$IPR_SERVICE_PASS@iprdevdb:5432/ipr-sync-dev"
Sequelize Migration Docs:http://docs.sequelizejs.com/manual/migrations.html
To avoid breaking the standard development setup or having your migrations overwritten by the synccrons. You can create a temporary copy of the database as follows
pg_dump -Fc -U ipr_service -h iprdevdb.bcgsc.ca -d ipr-sync-dev> ipr-sync-dev.dumpThat creates the dump file. Then create the new temp database. Add temp or the ticket number to the db name to make itobvious later that it can be deleted
createdb -U ipr_service -h iprdevdb.bcgsc.ca DEVSU-777-temp-ipr-sync-dev
This then needs to be restored as a new database.
WARNING: DO NOT RESTORE TO THE PRODUCTION DB SERVER
pg_restore -Fc -U ipr_service -h iprdevdb.bcgsc.ca ipr-sync-dev.dump -d DEVSU-777-temp-ipr-sync-dev
Finally connect to the newly created database
psql -h iprdevdb.bcgsc.ca -U ipr_service -d DEVSU-777-temp-ipr-sync-dev
Once you are done with testing, delete the temporary database
dropdb -U ipr_service -h iprdevdb.bcgsc.ca DEVSU-777-temp-ipr-sync-dev
When adding/inserting entries directly into a database table without using the API, be sure to update the primary key (id)sequence by running this SQL command after the insert:
SELECT setval('table_name_id_seq', (SELECT MAX(id) FROM "table_name"));
Otherwise, you will get a unique constraint error when inserting via the API because Sequelize will try use an idthat is now taken.
The production installation of IPR is run & managed bypm2
To interact with pm2, ssh to the server and cd to/var/www/ipr/api/[production]. From here, run pm2 commands to get information about the application.
<user>@<server>:/var/www/ipr/api/production$ pm2 list┌────────────────────┬─────┬──────┬───────┬────────┬─────────┬────────┬─────┬───────────┬──────────┐│ App name │ id │ mode │ pid │ status │ restart │ uptime │ cpu │ mem │ watching │├────────────────────┼─────┼──────┼───────┼────────┼─────────┼────────┼─────┼───────────┼──────────┤│ IPR-API │ 101 │ fork │ 23409 │ online │ 0 │ 23h │ 0% │ 92.7 MB │ disabled │└────────────────────┴─────┴──────┴───────┴────────┴─────────┴────────┴─────┴───────────┴──────────┘ Use `pm2 show <id|name>` to get more details about an appIt is possible to use pm2 to actively monitor the console of the applications by usingpm2 monit.
Note: The pm2 daemon will sometimes launch a new instance of pm2 when running pm2 commands as opposed to accessing the currently running version of pm2. You can tell if there are multiple instances running by executing the commandps aux | grep pm2
[<user>@<server> ~]$ ps aux | grep pm2user 13787 0.1 1.2 952276 50004 ? Ssl Jul20 9:28 PM2 v3.0.0: God Daemon (/home/user/.pm2)user 13800 0.1 1.5 1224456 61028 ? Ssl Jul20 9:49 node /home/user/.pm2/node_modules/pm2-logrotate/app.jsuser 18162 0.0 0.0 103312 856 pts/0 S+ 10:55 0:00 grep pm2At any given time, there should only be three processes listed in this command: one for the daemon, one for the node pm2 instance, and one for the grep command. If there are additional instances running (usually in a pair of a daemon process and node process) you should execute the kill for any daemon instances that are not the most recent one. Note that the logs are streamed in from the most recently activated instance of pm2 - therefore, if you have multiple instances running off of the same port, the logs will only indicate that the API initialization has failed due to the port already being in use until you kill old instances and allow the newest one to connect.
It is also important to note that the issue mentioned above will sometimes cause the production API deployment to fail. Before deploying a new production build of the API, you should instantiate a new pm2 instance by running the following on the production API server and then killing any old pm2 instances:
cd /var/www/ipr/api/productionpm2 start current/pm2.config.js --env production┌─ Process list ──────────────────────┐┌─ Global Logs ───────────────────────────────────────────────────────────────────────────┐│[101] IPR-API Mem: 88 MB ││ IPR-API > 2018-01-05 14:22 -08:00: [2018-01-05 14:22:20][INFO] Retrieved assembly ││ ││ results ││ ││ IPR-API > 2018-01-05 14:22 -08:00: [2018-01-05 14:22:20][INFO] Querying DB for all ││ ││ tracking tasks with symlinks pending ││ ││ IPR-API > 2018-01-05 14:22 -08:00: [2018-01-05 14:22:20][INFO] Tasks requiring symlink ││ ││ lookup 17 ││ ││ IPR-API > 2018-01-05 14:22 -08:00: [2018-01-05 14:22:20][INFO] Starting query for ││ ││ retrieving library lane targets for 51 libraries ││ ││ IPR-API > 2018-01-05 14:22 -08:00: [2018-01-05 14:22:20][INFO] Target lanes determined ││ ││ for each library. Querying for aligned libcores ││ ││ IPR-API > 2018-01-05 14:22 -08:00: [2018-01-05 14:22:21][INFO] Checked in symlinks for ││ ││ 0 tasks. ││ ││ IPR-API > 2018-01-05 14:22 -08:00: [2018-01-05 14:22:21][INFO] Resulting in 7 disease ││ ││ libraries that have pathology information and need additional library ││ ││ details to differentiate RNA vs DNA. │└─────────────────────────────────────┘└─────────────────────────────────────────────────────────────────────────────────────────┘┌─ Custom metrics ────────────────────┐┌─ Metadata ──────────────────────────────────────────────────────────────────────────────┐│ Loop delay 0.81ms ││ App Name IPR-API ││ ││ Restarts 0 ││ ││ Uptime 23h ││ ││ Script path /var/www/ipr/api/production/current/bin/www ││ ││ Script args N/A ││ ││ Interpreter node ││ ││ Interpreter args N/A │└─────────────────────────────────────┘└─────────────────────────────────────────────────────────────────────────────────────────┘.├── app # Application files| || ├── config.js # Main configuration module which handles all ENV specific defaults│ ││ ├── api # API Interfaces│ │ Factories for interacting with API services.│ ││ ├── libs # Libs│ │ Application specific libraries│ ││ ├── middleware # Middleware│ │ Location for all globally required middleware definitions.│ │ │ ├── schemas # Schemas│ │ Location for all globally required schema definitions.│ ││ ├── models # Models│ │ Application DB models. Sequelizejs models describe table schemas.│ ││ └── routes # Routes│ ├── dir Subdirectories contain .js files that return Express.router implementations that handle defined routes.│ │ Routes are namespaced into the Express route handler based on parent directory structure.│ ││ └── index.js Routing index file. Loads special routing files first, then recursively loads child folder structure│ routes. Route namespaces are defined on their nested directory structure.│││├── config # Config│ Other configuration files│├── database # Databases│ Local expression and disease data│├── package.json NodeJS package.json dependency and application definitions file.│├── pm2.config.js Configuration settings for running the app using PM2│├── Dockerfile Instructions for creating API docker image│├── Dockerfile.db Instructions for creating demo db docker image│├── bin # Bin│ Contains main application file. Run node server.js to initialize the API server│├── demo # Demo│ Contains files related to the IPR demo│├── docs # Docs│ Contains document related files│├── keys # Keys│ Keycloak dev and production keys│├── test # Testing│ ├── dir Files and directories of files used for testing│ ││ ├── graphkb.mock.js Mock graphkb reponses used for testing│ ││ └── testData # Test Data│ Files (images, json files, etc.) used by the test files for testing│├── migrations # Migrations│ Contains files for updating db schemas and data│ └── migrationTools # Migration Tools Contains files with functions to help with migrationsThe API image requires the DB image and the keycloak image to already have been build and running. If youare setting up the entire platform, see thefull platform repository
To build the API image
docker build -t bcgsc/pori-ipr-api -f Dockerfile.Then to start the container
docker run -e IPR_DATABASE_HOSTNAME=localhost \ -e IPR_DATABASE_NAME=ipr_demo \ -e IPR_DATABASE_PASSWORD=root \ -e IPR_DATABASE_USERNAME=ipr_service \ -e IPR_GRAPHKB_PASSWORD=ipr_graphkb_link \ -e IPR_GRAPHKB_USERNAME=ipr_graphkb_link \ -e IPR_GRAPHKB_URI='http://localhost:8080/api' \ -e IPR_KEYCLOAK_KEYFILE=/keys/keycloak.key \ -e IPR_KEYCLOAK_URI='http://localhost:8888/auth/realms/PORI/protocol/openid-connect/token' \ --mount type=bind,source="$(pwd)"/keys,target=/keys \ bcgsc/pori-ipr-api:latest
About
Integrated Pipeline Reports (IPR) API, the reporting API as part of the PORI platform