File and directory permissions

Naryn creates files and directories with a umask of007(except for read-only tracks), which means that files and directorieswould have permissions of660 (rw-rw----) and770 (rwxrwx---) respectively. This means that in order toaccess a database that someone outside the group created, the file andfolder permissions need to be changed first.

Records and References

Track is a data structure that stores a set of records of(id, time, ref, numeric value) type. For example,hemoglobin level of patients can be stored in this way, whereid would be the id of the patient andtimewould indicate the moment when the blood test was made. Another trackcan contain the code of the laboratory which carried out the test. Ifthe times of the records from the two tracks match, one would concludewhich lab performed the given test.

Time resolution is always in hours. It might happen that twodifferent blood tests are carried out by two different labs for the samepatient at the same hour. Assuming that each lab has certain bias due todifferent equipment used, the reads of the hemoglobin might come outdifferent. Since both of the tests are carried out at exactly the samehour it will be impossible later to link each result to the lab thatperformed it.

In those cases when two or more values share identicalid andtime Naryn requires them to use thendifferentref (references). A reference is aninteger number in the range of [-1, 254], which when no time collisionoccurs is normally set to -1. However, in cases of ambiguity it can giveadditional resolution to the time. In our blood example the results ofthe first lab could have been recorded withref = 0 and thesecond lab would do it withref = 1. This way the twohemoglobin readings could later be separated and correctly linked totheir originating labs.

Categorical and Quantitative Tracks

Tracks store numerical values assigned to the patients and times. Thenumerical data however can have different meaning and hence imposedifferent set of operations to be applied to it. Laboratory codes,diagnosis codes, binary information such as date of birth or doctorvisits are one type of data which we callcategorical. Anothertype of data indicate usually the readings of different instruments suchas the heartbeat rate or glucose level. This type of data is calledquantitative.

The operations that can be applied to both of these types can be verydifferent. One might want to search for the specific diagnosis code, yetit makes little sense to search for the very specific heartbeat rate,say “68”. On contrary heartbeat rate readings from different times canbe averaged or a mean value might be calculated - something that has nomeaning in case of categorical data.

During the track creation one must specify the type of the track:categorical or quantitative. Various operations that can be laterapplied to the track are bound to the track type.

Logical tracks

In addition to the physical tracks which are stored in the binaryfiles,naryn supports a concept of alogical trackwhich is an alias to a physical track. For example, assume we have atrack calledlab.103 which contains hemoglobin levels ofpatients. It would be more convenient to refer to it explicitly byhemoglobin instead of remembering the lab code.Logicaltracks do exactly this, we can create a logical track calledhemoglobin which refers to the physicallab.103:

emr_track.logical.create("hemoglobin","lab.103")emr_extract("hemoglobin")

You can also uselogical tracks to create an alias forspecific values from acategorical track. For example, supposewe have a track calleddiagnosis.250 which contains thediagnosis times of ICD code 250 (“250.*”), with the valuesbeing the sub-diagnosis (e.g. 1 for 250.1 and4 for 250.4).Logical tracks allow us to create analias for a specific sub-diagnosis value and then refer to it as aregular track:

emr_track.logical.create("dx.250.1_4","diagnosis.250",values =c(1,4))emr_extract("dx.250.1_4")

Under the hood logical tracks are implemented using the virtualtracks mechanism (see below), but unlike virtual tracks - they are partof the database and are persistent between sessions. You can delete alogical track by callingemr_track.logical.rm and list themusingemr_track.logical.ls.

Track Attributes

In addition to numeric data a track may store arbitrary meta-datasuch as description, source, etc. The meta-data is stored in the form ofname-value pairs or attributes where the value is a characterstring.

Though not officially enforced attributes are intended to storerelatively short character strings. Please usetrack variablesto store data in any other format.

A single attribute can be retrieved, added, modified or deleted usingemr_track.attr.get andemr_track.attr.setfunctions. Bulk access to more than one attribute is facilitated byemr_track.attr.export function.

Track names which attributes values match a pattern can be retrievedusingemr_track.ls,emr_track.global.ls andemr_track.user.ls functions.

Track Variables

Track statistics, results of time-consuming per-track calculations,historical data and any other data in arbitrary format can be stored ina track’s supplementary data in the form of track variables. Trackvariable can be retrieved, added, modified or deleted usingemr_track.var.get,emr_track.var.set andemr_track.var.rm functions. List of track variables can beretrieved usingemr_track.var.ls function.

Note: track variables created in R are not visible in Python and viceversa.

Track Attributes vs. Track Variables

Though both track attributes and track variables can be used to storemeta-data of a track, there are a few important differences between thetwo that are summed up in the following table:

Subsets

The analysis of data often involves dividing the data to train andtest sets. Naryn allows to subset the data viaemr_db.subset function.emr_db.subset acceptsa list of ids or samples the ids randomly. These ids constitute thesubset. The ids that are not in the subset are skipped by all theiterators,filters and various functions.

One may think of a subset as an additional layer, a “viewport”, thatfilters out some of the ids. Some lower-level functions such asemr_track.info oremr_track.unique ignore thesubsets. Same applies topercentile.* functions of thevirtual tracks.

Introduction

Track expression allows to retrieve numerical data that isrecorded in the tracks. Track expressions are widely used in variousfunctions (emr_screen,emr_extract,emr_dist, …).

Track expression is a character string that closely resembles a validR/Python expression. Just like any other R/Python expression it mayinclude conditions, function calls and variables defined beforehand."1 > 2","mean(1:10)" and"myvar < 17" are all valid track expressions. Unlikeregular R/Python expressions track expression might also contain tracknames and / orvirtual track names.

To understand how the track expression allows the access to thetracks we must explain how the track expression gets evaluated.

Every track expression is accompanied by aniterator thatproduces a set ofid-time points of(id, time, ref) type. For each each iterator point thetrack expression is evaluated. The value of the track expression"mean(1:10)" is constant regardless the iterator point.However the track expression might contain a track namemytrack, like:"mytrack * 3". Naryn recognizesthen thatmytrack is not a regular R/Python variable butrather a track name. A newrun-time track variable namedmytrack is added then to R environment (or Python modulelocal dictionary). For each iterator point this variable is assigned thevalue of the track that matches(id, time, ref) (or NaN ifno matching value exists in the track). Oncemytrack isassigned the corresponding value, the track expression is evaluated inR/Python.

Run-time Track Variable is a Vector

To boost the performance of the track expression evaluation, run-timetrack variables are actually defined as vectors in R rather thanscalars. The result of the evaluation is expected to be also a vector ofa similar size. One should always keep in his mind the vectorialnotation and write the track expressions accordingly.

For example, at first glance a track expression"min(mytrack, 10)" seems to be perfectly fine. However theevaluation of this expression produces always a scalar, i.e. a singlenumber even ifmytrack is actually a vector. The way tocorrect the specific track expression so that it works on vectors, is tousepmin function instead ofmin.

Python

Similarly to R, a track variable in Python is not a scalar but ratheran instance ofnumpy.ndarray. The evaluation of a trackexpression must therefore produce anumpy.ndarray as well.Various operations on numpy arrays indeed work the same way as withscalars, however logical operations require different syntax. Forinstance:

screen("mytrack1 > 1 and mytrack2 < 2", iterator="mytrack1")

will produce an error given thatmytrack1 andmytrack2 are numpy arrays. The correct way to write theexpression is:

screen("(mytrack1 > 1) & (mytrack2 < 2)", iterator="mytrack1")

One may coerce the track variable to behave like a scalar: by settingemr_eval.buf.size option to1 (see Appendixfor more details). Beware though that this might take its heavy toll onrun-time.

Matching Reference in the Track Expression

If the track expression contains a track (or virtual track) name,then the values from the track are fetched one-by-one into theidentically named R variable based onid,timeandref of the iterator point. If howeverrefof the iterator point equals to-1, we treat it as a“wildcard”: matching is required then only forid andtime.

“Wildcard” reference in the iterator might create a new issue: morethan one track value might match then a single iterator point. In thiscase the value placed in the track variable (e.g. mytrack)depends on the type of the track. If the track is categorical the trackvariable is set to-1, otherwise it is set to the averageof all matching values.

Virtual Tracks

So far we have shown that in some situationsmytrackvariable can be set to the average of the matching track values. Butwhat if we do not want to average the values but rather pick up themaximal, minimal or median value? What if we want to use the percentileof a track value rather than the value itself? And maybe we even want toalter the time of the iterator point: shift it or expand to a timewindow and by that look at the different set of track values? Forinstance: given an iterator point we might want to know what was themaximal level of glucose during the last year that preceded the time ofthe point.

This is where virtual tracks come in use.

Virtual track is a named set of rules that describe how the trackshould be proceeded, and how the time of the iterator point should bemodified. Virtual tracks are created byemr_vtrack.createfunction:

emr_vtrack.create("annual_glucose",src ="glucose_track",func ="quantile",param =0.5,time.shift =c(-year(),0))

This call creates a new virtual track namedannual_glucose based on the underlying physicalsourcetrackglucose_track. For each iterator point with timeT we look at values ofglucose_track in thetime window of[T-365*24,T], i.e. one year prior toT. We calculate then the median over the values(func="quantile",param=0.5).

There is a rich set of various functions besides “quantile” that canbe applied to the track values. Some of these functions can be used onlywith categorical tracks, other ones - only with quantitative tracks andsome functions can be applied to both types of the track. Please referthe documentation ofemr_vtrack.create.

Once a virtual track is created it can be used in a trackexpression:

emr_extract("annual_glucose",iterator =list(year(),"patients.dob"))

This would give us a median of an annual glucose level in year-stepsstarting from the patient’s birthday. (This example makes use of anExtended Beat Iterator that would be explained later.)

Let’s expand our example further and ignore in our calculations theglucose readings that had been made within a week after steroids hadbeen prescribed. We can use an additionalfilter parameterto do that.

emr_filter.create("steroids_filter","steroids_track",time.shift=c(-week(),0))emr_vtrack.create("annual_glucose",src ="glucose_track",func ="quantile",param =0.5,time.shift =c(-year(),0),filter ="!steroids_filter")emr_extract("annual_glucose",iterator =list(year(),"date_of_birth_track"))

Filter is applied to the ID-Time points of the source track(e.g. glucose_track in our example). The virtual trackfunction (quantile, …) is applied then only to the pointsthat pass the filter. The concept of filters is explained extensively ina separate chapter.

Virtual tracks allow also to remap the patient ids. This is done viaid.map parameter which accepts a data frame that definesthe id mapping. Remapping ids might be useful if family ties areexplored. For example, instead of glucose level of the patient we areinterested to check the glucose level of one of his family members.

Track Iterator

Track iterator returns the points (including the reference)from the specified track. Track name is specified as a string.

Ifkeepref=FALSE the reference of each point is set to-1.

Example:

# Returns the level of glucose one hour after the insulin shot was madeemr_vtrack.create("glucose","glucose_track",func="avg",time.shift=1)emr_extract("glucose",iterator="insulin_shot_track")

Id-Time Points Iterator

Id-Time points iterator generates points from anid-timepoints table (see: Appendix). Ifkeepref=FALSE thereference of each point is set to-1.

Example:

# Returns the level of glucose one hour after the insulin shot was madeemr_vtrack.create("glucose","glucose_track",func="avg",time.shift=1)r<-emr_extract("insulin_shot_track")# <-- implicit iterator is used hereemr_extract("glucose",iterator=r)

Ids Iterator

Ids iterator generates points with ids taken from anidstable (see: Appendix) and times that run fromstime toetime with a step of 1.

Ifkeepref=TRUE for each id-time pair the iteratorgenerates 255 points with references running from0 to254. Ifkeepref=FALSE only one point isgenerated for the given id and time, and its reference is set to-1.

Example:

# Returns the level of glucose for each hour in year 2016 for ids 2 and 5stime<-emr_date2time(1,1,2016,0)etime<-emr_date2time(31,12,2016,23)emr_extract("glucose",iterator=data.frame(id=c(2,5)),stime=stime,etime=etime)

Time Intervals Iterator

Time intervals iterator generates points for all the idsthat appear in ‘patients.dob’ track with times taken from atimeintervals table (see: Appendix). Each time starts at the beginningof the time interval and runs to the end of it with a step of 1. Thatbeing said the points that lie outside of[stime, etime]range are skipped.

Ifkeepref=TRUE for each id-time pair the iteratorgenerates 255 points with references running from0 to254. Ifkeepref=FALSE only one point isgenerated for the given id and time, and its reference is set to-1.

Example:

# Returns the level of hangover for all patients the next day after New Year Eve# for the years 2015 and 2016stime1<-emr_date2time(1,1,2015,0)etime1<-emr_date2time(1,1,2015,23)stime2<-emr_date2time(1,1,2016,0)etime2<-emr_date2time(1,1,2016,23)emr_extract("alcohol_level_track",iterator=data.frame(stime=c(stime1, stime2),etime=c(etime1, etime2)))

Id-Time Intervals Iterator

Id-Time intervals iterator generates for each id points thatcover['stime', 'etime'] time range as specified inid-time intervals table (see: Appendix). Each time starts atthe beginning of the time interval and runs to the end of it with a stepof 1. That being said the points that lie outside of[stime, etime] range are skipped.

Ifkeepref=TRUE for each id-time pair the iteratorgenerates 255 points with references running from0 to254. Ifkeepref=FALSE only one point isgenerated for the given id and time, and its reference is set to-1.

Beat Iterator

Beat Iterator generates a “time beat” at the given periodfor each id that appear in ‘patients.dob’ track. The period is givenalways in hours.

Example:

emr_extract("glucose_track",iterator=10,stime=1000,etime=2000)

This will create a beat iterator with a period of 10 hours startingatstime up untiletime is reached. If, forexample,stime equals1000 then the beatiterator will create for each id iterator points at times: 1000, 1010,1020, …

Ifkeepref=TRUE for each id-time pair the iteratorgenerates 255 points with references running from0 to254. Ifkeepref=FALSE only one point isgenerated for the given id and time, and its reference is set to-1.

Extended Beat Iterator

Extended beat iterator is as its name suggests a variationon the beat iterator. It works by the same principle of creating timepoints with the given period however instead of basing the times countonstime it accepts an additional parameter - a track or aId-Time Points table - that instructs what should be theinitial time point for each of the ids. The two parameters (period andmapping) should come in a list. Each id is required to appear only onceand if a certain id does not appear at all, it is skipped by theiterator.

Anyhow points that lie outside of[stime, etime] rangeare not generated.

Example:

# Returns the maximal weight of patients at one year span starting from their birthdaysemr_vtrack.create("weight","weight_track",func ="max",time.shift =c(0,year()))emr_extract("weight",iterator =list(year(),"birthday_track"),stime =1000,etime =2000)

Periodic Iterator

periodic iterator goes over every year/month. Youcan use it by runningemr_monthly_iterator oremr_yearly_iterator.

Example:

iter<-emr_yearly_iterator(emr_date2time(1,1,2002),emr_date2time(1,1,2017))emr_extract("dense_track",iterator = iter,stime =1,etime =3)iter<-emr_monthly_iterator(emr_date2time(1,1,2002),n =15)emr_extract("dense_track",iterator = iter,stime =1,etime =3)

Implicit Iterator

The iterator is set implicitly if its value remainsNULL(which is the default). In that case the track expression is analyzedand searched for track names. If all the track variables or virtualtrack variables point to the same track, this track is used as a sourcefor a track iterator. If more then one track appears in the trackexpression, an error message is printed out notifying ambiguity.

Revealing Current Iterator Time

During the evaluation of a track expression one can access aspecially defined variable namedEMR_TIME (Python:TIME). This variable contains a vector(numpy.ndarray in Python) of current iterator times. Thelength of the vector matches the length of the track variable (which isa vector too).

Note that some values inEMR_TIME might be set 0. Skipthose intervals and the values of the track variables at thecorresponding indices.

# Returns times of the current iterator as a day of monthemr_extract("emr_time2dayofmonth(EMR_TIME)",iterator ="sparse_track")

Named Filters

Suppose we are interested in hemoglobin levels of patients who wereprescribed either drugX or drugY but not drugZ within a time window ofone week before the test. Assume that drugX, drugY and drugZ areresiding each in its separate track. Without filters we would need tocallemr_extract four times, store potentially huge dataframe results in the memory and finally merge the tables within R whilecaring about time windows. With filters we can do it much easier:

emr_filter.create("filterX","drugX",time.shift =c(week(),0))emr_filter.create("filterY","drugY",time.shift =c(week(),0))emr_filter.create("filterZ","drugZ",time.shift =c(week(),0))emr_extract("hemoglobin",filter ="(filterX | filterY) & !filterZ")

We can further expand the example above by specifying the ‘operator’argument on filter creation. If we wish to extract, the same informationas before, but in this case we are interested only in patients whichhave an hemoglobin level of at least 16 (in addition to our drugtreatment requirements). Under the same assumptions in the previousexample, our code would look like:

emr_filter.create("filterX","drugX",time.shift =c(week(),0))emr_filter.create("filterY","drugY",time.shift =c(week(),0))emr_filter.create("filterZ","drugZ",time.shift =c(week(),0))emr_filter.create("hemoglobin_gt_16","hemoglobin",val=16,operator=">")emr_extract("hemoglobin",filter ="(filterX | filterY) & !filterZ & hemoglobin_gt_16")

Python

Filter with logical conditions will use Python’s notation like:

extract("hemoglobin",filter="(filterX or filterY) and not filterZ")

Each call toemr_filter.create creates anamedelementary filter (or simply: named filter) with a unique name. Thenamed filter can then be used infilter parameter of aniterator and be combined with other named filters using the logicaloperators.

Other Objects within Filters

In our previous example we created three named filters based on threetracks. If time window was not required, we could have used the names ofthe tracks directly in the filter, like:filter = "(drugX | drugY) & !drugZ".

In addition to track names other types of objects can be used withinthe filter. These are:Id-Time Points Table,IdsTable,Time Intervals Table andId-Time IntervalsTable (seeAppendix for the format of these tables). Whenused in the filter the object should be constructed in advanced and bereferred by its name. “In place” construction (aka:filter = "data.frame(...)" is not allowed.

Managing Reference in Filters

The ID-Time Point embeds within itself a reference value. Namedfilters allow to specify whether the reference should be used formatching or not. Whenkeepref=TRUE is set withinemr_filter.create, the candidate point’s reference ismatched with the filter’s reference. Otherwise the references areignored.

It is important to remember that references are always ignored whenany object but a named filter is used within a filter. For instance, iffilter = "drug" anddrug is a name of a track(and not a name of a named filter), then the references will be ignoredduring the matching. To ensure the filter matches the references ofdrug track, one must define a named filter withkeepref=TRUE parameter:

emr_filter.create("drug_filter","drug",keepref=TRUE)emr_extract(my.track.expression,filter="drug_filter",keepref=TRUE)