library(hdf5r)test_filename<-tempfile(fileext =".h5")file.h5<- H5File$new(test_filename,mode ="w")file.h5

mtcars.grp<- file.h5$create_group("mtcars")flights.grp<- file.h5$create_group("flights")

library(datasets)library(nycflights13)library(reshape2)mtcars.grp[["mtcars"]]<- datasets::mtcarsflights.grp[["weather"]]<- nycflights13::weatherflights.grp[["flights"]]<- nycflights13::flights

weather_wind_dir<-subset(nycflights13::weather, origin=="EWR",select =c("year","month","day","hour","wind_dir"))weather_wind_dir<-na.exclude(weather_wind_dir)weather_wind_dir$wind_dir<-as.integer(weather_wind_dir$wind_dir)weather_wind_dir<-acast(weather_wind_dir, year+ month+ day~ hour,value.var ="wind_dir")

flights.grp[["wind_dir"]]<- weather_wind_dir

weather_wind_speed<-subset(nycflights13::weather, origin=="EWR",select =c("year","month","day","hour","wind_speed"))weather_wind_speed<-na.exclude(weather_wind_speed)weather_wind_speed<-acast(weather_wind_speed, year+ month+ day~ hour,value.var ="wind_speed")

flights.grp[["wind_speed"]]<- weather_wind_speed

h5attr(flights.grp[["wind_dir"]],"colnames")<-colnames(weather_wind_dir)h5attr(flights.grp[["wind_dir"]],"rownames")<-rownames(weather_wind_dir)h5attr(flights.grp[["wind_speed"]],"colnames")<-colnames(weather_wind_speed)h5attr(flights.grp[["wind_speed"]],"rownames")<-rownames(weather_wind_speed)

2.2.1 Content of filesand groups

With respect to groups and files, we also want to have a simple wayto extract the contents. With thenames function, wecan get all names of objects in a group or in the root directory of afile

names(file.h5)

## [1] "flights" "mtcars"

names(flights.grp)

## [1] "flights"    "weather"    "wind_dir"   "wind_speed"

Another option that gives more information isls, amethod of the classesH5File andH5Group

flights.grp$ls()

##         name     link.type    obj_type num_attrs group.nlinks group.mounted## 1    flights H5L_TYPE_HARD H5I_DATASET         0           NA            NA## 2    weather H5L_TYPE_HARD H5I_DATASET         0           NA            NA## 3   wind_dir H5L_TYPE_HARD H5I_DATASET         2           NA            NA## 4 wind_speed H5L_TYPE_HARD H5I_DATASET         2           NA            NA##   dataset.rank dataset.dims dataset.maxdims dataset.type_class## 1            1       336776             Inf       H5T_COMPOUND## 2            1        26115             Inf       H5T_COMPOUND## 3            2     364 x 24       Inf x Inf        H5T_INTEGER## 4            2     364 x 24       Inf x Inf        H5T_INTEGER##   dataset.space_class committed_type## 1          H5S_SIMPLE           <NA>## 2          H5S_SIMPLE           <NA>## 3          H5S_SIMPLE           <NA>## 4          H5S_SIMPLE           <NA>

2.2.2 Information onattributes, datatypes and datasets

If you have an HDF5-File, it is of course important to look upvarious information not only about groups, but also about theinformation contained in it. First, we want to get more informationabout the dataset.ls on the group already gives a lotof information about the datatype, the size, the maximum size etc.However there are also other, more direct, ways to get the sameinformation. In order to investigate the datatype we can

weather_ds<- flights.grp[["weather"]]weather_ds_type<- weather_ds$get_type()weather_ds_type$get_class()

## [1] H5T_COMPOUND## 13 Levels: H5T_NO_CLASS H5T_INTEGER H5T_FLOAT H5T_TIME ... H5T_NCLASSES## 13 Values: -1 0 1 2 ... 11

cat(weather_ds_type$to_text())

## H5T_COMPOUND {##       H5T_STRING {##          STRSIZE H5T_VARIABLE;##          STRPAD H5T_STR_NULLTERM;##          CSET H5T_CSET_ASCII;##          CTYPE H5T_C_S1;##       } "origin" : 0;##       H5T_STD_I32LE "year" : 8;##       H5T_STD_I32LE "month" : 12;##       H5T_STD_I32LE "day" : 16;##       H5T_STD_I32LE "hour" : 20;##       H5T_IEEE_F64LE "temp" : 24;##       H5T_IEEE_F64LE "dewp" : 32;##       H5T_IEEE_F64LE "humid" : 40;##       H5T_IEEE_F64LE "wind_dir" : 48;##       H5T_IEEE_F64LE "wind_speed" : 56;##       H5T_IEEE_F64LE "wind_gust" : 64;##       H5T_IEEE_F64LE "precip" : 72;##       H5T_IEEE_F64LE "pressure" : 80;##       H5T_IEEE_F64LE "visib" : 88;##       H5T_IEEE_F64LE "time_hour" : 96;##    }

telling us that our dataset consists of aH5T_COMPOUNDdatatype and prints more detailed information on its content of everycolumn. Regarding the size of the dataset and the size of the chunks(datasets are by default chunked; more about this below) we do:

weather_ds$dimsweather_ds$maxdimsweather_ds$chunk_dims

## [1] 26115## [1] Inf## [1] 78

In order to get information on attributes we also have variousfunction available. Which attributes are attached to an object we cansee with

h5attr_names(flights.grp[["wind_dir"]])

## [1] "colnames" "rownames"

and the content of one attribute can be extracted withh5attr, the content of all of them with a list ash5attributes.

h5attr(flights.grp[["wind_dir"]],"colnames")

##  [1] "0"  "1"  "2"  "3"  "4"  "5"  "6"  "7"  "8"  "9"  "10" "11" "12" "13" "14"## [16] "15" "16" "17" "18" "19" "20" "21" "22" "23"

2.2.3 Detailedinformation about various objects

In HDF5, there are also various ways of getting more detailedinformation about objects. The most detailed methods for this are

• get_obj_info: Various information on the number ofattributes, the type of the object, the reference count, access times(if set to be recorded) and other more technical information
• get_link_info: For links, mainly yields informationon the link type, i.e. hard link or soft link. The difference betweenthem and how to create them will be discussed further below.
• get_group_info: Information about the storage typeof the group, if a file is mounted to the group and the number of itemsin the group. For the casual user, the most interesting information isthe number of items in the group, which can also be retrieved using thenames function. For very large groups, this way ishowever more efficient.
• get_file_name: For anH5File orH5Group,H5D orH5T (where D is for datasetand T stands for a committed type) object, returns the name of the fileit is in.
• get_obj_name: Similar asget_file_name,applies to the same object, but returns the pathinsidethe file to the object
• file_info: It extracts relatively technicalinformation about a file. It can only be applied to an object of classH5File. This function is usually not of interest to thecasual user

Most of these are somewhat advanced. They key information can usuallyalso be extracted with one of the “higher-level” methods shown above,but sometimes theinfo methods are more efficient.

weather_ds[1:5]

wind_dir_ds<- flights.grp[["wind_dir"]]wind_dir_ds[1:3, ]

wind_dir_ds[1, ]<-rep(1,24)wind_dir_ds[1, ]

wind_dir_ds$get_fill_value()

wind_dir_ds[1,25]<-1wind_dir_ds[1:2, ]

flights.grp$link_delete("wind_dir")flights.grp$ls()

file.h5$close_all()

uint2_dt<- h5types$H5T_NATIVE_UINT32$set_size(1)$set_precision(2)$set_sign(h5const$H5T_SGN_NONE)

space_ds<- H5S$new(dims =c(10,10),maxdims =c(Inf,10))

ds_create_pl_nbit<- H5P_DATASET_CREATE$new()ds_create_pl_nbit$set_chunk(c(10,10))$set_fill_value(uint2_dt,1)$set_nbit()

uint2.grp<- file.h5$create_group("uint2")uint2_ds_nbit<- uint2.grp$create_dataset(name ="nbit_filter",space = space_ds,dtype = uint2_dt,dataset_create_pl = ds_create_pl_nbit,chunk_dim =NULL,gzip_level =NULL)uint2_ds_nbit[, ]<-sample(0:3,size =100,replace =TRUE)uint2_ds_nbit$get_storage_size()

ds_create_pl_nbit_deflate<- ds_create_pl_nbit$copy()$set_deflate(9)ds_create_pl_deflate<- ds_create_pl_nbit$copy()$remove_filter()$set_deflate(9)ds_create_pl_none<- ds_create_pl_nbit$copy()$remove_filter()uint2_ds_nbit_deflate<- uint2.grp$create_dataset(name ="nbit_deflate_filter",space = space_ds,dtype = uint2_dt,dataset_create_pl = ds_create_pl_nbit_deflate,chunk_dim =NULL,gzip_level =NULL)uint2_ds_nbit_deflate[, ]<- uint2_ds_nbit[, ]uint2_ds_deflate<- uint2.grp$create_dataset(name ="deflate_filter",space = space_ds,dtype = uint2_dt,dataset_create_pl = ds_create_pl_deflate,chunk_dim =NULL,gzip_level =NULL)uint2_ds_deflate[, ]<- uint2_ds_nbit[, ]uint2_ds_none<- uint2.grp$create_dataset(name ="none_filter",space = space_ds,dtype = uint2_dt,dataset_create_pl = ds_create_pl_none,chunk_dim =NULL,gzip_level =NULL)uint2_ds_none[, ]<- uint2_ds_nbit[, ]

uint2_ds_nbit_deflate$get_storage_size()uint2_ds_nbit$get_storage_size()uint2_ds_deflate$get_storage_size()uint2_ds_none$get_storage_size()

3.2.1 Integer, Float

For integer-datatypes we have already seen that we have control overessentially everything, i.e. signed/unsigned as well as precision downto the exact number of bits. For floats we have similar control, beingable to customize the size of the mantissa as well as the exponent(although in practice this is likely less relevant than being able tocustomize integer types). To learn more about this functionality forfloats, we recommend to read the relevant section of the manual.

3.2.2 Strings

HDF5 itself provides access to both C-type strings and FORTRAN typestrings. As R internally uses C-strings, only C-type strings aresupported (i.e. strings that are NULL delimited). In terms of the sizeof the strings, there are fixed and variable length stringsavailable.

str_fixed_len<- H5T_STRING$new(size =20)str_var_length<- H5T_STRING$new(size =Inf)

These two types of strings have implications for efficiency andusability. For obvious reasons, variable length strings are moreconvenient as they are never too small hold a piece of information.However, internally in HDF5, these aren’t stored in the dataset itself -only a pointer to the HDF5-internal heap is stored. This has 2implications:

• Retrieving the string is somewhat slower
• As the heap is not compressed, compression of datasets does notyield much space saving for variable length data

From this perspective, fixed length strings are considerably betteras they are both faster (if not too long) and compressible. However, theuser has to be careful that their strings aren’t getting too long, orthey will be truncated.

3.2.3 Enum

The equivalent to factors in R areENUM datatypes.These are stored internally as integers, but each integer has a stringlabel attached to it. In contrast to R-factor variables, the integervalues do not have to start at 1 and do not have to to consecutiveeither. In order to support this more flexible datatype also optimallyon the R side, hdf5r comes with thefactor_extendedclass. In the HDF5 API - each enum level is inserted one at a time. Asthis is rather inconvenient for a vector-oriented language like R, thisfunctionality has not been exposed. We instead provide an R6-classconstructor that lets us set all labels and values in one go.

enum_example<- H5T_ENUM$new(c("Label 1","Label 2","Label 3"),values =c(-3,5,10))

For efficiency reasons, an integer datatype is automaticallygenerated that provides exactly the needed precision in order to storethe values of the enum. Given an enum, variable, we can also find outwhat labels and values it has

enum_example$get_labels()enum_example$get_values()

## [1] "Label 1" "Label 2" "Label 3"## [1] -3  5 10

In addition, we can also get the datatype back that the enum is basedon

enum_example$get_super()

## Class: H5T_INTEGER## Datatype: undefined integer

logical_example<- H5T_LOGICAL$new(include_NA =TRUE)## we could also use h5types$H5T_LOGICAL or h5types$H5T_LOGICAL_NAlogical_example$get_labels()logical_example$get_values()

## [1] "FALSE" "TRUE"  "NA"   ## [1] 0 1 2

3.2.4 Compounds(Tables)

Tables are represented asCOMPOUND HDF5 objects, which arethe equivalent of C-struct. As R does not know this datatype natively,it has to be converted from structs to the list-based construct of Rdata-frames. Similar as with ENUMs, we don’t expose the underlying C-APIthat builds the compound on element at a time but instead provideconstructors that create it in one go.

cpd_example<- H5T_COMPOUND$new(c("Double_col","Int_col","Logical_col"),dtypes =list(h5types$H5T_NATIVE_DOUBLE,    h5types$H5T_NATIVE_INT, logical_example))

and similar to enums, we can also get back the column names, theclasses of the datatypes as well as identifiers for the datatypesitself.

cpd_example$get_cpd_labels()

## [1] "Double_col"  "Int_col"     "Logical_col"

cpd_example$get_cpd_classes()

## [1] H5T_FLOAT   H5T_INTEGER H5T_ENUM   ## 13 Levels: H5T_NO_CLASS H5T_INTEGER H5T_FLOAT H5T_TIME ... H5T_NCLASSES## 13 Values: -1 0 1 2 ... 11

cpd_example$get_cpd_types()

## [[1]]## Class: H5T_FLOAT## Datatype: H5T_IEEE_F64LE## ## [[2]]## Class: H5T_INTEGER## Datatype: H5T_STD_I32LE## ## [[3]]## Class: H5T_LOGICAL## Datatype: H5T_ENUM {##       undefined integer;##       "FALSE"            0;##       "TRUE"             1;##       "NA"               2;##    }

A textual description is also available

cat(cpd_example$to_text())

## H5T_COMPOUND {##       H5T_IEEE_F64LE "Double_col" : 0;##       H5T_STD_I32LE "Int_col" : 8;##       H5T_ENUM {##          undefined integer;##          "FALSE"            0;##          "TRUE"             1;##          "NA"               2;##       } "Logical_col" : 12;##    }

cplx_example<- H5T_COMPLEX$new()cplx_example$get_cpd_labels()cplx_example$get_cpd_classes()

## [1] "Real"      "Imaginary"## [1] H5T_FLOAT H5T_FLOAT## 13 Levels: H5T_NO_CLASS H5T_INTEGER H5T_FLOAT H5T_TIME ... H5T_NCLASSES## 13 Values: -1 0 1 2 ... 11

3.2.5 Arrays

A special datatype is theH5T_ARRAY. As datasets areitself arrays, they are not needed to represent arrays itself. Rather,the are useful in cases where one datatype is wrapped inside another, somainly if a column of a compound object is supposed to be an array. Solets create an array and put it into a compound object together withsome other columns

array_example<- H5T_ARRAY$new(dims =c(3,4),dtype_base = h5types$H5T_NATIVE_INT)cpd_several<- H5T_COMPOUND$new(c("STRING_fixed","Double","Complex","Array"),dtypes =list(str_fixed_len, h5types$H5T_NATIVE_DOUBLE, cplx_example, array_example))cat(cpd_several$to_text())

## H5T_COMPOUND {##       H5T_STRING {##          STRSIZE 20;##          STRPAD H5T_STR_NULLTERM;##          CSET H5T_CSET_ASCII;##          CTYPE H5T_C_S1;##       } "STRING_fixed" : 0;##       H5T_IEEE_F64LE "Double" : 20;##       H5T_COMPOUND {##             H5T_IEEE_F64LE "Real" : 0;##             H5T_IEEE_F64LE "Imaginary" : 8;##          } "Complex" : 28;##       H5T_ARRAY {##          [4][3] H5T_STD_I32LE##       } "Array" : 44;##    }

And to see what this would look like as an R object

obj_empty<-create_empty(1, cpd_several)obj_empty

## Warning in format.data.frame(if (omit) x[seq_len(n0), , drop = FALSE] else x, :## corrupt data frame: columns will be truncated or padded with NAs

obj_empty$Array

##   STRING_fixed Double Complex Array## 1                   0    0+0i     0##  [1] 0 0 0 0 0 0 0 0 0 0 0 0

3.2.6 Variable lengthdata types

And last, there are also variable length datatypes - corresponding toa list in R where each item of the list has the same datatype (general Rlist, where each item can have a different type cannot be represented inHDF5).

vlen_example<- H5T_VLEN$new(dtype_base = cpd_several)

This would represent a list where each item is a table with anarbitrary number of rows.

4.1.1 Opening and closingof files

In this context, let us quickly also discuss the special way HDF5handles files. In HDF5, in principle a file can always only be openedonce. This can lead to problems as users in R are used to being able toopen files as often as they like. Furthermore, it is possible in HDF5 toclose the ID of a file without closing all objects in the file. Then,however, the file actually stays open until the last ID pointing intothe file is closed and it cannot be opened again without it.

Therefore, as already explained above (and as recommended by the HDF5manual), do not discard or close files that still have open objects inthem. It is preferable to keep the HDF5-file-id pointer around and closeit when it is no longer needed (and all objects inside the file) usingtheclose_all method.

4.2.1 Numericdatatypes

For numeric datatypes, the situation is in certain circumstances abit tricky. In general, R numerical objects are either represented as64-bit floating point values (doubles) or 32-but integers. R switchesrelatively transparently between these types as needed (forcomputations, integers are converted to doubles and conversely, arraypositions can be addressed by doubles). The main issue when working withHDF5 occurs as R doesn’t have either a 64bit signed or unsigned integerdatatype (and also not a long double). In order to work around thisissue, the following conventions are being used

• The package uses thebit64 package to providesupport for 64-bit integers. These are used extensively (e.g. for ids)and also for numeric integer data types.
• 32 bit and 64 bit floats from HDF5 are always returned as 64 bitfloats in R. Writing 32 bit floats from R, may always incur loss ofprecision of the underlying 64 bit double that is used to represent itin R.
• For integer data types, any HDF5 integer type that can accurately berepresented as a 32-bit signed integer will be returned to R as aregular integer (can be changed using flags). Any HDF5 64-bit integercan be returned as a signed 64-bit integer - with the option ofreturning it as a 32 bit integer or double if it can be done withoutloss of precision. For unsigned 64-bit integers, they will be returnedas floats, incurring loss of precision but avoiding truncation.

An overview of how the data conversion is being done can be seenhere:

Schematic of dataype conversion

The underlying principle is that any internal conversion between Rtypes is done by R (with the resulting handling of NA’s and overflows),whereas any conversion between R-types and Non-R-types is done by theHDF5 library (usually meaning that on overflow, truncation occurs).

4.2.2 Strings

In HDF5, strings can either be variable length or fixed lengthstrings. In R, they are always variable length. Therefore, strings fromR to HDF5 that are written into fixed-length fields will be truncated.Conversely, strings from HDF5 that are fixed length to R will only bereturned up the the NULL character that ends strings in C.

4.2.3Data-frames/Compounds

The situation is a bit more tricky for table-like objects. In R,these are data-frames, which internally are a list of vectors. In HDF5,a table is a Compound object, that is equivalent to C-struct -i.e. every row is represented together whereas in R every column isrepresented together. Each of these approaches has certain advantages,but the challenge here is to translate between them.

This is done in the straightforward manner. When converting from R toHDF5, the columns of the tables are copied into the struct whereas inthe reverse direction, every struct is decomposed into the correspondingcolumns.

The Data-frame <-> Compound conversion is also extensively usedfor HDF5-API functions that return structs as result (and thereforereturn data-frames).

4.2.4 Arraysdata-types

In HDF5, datasets itself can have arbitrary dimensions. In additionto that, there are also array-datatypes that allow for the inclusion forarrays e.g. inside a compound object. Translation to and from arrays isrelatively straightforward and only involves setting the correctdim attribute in R.

In addition to that, however, there is small complication. In R, thefirst dimension is the fastest changing dimension. In HDF5 (same as inC), the last dimension is however the fastest changing one. Fordatasets, we work around this problem by always reversing the dimensionsthat are passed between R and HDF5 and therefore making the distinctiontransparent. For arrays, this is however a bit trickier. For example letus assume that we have a dataset that is a one-dimensional vector oflength 10, each element of which is an array-datatype of length 4,resulting in a 10 x 4 dataset. However, it is now not quite clear howthis should be represented in R. If we follow the notion, that thefastest changing dimension in R is the first one, the result would be adataset with 4 rows and 10 columns, i.e. 10 x 4.

This does feel rather unintuitive, forcing a user to specify thesecond dimension to get all items of the array. Therefore, we haveimplemented it so that a 10 x 4 dataset is returned, with each rowcorresponding to the array-datatype. In order to achieve this we have todeviate from the ordering principle in HDF5. Where in HDF5, the elementsof the first internal array are in position 1, 2, 3 and 4 (or 0 to 3when you start counting at 0), in R they are now in position 1, 11, 21,and 31. In order to do this, we first internally read the HDF5 arrayinto an R-array of shape 4 x 10 and then transpose the result.

4.2.5 Variable-lengthdata types

In HDF5, there are also variable-length data types. Essentially, thiscorresponds to an R list-like object, with the additional restrictionthat every item of the list has to be of the same datatype. This is alsohow it is implemented. R list where all items are vectors (of arbitrarylength) of the same type can be converted to HDF5-VLEN objects and viceversa.

4.2.6 Referenceobjects

As of the writing of this vignette, these have not yet beenimplemented.