Repository files navigation

Tools for raster data including scientific and geophysical applications.

We have two main documentation pages:

1. The Gridfour Project Notes give information onthe underlying concepts and algorithms used by this project. The Notes page isn't just about Gridfour.It covers ideas and topics related to raster data processing in general.

2. The Gridfour Wiki gives lots of helpful informationon using Gridfour software including our Gridfour Virtual Raster Store (GVRS). It also gives informationabout our project goals and roadmap.

Although there are many tools for image processing and gridded-data applications,the Gridfour Project believes that there is stilla need for general-purpose software utilities for the processing of raster (grid)products. Potential applications in these areas run the gamut from rendering,data compression, contouring, surface analysis, and other operations aimedat analyzing and understanding data stored in raster form.

The Gridfour API for this project page is written in Java. A C-language API is being developedat theGridfourC API project.

One of the main components of our API is the Gridfour Virtual Raster Store, or "GVRS"(pronounced "givers").GVRS is a file-backed system that provides memory-efficient access to large and very large raster (grid) data sets.GVRS was created with three main purposes in mind:

1. Authoring Data: For applications that produce data sets, GVRS provides a high-performancevirtual management system for handling content. Applications may store their final resultsin persisent GVRS files, or transcribe them to conventional formats (NetCDF, HDF5, TIFF and GeoTIFF, etc.).

2. Experiments in Data Compression: GVRS provides a convenient testbed for developerswho are exploring new ways of performing data compression for raster data sources.

3. Distribution: GVRS provides a light-weight API and data format suitable for distributingdata to other systems. GVRS was originally conceived as a way of providing environment data for small systems,single-board computers, and platforms such as Autonomous Underwater Vehicles. But it is a feasiblesolution for many other use cases.

Recently, there has been a lot of news about theSeabed 2030 Project . That ambitiousundertaking aims to map 100 percent of the ocean floor by 2030. To put that in perspective,the project organizers estimate that, today, only about 20 percent of the world's oceans are fullymapped (seeSeabed 2030, FAQ ). So there's a lot of work to be donein the next decade.

On thing is certain, the existence of projects like Seabed 2030 will result in massive collectionsof grid-based (raster) data sets. Seabed 2030 itself will include about 7.6 billion grid points[1].There is a need for software libraries that can assist inthe processing of that data. In particular, we see a need for more efficient techniques for storageand data compression for grid data. That need inspired us to create Gridfour.

Some of the algorithms used in GVRS have been around for a long time.Our data compression techniques were originally developed for a project namedGem93 that was completed in 1993. Gem93 includeda number of tools for working with raster data, among them a data compression techniqueinspired by the work of Kidner and Smith (1992). You can read more about them at our project documentation pageGridfour Raster Data Compression Algorithms.

Of course, the state of the art has advanced quite a bit since 1993.And although the foundation for GVRS comes from old ideas, we hope you find that our API provides a fresh take on their implementationWe intend our GVRS library to provide a convenient tool for investigators developing new techniques for compressinggeophysical and scientific data in raster form. GVRS makes it very easy toextend the Gridfour code and add new data compression capabilities ( to seejust how easy it is, visit our wiki page onCustom Data Compressors ).Our hope is that by providing GVRS, we will help investigatorsfocus on their own research and leave the details of memory and file-management tothe Gridfour API.

We are finishing up the initial implementation of GVRS. We are looking fordevelopers interested in porting it to additional languages(such as C#, C++, Rust) and creating Python bindings. We are also looking forusers with ideas for new features and how to apply the library.

The Gridfour Software Project is still in its infancy. There is a lotof opportunity for new ideas and new software development. In the futurewe hope to include implementations of contouring, statistical analysis,and physical modeling logic to our collection. We are also building tools tosimplify access to data from the Shuttle Radar Topography Mission (SRTM )and the U.S. Geological Survey's high-resolution3D Elevation Program.

In the meantime, you are welcome to visit our companion Tinfour Software Project athttps://github.com/gwlucastrig/Tinfour

Finally, we end with a picture that was created using GVRS and a set of elevationand bathymetry data taken from the GEBCO_2019 global data set. The pictureshows a shaded-relief rendering of the Island of Hokkaido, Japan.GEBCO_2019 was one of the data sets used for the GVRS pilot project and a good example of thepotential of systems like it. Color-coding was based on elevation obtained from a GVRS fileand shading was computed using the surface normal obtained with Gridfour's B-Spline raster interpolation class.The GVRS data compression reduces the size of this data set down to about 17.2 percent ofits original size (seeGVRS Performancefor more details ). Future work may bring about more improvements.

General Bathymetric Chart of the Oceans [GEBCO], 2019.GEBCO Gridded Bathymetry Data.Accessed December 2019 fromhttps://www.gebco.net/data_and_products/gridded_bathymetry_data/

Kidner, D.B. and Smith, D.H. (1992).Compression of digital elevation models by Huffman coding,Computers and Geosciences, 18(8), 1013-1034.

National Oceanographic and Atmospheric Administration [NOAA], 2019.ETOPO1 Global Relief Model. Accessed December 2019 fromhttps://www.ngdc.noaa.gov/mgg/global/

[1]Point count estimated using survey-resolution table given inSeabed 2030.