 | |
| Developer(s) | MonetDB Foundation |
|---|---|
| Stable release | Aug2024 (11.51)[1] / () |
| Repository | |
| Written in | C |
| Operating system | Cross-platform |
| Type | Column-oriented DBMS RDBMS |
| License | Mozilla Public License, version 2.0 |
| Website | www |
MonetDB is anopen-sourcecolumn-orientedrelational database management system (RDBMS) originally developed at theCentrum Wiskunde & Informatica (CWI) in theNetherlands.It is designed to provide high performance on complex queries against large databases, such as combiningtables with hundreds of columns and millions of rows.MonetDB has been applied in high-performance applications foronline analytical processing,data mining,geographic information system (GIS),[2]Resource Description Framework (RDF),[3] text retrieval andsequence alignment processing.[4]
Data mining projects in the 1990s required improved analytical database support. This resulted in aCWIspin-off called Data Distilleries, which used early MonetDB implementations in its analytical suite. Data Distilleries eventually became a subsidiary ofSPSS in 2003, which in turn was acquired byIBM in 2009.[5]
MonetDB in its current form was first created in 2002 by doctoral studentPeter Boncz and professorMartin L. Kersten as part of the 1990s' MAGNUM research project atUniversity of Amsterdam.[6] It was initially called simply Monet, after the French impressionist painterClaude Monet. The first version under anopen-source software license (a modified version of theMozilla Public License) was released on September 30, 2004. When MonetDB version 4 was released into the open-source domain, many extensions to the code base were added by the MonetDB/CWI team, including a new SQL front end, supporting theSQL:2003 standard.[7]
MonetDB introduced innovations in all layers of theDBMS: a storage model based on vertical fragmentation, a modernCPU-tuned query execution architecture that often gave MonetDB a speed advantage over the samealgorithm over a typicalinterpreter-basedRDBMS. It was one of the first database systems to tune query optimization forCPU caches. MonetDB includes automatic and self-tuning indexes, run-time query optimization, and a modular software architecture.[8][9]
By 2008, a follow-on project called X100 (MonetDB/X100) started, which evolved into theVectorWise technology. VectorWise was acquired byActian Corporation, integrated with theIngres database and sold as a commercial product.[10][11]
In 2011 a major effort to renovate the MonetDB codebase was started. As part of it, the code for the MonetDB 4 kernel and its XQuery components were frozen. In MonetDB 5, parts of the SQL layer were pushed into the kernel.[7] The resulting changes created a difference in internalAPIs, as it transitioned from MonetDB Instruction Language (MIL) to MonetDB Assembly Language (MAL). Older, no-longer maintained top-level query interfaces were also removed. First wasXQuery, which relied on MonetDB 4 and was never ported to version 5.[12] The experimentalJaql interface support was removed with the October 2014 release.[13] With the July 2015 release, MonetDB gained support for read-onlydata sharding and persistent indices. In this release the deprecated streaming data module DataCell was also removed from the main codebase in an effort to streamline the code.[14] In addition, the license has been changed into theMozilla Public License, version 2.0.
 | This article needs to beupdated. Please help update this section to reflect recent events or newly available information.(June 2017) |
MonetDB architecture is represented in three layers, each with its own set of optimizers.[15]The front end is the top layer, providing query interface forSQL, with SciQL andSPARQL interfaces under development. Queries are parsed into domain-specific representations, like relational algebra for SQL, and optimized. The generated logical execution plans are then translated into MonetDB Assembly Language (MAL) instructions, which are passed to the next layer. The middle or back-end layer provides a number of cost-based optimizers for the MAL. The bottom layer is the database kernel, which provides access to the data stored in Binary Association Tables (BATs). Each BAT is a table consisting of an Object-identifier and value columns, representing a single column in the database.[15]
MonetDB internal data representation also relies on the memory addressing ranges of contemporary CPUs usingdemand paging of memory mapped files, and thus departing from traditional DBMS designs involving complex management of large data stores in limited memory.
Query recycling is an architecture for reusing the byproducts of the operator-at-a-time paradigm in a column store DBMS. Recycling makes use of the generic idea of storing and reusing the results of expensive computations. Unlike low-level instruction caches, query recycling uses an optimizer to pre-select instructions to cache. The technique is designed to improve query response times and throughput, while working in a self-organizing fashion.[16] The authors from theCWI Database Architectures group, composed of Milena Ivanova,Martin Kersten, Niels Nes and Romulo Goncalves, won the "Best Paper Runner Up" at theACM SIGMOD 2009 conference for their work on Query Recycling.[17][18]
MonetDB was one of the first databases to introduce Database Cracking. Database Cracking is an incremental partial indexing and/or sorting of the data. It directly exploits the columnar nature of MonetDB. Cracking is a technique that shifts the cost of index maintenance from updates to query processing. The query pipeline optimizers are used to massage the query plans to crack and to propagate this information. The technique allows for improved access times and self-organized behavior.[19] Database Cracking received theACM SIGMOD 2011 J.Gray best dissertation award.[20]
A number of extensions exist for MonetDB that extend the functionality of the database engine. Due to the three-layer architecture, top-level query interfaces can benefit from optimizations done in the backend and kernel layers.
MonetDB/SQL is a top-level extension, which provides complete support for transactions in compliance with theSQL:2003 standard.[15]
MonetDB/GIS is an extension to MonetDB/SQL with support for theSimple Features Access standard ofOpen Geospatial Consortium (OGC).[2]
SciQL an SQL-based query language for science applications with arrays as first class citizens. SciQL allows MonetDB to effectively function as anarray database. SciQL is used in theEuropean UnionPlanetDataArchived 2014-05-30 at theWayback Machine andTELEIOS project, together with the Data Vault technology, providing transparent access to large scientific data repositories.[21] Data Vaults map the data from the distributed repositories to SciQL arrays, allowing for improved handling ofspatio-temporal data in MonetDB.[22] SciQL will be further extended for theHuman Brain Project.[23]
Data Vault is a database-attached external file repository for MonetDB, similar to theSQL/MED standard. The Data Vault technology allows for transparent integration with distributed/remote file repositories. It is designed for scientific datadata exploration andmining, specifically forremote sensing data.[22] There is support for theGeoTIFF (Earth observation),FITS (astronomy), MiniSEED (seismology) andNetCDF formats.[22][24]The data is stored in the file repository in the original format, and loaded in the database in alazy fashion, only when needed. The system can also process the data upon ingestion, if the data format requires it.[25]As a result, even very large file repositories can be efficiently analyzed, as only the required data is processed in the database. The data can be accessed through either the MonetDB SQL or SciQL interfaces. The Data Vault technology was used in theEuropean Union'sTELEIOS project, which was aimed at building avirtual observatory for Earth observation data.[24] Data Vaults for FITS files have also been used for processingastronomical survey data forThe INT Photometric H-Alpha Survey (IPHAS)[26][27]
MonetDB has aSAM/BAM module for efficient processing ofsequence alignment data. Aimed at thebioinformatics research, the module has a SAM/BAM data loader and a set of SQL UDFs for working withDNA data.[4] The module uses the popularSAMtools library.[28]
MonetDB/RDF is aSPARQL-based extension for working with linked data, which adds support forRDF and allowing MonetDB to function as atriplestore. Under development for theLinked Open Data 2 project.[3]
MonetDB/R module allows forUDFs written inR to be executed in the SQL layer of the system. This is done using the native R support for running embedded in another application, inside the RDBMS in this case. Previously theMonetDB.R connector allowed the using MonetDB data sources and process them in an R session. The newer R integration feature of MonetDB does not require data to be transferred between the RDBMS and the R session, reducing overhead and improving performance. The feature is intended to give users access to functions of the R statistical software for in-line analysis of data stored in the RDBMS. It complements the existing support forC UDFs and is intended to be used forin-database processing.[29]
Similarly to the embedded R UDFs in MonetDB, the database now has support for UDFs written inPython/NumPy. The implementation uses Numpy arrays (themselves Python wrappers for C arrays), as a result there is limited overhead - providing a functional Python integration with speed matching native SQL functions. The Embedded Python functions also support mapped operations, allowing user to execute Python functions in parallel within SQL queries. The practical side of the feature gives users access to Python/NumPy/SciPy libraries, which can provide a large selection of statistical/analytical functions.[30]
Following the release of an embedded driver for R and R UDFs in MonetDB (MonetDB/R), the authors created an embedded version of MonetDB in R calledMonetDBLite, embedded versions for Python and Java followed. They are distributed as embeddable packages, removing the need to manage a database server, required for the previous API integrations. The DBMS runs within the process itself, eliminating socket communication and serialisation overhead - greatly improving efficiency. The idea behind it is to easily embed anSQLite-like package with the performance of an in-memory optimized columnar store.[31]
A number of former extensions have been deprecated and removed from the stable code base over time. Some notable examples include anXQuery extension removed in MonetDB version 5; aJAQL extension, and astreaming data extension calledData Cell.[15][32][33]
The MonetDB Foundation is the independent non-profit organisation behind MonetDB. The foundation holds the intellectual property (IP) of MonetDB and is dedicated to advance the development and long-term maintenance of MonetDB. The foundation is funded by charitable donations.[34]
