Repository files navigation

Over the last few years, the Go client has become aprimary DuckDB client.We'd like to thankMarc Boeker for all his work on creating this driver and implementing the various interfaces of thedatabase/sql package!We'd also like to thank all the other external contributors for their various PRs and other contributions.

With the driver being a primary DuckDB client, over the last years, the DuckDB team has gradually increased its involvement in the maintenance of the driver,to guarantee that it is constantly updated and that critical bugs (e.g., crashes) are fixed.Now we have a Long-Term Support release, so starting from early next year, we will have two releases in parallel (v1.4 LTS and v1.5).Additionally, more DuckDB customers use and rely on the Go client, which necessitates prioritization of certain features from us.

These points all add to the maintenance work, which the DuckDB team is happy to perform!However, the motivation behind this fork, which is a joint effort of Marc Boeker and the DuckDB team,is to fully transfer the maintenance and day-to-day work of the driver to the DuckDB team.That being said, the DuckDB Go client has become what it is also due to its many contributions from the community,and we are looking forward to your future PRs, issues, and discussions!

The license is unchanged: the migrated repository keeps the original MIT license, which is the same for core DuckDB and other primary clients.

The DuckDB driver conforms to the built-indatabase/sql interface.

Current DuckDB version:v1.4.1.

The first go-duckdb tag with that version isv2.4.2.

Previous DuckDB versions:

Starting withv2, go-duckdb drops pre-built FreeBSD support.This change is because DuckDB does not publish any bundled FreeBSD libraries.Thus, you must build your static library for FreeBSD using the steps below.

TheDuckDB Arrow Interface is a heavy dependency.Starting withv2, the DuckDB Arrow Interface is opt-in instead of opt-out.If you want to use it, you can enable it by passing-tags=duckdb_arrow togo build.

The pre-built libraries ship DuckDB's JSON extension containing theJSON type.Pre-v2, it was possible to scan a JSON type into[]byte viaRows.Scan.However, scanning intoany (driver.Value) would cause the JSON string to contain escape characters and other unexpected behavior.

It is now possible to scan intoany, or directly into go-duckdb'sComposite type,as shown in theJSON example.Scanning directly intostring or[]byte is no longer possible.A workaround is casting to::VARCHAR or::BLOB in DuckDB if you do not need to scan the result into a JSON interface.

go get github.com/marcboeker/go-duckdb/v2

You must have the correct version of gcc and the necessary runtime libraries installed on Windows.One method to do this is using msys64.To begin, install msys64 using their installer.Once you installed msys64, open a msys64 shell and run:

pacman -S mingw-w64-ucrt-x86_64-gcc

Select "yes" when necessary; it is okay if the shell closes.Then, add gcc to the path using whatever method you prefer.In powershell this is$env:PATH = "C:\msys64\ucrt64\bin:$env:PATH".After, you can compile this package in Windows.

You can usego mod vendor to make a copy of the third-party packages in this package, including the pre-built DuckDB libraries induckdb-go-bindings.

Note: For readability, we omit error handling in most examples.

go-duckdb hooks into thedatabase/sql interface provided by the Gostdlib.To open a connection, specify the driver type asduckdb.

db,err:=sql.Open("duckdb","")deferdb.Close()

The above lines create an in-memory instance of DuckDB.To open a persistent database, specify a file path to the database file.If the file does not exist, then DuckDB creates it.

db,err:=sql.Open("duckdb","/path/to/foo.db")deferdb.Close()

If you want to set specificconfig options for DuckDB,you can add them as query style parameters in the form ofname=value pairs to the DSN.

db,err:=sql.Open("duckdb","/path/to/foo.db?access_mode=read_only&threads=4")deferdb.Close()

Alternatively, you can usesql.OpenDB.That way, you can perform initialization steps in a callback function before opening the database.Here's an example that configures some parameters when opening a database withsql.OpenDB(connector).

c,err:=duckdb.NewConnector("/path/to/foo.db?access_mode=read_only&threads=4",func(execer driver.ExecerContext)error {bootQueries:= []string{`SET schema=main`,`SET search_path=main`,    }for_,query:=rangebootQueries {_,err=execer.ExecContext(context.Background(),query,nil)iferr!=nil {returnerr        }    }returnnil})deferc.Close()db:=sql.OpenDB(c)deferdb.Close()

Please refer to thedatabase/sql documentation for further instructions on usage.

By default,go-duckdb statically links pre-built DuckDB libraries into your binary.Statically linking DuckDB increases your binary size.

go-duckdb bundles the following pre-compiled static libraries.

• MacOS: amd64, arm64.
• Linux: amd64, arm64.
• Windows: amd64.

If none of the pre-built libraries satisfy your needs, you can build a custom static library.

1. Clone and build the DuckDB source code.
   • Use theirbundle-library Makefile target (e.g.,make bundle-library).
   • Common build flags are:DUCKDB_PLATFORM=any BUILD_EXTENSIONS="icu;json;parquet;autocomplete".
   • See DuckDB'sdevelopment instructions for more details.
2. Link against the resulting static library, which you can find in:duckdb/build/release/libduckdb_bundle.a.

For Darwin ARM64, you can then build your module like so:

CGO_ENABLED=1 CPPFLAGS="-DDUCKDB_STATIC_BUILD" CGO_LDFLAGS="-lduckdb_bundle -lc++ -L/path/to/libs" go build -tags=duckdb_use_static_lib

You can also find these steps in theMakefile and thetests.yaml.

The DuckDB team also publishes pre-built libraries as part of theirreleases.The published zipped archives contain libraries for DuckDB core, the third-party libraries, and the default extensions.When linking, you might want to bundle these libraries into a single archive first.You can use any archive tool (e.g.,ar).DuckDB'sbundle-library Makefile target contains an example ofar, or you can look at the Docker filehere.

Starting withv2, go-duckdb drops pre-built FreeBSD support.This change is because DuckDB does not publish any bundled FreeBSD libraries.Thus, you must build your static library for FreeBSD using the steps above.

Alternatively, you can dynamically link DuckDB by passing-tags=duckdb_use_lib togo build.You must have a copy oflibduckdb available on your system (.so on Linux or.dylib on macOS),which you can download from the DuckDBreleases page.

For example:

# On Linux.CGO_ENABLED=1 CGO_LDFLAGS="-lduckdb -L/path/to/libs" go build -tags=duckdb_use_lib main.goLD_LIBRARY_PATH=/path/to/libs ./main# On MacOS.CGO_ENABLED=1 CGO_LDFLAGS="-lduckdb -L/path/to/libs" go build -tags=duckdb_use_lib main.goDYLD_LIBRARY_PATH=/path/to/libs ./main

You can also find these steps in theMakefile and thetests.yaml.

undefined: conn

Some people encounter anundefined: conn error when building this package.This error is due to the Go compiler determining that CGO is unavailable.This error can happen due to a few issues.

The first cause, as noted in thecomment here,might be that thebuildtools are not installed.To fix this for ubuntu, you can install them using:

sudo apt-get update && sudo apt-get install build-essential

Another cause can be cross-compilation since the Go compiler automatically disables CGO when cross-compiling.To enable CGO when cross-compiling, useCC={C cross compiler} CGO_ENABLED=1 {command} to force-enable CGO and set the right cross-compiler.

TIMESTAMP vs. TIMESTAMP_TZ

In the C API, DuckDB stores bothTIMESTAMP andTIMESTAMP_TZ asduckdb_timestamp, which holds the number ofmicroseconds elapsed since January 1, 1970, UTC (i.e., an instant without offset information).When passing atime.Time to go-duckdb, go-duckdb transforms it to an instant withUnixMicro(),even when usingTIMESTAMP_TZ. Later, scanning either type of value returns an instant, as SQL types do not modeltime zone information for individual values.

Connection lifetime

Temporary objects and state, such as temporary tables, are scoped to connections.When closing a connection, Go'sdatabase.sql pooling logic might cache it as an idle connection,instead of invoking its clean-up code by closing the connection.That behavior can lead to, e.g., temporary tables persisting longer than expected.To disable keeping idle connections alive, usedb.SetMaxIdleConns(0).

DuckDB lives in process.Therefore, all its memory lives in the driver.All allocations live in the host process, which is the Go application.Especially for long-running applications, it is crucial to call the correspondingClose-functions as specified indatabase/sql.

Additionally, it is crucial to callClose() on the database and/or connector of a persistent DuckDB database.That way, DuckDB synchronizes all changes from the WAL to its persistent storage.

The following is a list of examples ofClose()-functions.

db,err:=sql.Open("duckdb","")deferdb.Close()conn,err:=db.Conn(context.Background())deferconn.Close()rows,err:=conn.QueryContext(context.Background(),"SELECT 42")// Alternatively, rows.Next() has to return false.rows.Close()appender,err:=duckdb.NewAppenderFromConn(conn,"","test")deferappender.Close()c,err:=duckdb.NewConnector("",nil)// Optional, if passed to sql.OpenDB.deferc.Close()

If you want to use theDuckDB Appender API, you can obtain a newAppender by passing a DuckDB connection toNewAppenderFromConn().Seeexamples/appender.go for a complete example.

c,err:=duckdb.NewConnector("test.db",nil)deferc.Close()conn,err:=c.Connect(context.Background())deferconn.Close()// Obtain an appender from the connection.// NOTE: The table 'test_tbl' must exist in test.db.appender,err:=NewAppenderFromConn(conn,"","test_tbl")deferappender.Close()err= appender.AppendRow(...)

This section describes using theDuckDB Profiling API.DuckDB's profiling information is connection-local.The following example walks you through the necessary steps to obtain theProfilingInfo type, which contains all available metrics.Please refer to theDuckDB documentation on configuring and collecting specific metrics.

• First, you need to obtain a connection.
• Then, you enable profiling for the connection.
• Now, for each subsequent query on this connection, DuckDB will collect profiling information.
  • Optionally, you can turn off profiling at any point.
• Next, you execute the query for which you want to obtain profiling information.
• Finally, directly after executing the query, retrieve any available profiling information.

db,err:=sql.Open("duckdb","")deferdb.Close()conn,err:=db.Conn(context.Background())deferconn.Close()_,err=conn.ExecContext(context.Background(),`PRAGMA enable_profiling = 'no_output'`)_,err=conn.ExecContext(context.Background(),`PRAGMA profiling_mode = 'detailed'`)res,err:=conn.QueryContext(context.Background(),`SELECT 42`)deferres.Close()info,err:=GetProfilingInfo(conn)_,err=conn.ExecContext(context.Background(),`PRAGMA disable_profiling`)

TheDuckDB Arrow Interface is a heavy dependency.Starting withv2, the DuckDB Arrow Interface is opt-in instead of opt-out.If you want to use it, you can enable it by passing-tags=duckdb_arrow togo build.

go build -tags="duckdb_arrow"

You can obtain a newArrow by passing a DuckDB connection toNewArrowFromConn().

c,err:=duckdb.NewConnector("",nil)deferc.Close()conn,err:=c.Connect(context.Background())deferconn.Close()// Obtain the Arrow from the connection.arrow,err:=duckdb.NewArrowFromConn(conn)rdr,err:=arrow.QueryContext(context.Background(),"SELECT * FROM generate_series(1, 10)")deferrdr.Release()forrdr.Next() {// Process each record.}

go-duckdb relies on theduckdb-go-bindings module.Any pre-built library induckdb-go-bindings statically links the default extensions: ICU, JSON, Parquet, and Autocomplete.Additionally, automatic extension loading is enabled.

1. Create a new branch.
2. Update theduckdb-go-bindings dependencies inmapping/mod.go andarrowmapping/mod.go.
3. Rungo mod tidy insidemapping and insidearrowmapping.
4. Commit and PR changes.
5. Push two new tagged releases,mapping/vx.x.x andarrowmapping/vx.x.x.

1. Create a new branch.
2. Update themapping andarrowmapping dependencies inmod.go.
3. Rungo mod tidy.
4. UpdateVERSION= intests.yaml.
5. Update the latest version inREADME.md.
6. Commit and PR changes.
7. Push a new tagged release,vx.x.x.

git tag <tagname>git push origin <tagname>

Example PRs:

• Update the Mappings:#473
• Update the Main Module:#474