dolthub/go-mysql-serverPublic

NotificationsYou must be signed in to change notification settings
Fork238
Star2.6k

A MySQL-compatible relational database with a storage agnostic query engine. Implemented in pure Go.

License

Apache-2.0 license

2.6k stars 238 forks Branches Tags Activity

Star

Notifications

You must be signed in to change notification settings

Branches Tags

Folders and files

Name		Name	Last commit message	Last commit date
Latest commit History 12,591 Commits
.github		.github
_example		_example
_integration		_integration
driver		driver
enginetest		enginetest
eventscheduler		eventscheduler
internal		internal
memory		memory
optgen		optgen
server		server
sql		sql
test		test
.gitattributes		.gitattributes
.gitignore		.gitignore
ARCHITECTURE.md		ARCHITECTURE.md
BACKEND.md		BACKEND.md
CONTRIBUTING.md		CONTRIBUTING.md
LICENSE		LICENSE
MAINTAINERS		MAINTAINERS
Makefile		Makefile
README.md		README.md
SECURITY.md		SECURITY.md
SUPPORTED_CLIENTS.md		SUPPORTED_CLIENTS.md
check_repo.sh		check_repo.sh
engine.go		engine.go
engine_test.go		engine_test.go
format_repo.sh		format_repo.sh
go.mod		go.mod
go.sum		go.sum
log.go		log.go
mascot.png		mascot.png
processlist.go		processlist.go
processlist_test.go		processlist_test.go
tools.go		tools.go

Repository files navigation

A MySQL compatible database engine written in Go

go-mysql-server is a data-source agnostic SQL engine and serverwhich runs queries on data sources you provide, using the MySQLdialect and wire protocol. A simple in-memory database implementationis included, and you can query any data source you want byimplementing your own backend.

Dolt, a SQL database with Git-styleversioning, is the main production database implementation of thispackage.Checkout that projectfor a reference implementation. Or, hop into the Dolt Discord serverhere if you want to talk to thecore developers behindgo-mysql-server and Dolt.

Compatibility

With the exception of specific limitations (see below),go-mysql-server is a drop-in replacement for MySQL. Any clientlibrary, tool, query, SQL syntax, SQL function, etc. that works withMySQL should also work withgo-mysql-server. If you find a gap infunctionality, please file an issue.

For full MySQL compatibility documentation, see theDoltdocs on thistopic.

Scope of this project

SQL server and engine to query your data sources.
In-memory database backend implementation suitable for use in tests.
Interfaces you can use to implement new backends to query your owndata sources.
With a few caveats and using a full database implementation, adrop-in MySQL database replacement.

go-mysql-server has two primary uses case:

Stand-in for MySQL in a golang test environment, using the built-inmemory database implementation.
Providing access to arbitrary data sources with SQL queries byimplementing a handful of interfaces. The most complete real-worldimplementation isDolt.

Installation

Addgo-mysql-server as a dependency to your project. In thedirectory with thego.mod file, run:

go get github.com/dolthub/go-mysql-server@latest

To implement ICU-compatible regexes,go-mysql-server has a dependency ongo-icu-regex, which has a Cgo dependency onICU4C. To build a projectwhich depends ongo-mysql-server, you should have a C/C++ toolchain, youshould build with Cgo enabled, and you should have libicu-dev, or theequivalent for your environment, installed and available to your C++ toolchain.

For convenience,go-mysql-server also includes a non-compatible regeximplementation based on the Go standard libraryregex.Regex. To build againstthat, instead of thego-icu-regex implementation, you must compile with-tags=gms_pure_go. Please note that some of go-mysql-server's tests do notpass with-tags=gms_pure_go and in generalgms_pure_go is not recommendedfor users seeking MySQL compatibility.

Using the in-memory test server

The in-memory test server can replace a real MySQL server intests. Start the server using the code in the_exampledirectory, also reproduced below.

package mainimport ("context""fmt""time""github.com/dolthub/vitess/go/vt/proto/query"sqle"github.com/dolthub/go-mysql-server""github.com/dolthub/go-mysql-server/memory""github.com/dolthub/go-mysql-server/server""github.com/dolthub/go-mysql-server/sql""github.com/dolthub/go-mysql-server/sql/types")// This is an example of how to implement a MySQL server.// After running the example, you may connect to it using the following://// > mysql --host=localhost --port=3306 --user=root mydb --execute="SELECT * FROM mytable;"// +----------+-------------------+-------------------------------+----------------------------+// | name     | email             | phone_numbers                 | created_at                 |// +----------+-------------------+-------------------------------+----------------------------+// | Jane Deo | janedeo@gmail.com | ["556-565-566","777-777-777"] | 2022-11-01 12:00:00.000001 |// | Jane Doe | jane@doe.com      | []                            | 2022-11-01 12:00:00.000001 |// | John Doe | john@doe.com      | ["555-555-555"]               | 2022-11-01 12:00:00.000001 |// | John Doe | johnalt@doe.com   | []                            | 2022-11-01 12:00:00.000001 |// +----------+-------------------+-------------------------------+----------------------------+//// The included MySQL client is used in this example, however any MySQL-compatible client will work.var (dbName="mydb"tableName="mytable"address="localhost"port=3306)funcmain() {pro:=createTestDatabase()engine:=sqle.NewDefault(pro)session:=memory.NewSession(sql.NewBaseSession(),pro)ctx:=sql.NewContext(context.Background(),sql.WithSession(session))ctx.SetCurrentDatabase(dbName)// This variable may be found in the "users_example.go" file. Please refer to that file for a walkthrough on how to// set up the "mysql" database to allow user creation and user checking when establishing connections. This is set// to false for this example, but feel free to play around with it and see how it works.ifenableUsers {iferr:=enableUserAccounts(ctx,engine);err!=nil {panic(err)}}config:= server.Config{Protocol:"tcp",Address:fmt.Sprintf("%s:%d",address,port),}s,err:=server.NewServer(config,engine,sql.NewContext,memory.NewSessionBuilder(pro),nil)iferr!=nil {panic(err)}iferr=s.Start();err!=nil {panic(err)}}funccreateTestDatabase()*memory.DbProvider {db:=memory.NewDatabase(dbName)db.BaseDatabase.EnablePrimaryKeyIndexes()pro:=memory.NewDBProvider(db)session:=memory.NewSession(sql.NewBaseSession(),pro)ctx:=sql.NewContext(context.Background(),sql.WithSession(session))table:=memory.NewTable(db,tableName,sql.NewPrimaryKeySchema(sql.Schema{{Name:"name",Type:types.Text,Nullable:false,Source:tableName,PrimaryKey:true},{Name:"email",Type:types.Text,Nullable:false,Source:tableName,PrimaryKey:true},{Name:"phone_numbers",Type:types.JSON,Nullable:false,Source:tableName},{Name:"created_at",Type:types.MustCreateDatetimeType(query.Type_DATETIME,6),Nullable:false,Source:tableName},}),db.GetForeignKeyCollection())db.AddTable(tableName,table)creationTime:=time.Unix(0,1667304000000001000).UTC()_=table.Insert(ctx,sql.NewRow("Jane Deo","janedeo@gmail.com",types.MustJSON(`["556-565-566", "777-777-777"]`),creationTime))_=table.Insert(ctx,sql.NewRow("Jane Doe","jane@doe.com",types.MustJSON(`[]`),creationTime))_=table.Insert(ctx,sql.NewRow("John Doe","john@doe.com",types.MustJSON(`["555-555-555"]`),creationTime))_=table.Insert(ctx,sql.NewRow("John Doe","johnalt@doe.com",types.MustJSON(`[]`),creationTime))returnpro}

This example populates the database by creatingmemory.Database andmemory.Table objects via golang code, but you can also populate itby issuingCREATE DATABASE,CREATE TABLE, etc. statements to theserver once it's running.

Once the server is running, connect with any MySQL client, includingthe golang MySQL connector and themysql shell.

> mysql --host=localhost --port=3306 --user=root mydb --execute="SELECT * FROM mytable;"+----------+-------------------+-------------------------------+----------------------------+| name| email| phone_numbers| created_at|+----------+-------------------+-------------------------------+----------------------------+| Jane Deo| janedeo@gmail.com| ["556-565-566","777-777-777"]| 2022-11-01 12:00:00.000001|| Jane Doe| jane@doe.com| []| 2022-11-01 12:00:00.000001|| John Doe| john@doe.com| ["555-555-555"]| 2022-11-01 12:00:00.000001|| John Doe| johnalt@doe.com| []| 2022-11-01 12:00:00.000001|+----------+-------------------+-------------------------------+----------------------------+

Limitations of the in-memory database implementation

The in-memory database implementation included with this package isintended for use in tests. It has specific limitations that we knowof:

Notthreadsafe. Toavoid concurrency issues, limit DDL and DML statements (CREATE TABLE,INSERT, etc.) to a single goroutine.
No transactionsupport. StatementslikeSTART TRANSACTION,ROLLBACK, andCOMMIT are no-ops.
Non-performant indeximplementation. Indexedlookups and joins perform full table scans on the underlying tables.

Custom backend implementations

You can create your own backend to query your own data sources byimplementing some interfaces. For detailed instructions, see thebackend guide.

Technical documentation for contributors and backend developers

Architecture is an overview of the variouspackages of the project and how they fit together.
Contribution guide for new contributors,including instructions for how to get your PR merged.

Powered by go-mysql-server

dolt
Grafana
gitbase (defunct)

Are you building a database backend usinggo-mysql-server? Wewould like to hear from you and include you in this list.

Security Policy

go-mysql-server's securitypolicy ismaintained in this repository. Please follow the disclosure instructions there.Please do not initially report security issues in this repository's publicGitHub issues.

Acknowledgements

go-mysql-server was originally developed by the{source-d}organzation, and this repository was originally forked fromsrc-d. We want to thankthe entire{source-d} development team for their work on thisproject, especially Miguel Molina (@erizocosmico) and Juanjo ÁlvarezMartinez (@juanjux).