koxudaxi/local-data-apiPublic

NotificationsYou must be signed in to change notification settings
Fork21
Star114

Data API for local, you can write unittest for AWS Aurora Serverless's Data API

License

MIT license

114 stars 21 forks Branches Tags Activity

Star

Notifications

You must be signed in to change notification settings

Branches Tags

Folders and files

Name		Name	Last commit message	Last commit date
Latest commit History 161 Commits
.github		.github
docs		docs
kotlin/local-data-api		kotlin/local-data-api
local_data_api		local_data_api
scripts		scripts
tests		tests
.coveragerc		.coveragerc
.gitignore		.gitignore
CODE_OF_CONDUCT.md		CODE_OF_CONDUCT.md
Dockerfile		Dockerfile
FUNDING.yml		FUNDING.yml
LICENSE		LICENSE
README.md		README.md
codecov.yml		codecov.yml
docker-compose-mysql.yml		docker-compose-mysql.yml
docker-compose-postgres.yml		docker-compose-postgres.yml
docker-compose.override-ci.yml		docker-compose.override-ci.yml
docker-compose.yml		docker-compose.yml
mkdocs.yml		mkdocs.yml
mypy.ini		mypy.ini
pyproject.toml		pyproject.toml
requirements-docs.txt		requirements-docs.txt
setup.cfg		setup.cfg
setup.py		setup.py

Repository files navigation

local-data-api - Local Data API for AWS Aurora Serverless Data API

If you want to run tests on your local machine and CI then, local-data-api can run in your local machine with MySQL and PostgreSQL Servers.

What's AWS Aurora Serverless's Data API?

https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/data-api.html

How does local-data-api work?

local-data-api is "proxy server" to real databases.

The API converts RESTful request to SQL statements.

Support Database Types

MySQL
PostgreSQL

How to insert array data into PostgreSQL

DataAPI have not support inserting array data withSqlParameter yet.But, @ormu5 give us this workaround to insert array data.

insert intocfg.attributes(id, type, attributes)values(:id, :type, cast(:attributesastext[]));

where the value for attributes parameter is a string properly formatted as Postgres array, e.g.,'{"Volume","Material"}'.

Thanks to @ormu5.

Version 0.6.0

local-data-api has been re-written in Kotlin

Motivation

This project was written in Python at the start.

But, We need a JDBC driver that needs Java to reproduce real Data API behavior.

I have re-written local-data-api in Kotlin. The design is the same as the old version.

Also, I wrote unittest and made coverage 100%. We welcome your feedback and error reports.

Related issue:#70

How to use this image

You set your MYSQL Server configs as environments.

docker run --rm -it --name my-data-api -p 8080:80  -e MYSQL_HOST=<YOUR_MYSQL_HOST> -e MYSQL_PORT=<YOUR_MYSQL_PORT> -e MYSQL_USER=<YOUR_MYSQL_USER> -e MYSQL_PASSWORD=<YOUR_MYSQL_PASS>  -e RESOURCE_ARN=arn:aws:rds:us-east-1:123456789012:cluster:dummy -e SECRET_ARN=arn:aws:secretsmanager:us-east-1:123456789012:secret:dummy  koxudaxi/local-data-api

In this case, you give local-data-api URL to aws client (like aws-cli).

$ aws --endpoint-url http://127.0.0.1:8080 rds-data execute-statement --resource-arn"arn:aws:rds:us-east-1:123456789012:cluster:dummy" --sql"show databases"  --secret-arn"arn:aws:secretsmanager:us-east-1:123456789012:secret:dummy" --database'test'

docker-compose

MySQL

docker-compose-mysql.yml

version:'3.1'services:local-data-api:image:koxudaxi/local-data-apirestart:alwaysenvironment:MYSQL_HOST:dbMYSQL_PORT:3306MYSQL_USER:rootMYSQL_PASSWORD:exampleRESOURCE_ARN:'arn:aws:rds:us-east-1:123456789012:cluster:dummy'SECRET_ARN:'arn:aws:secretsmanager:us-east-1:123456789012:secret:dummy'ports:      -"8080:80"db:image:mysql:5.6command:--default-authentication-plugin=mysql_native_passwordrestart:alwaysenvironment:MYSQL_ROOT_PASSWORD:exampleMYSQL_DATABASE:testports:        -"3306:3306"

docker-compose with Python's aws-sdk client(boto3)

start local-data-api containers

$ docker-compose up -d

change a endpoint to local-data-api in your code.

$ ipython

In [1]:importboto3;client=boto3.client('rds-data',endpoint_url='http://127.0.0.1:8080',aws_access_key_id='aaa',aws_secret_access_key='bbb')

execute a sql statement

In [2]:client.execute_statement(resourceArn='arn:aws:rds:us-east-1:123456789012:cluster:dummy',secretArn='arn:aws:secretsmanager:us-east-1:123456789012:secret:dummy',sql='show databases',database='test')

If you use PostgreSQL, then you should run this line to check databases.

In [2]:client.execute_statement(resourceArn='arn:aws:rds:us-east-1:123456789012:cluster:dummy',secretArn='arn:aws:secretsmanager:us-east-1:123456789012:secret:dummy',sql='SELECT datname FROM pg_database',database='test')

local-data-api return the result from a MySQL Server.

Out[2]: {'ResponseMetadata': {'HTTPStatusCode':200,'HTTPHeaders': {'date':'Sun, 09 Jun 2019 18:35:22 GMT','server':'uvicorn','content-length':'492','content-type':'application/json'},'RetryAttempts':0},'numberOfRecordsUpdated':0,'records': [[{'stringValue':'information_schema'}],  [{'stringValue':'mysql'}],  [{'stringValue':'performance_schema'}],  [{'stringValue':'sys'}],  [{'stringValue':'test'}]]}

If a table has some records, then the local-data-api can runselect

In [3]:client.execute_statement(resourceArn='arn:aws:rds:us-east-1:123456789012:cluster:dummy',secretArn='arn:aws:secretsmanager:us-east-1:123456789012:secret:dummy',sql='select * from users',database='test')

Out[3]: {'ResponseMetadata': {'HTTPStatusCode':200,'HTTPHeaders': {'date':'Sun, 09 Jun 2019 18:35:22 GMT','server':'uvicorn','content-length':'492','content-type':'application/json'},'RetryAttempts':0},'numberOfRecordsUpdated':0,'records': [[{'longValue':1}, {'stringValue':'ichiro'}, {'longValue':17}],  [{'longValue':2}, {'stringValue':'ken'}, {'longValue':20}],  [{'longValue':3}, {'stringValue':'lisa'}, {'isNull':True}],}

PostgreSQL

Now, local-data-api supports PostgreSQL

docker-compose-postgres.yml

version:'3.1'services:local-data-api:image:koxudaxi/local-data-apirestart:alwaysenvironment:ENGINE:PostgreSQLJDBCPOSTGRES_HOST:dbPOSTGRES_PORT:5432POSTGRES_USER:postgresPOSTGRES_PASSWORD:exampleRESOURCE_ARN:'arn:aws:rds:us-east-1:123456789012:cluster:dummy'SECRET_ARN:'arn:aws:secretsmanager:us-east-1:123456789012:secret:dummy'ports:      -"8080:80"db:image:postgres:10.7-alpinerestart:alwaysenvironment:POSTGRES_PASSWORD:examplePOSTGRES_DB:testports:        -"5432:5432"

Contribute

We are waiting for your contributions tolocal-data-api.

How to contribute

## 1. Clone your fork repository$ git clone git@github.com:<your username>/local-data-api.git$cd local-data-api## 2. Open the project with IDE/EditorThe project path is`./kotlin/local-data-api`## 3. Run on your local machine### Shell on Linux or MacOS$cd ./kotlin/local-data-api$ ./gradlew run### Command Prompt on Windows$cd ./kotlin/local-data-api$ gradlew.bat run## 4. Create new branch and rewrite code.$ git checkout -b new-branch## 5. Run unittest$cd ./kotlin/local-data-api$ ./gradlewtest jacocoTestReport## 6. Run integration-test$ ./scripts/integration-test.sh## 7. Commit and Push...