Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

Commit53b12f9

Browse files
authored
feat: add native database support for e2e tests without Docker (#4236)
1 parenta8b20cc commit53b12f9

File tree

7 files changed

+629
-16
lines changed

7 files changed

+629
-16
lines changed

‎internal/endtoend/CLAUDE.md‎

Lines changed: 117 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,117 @@
1+
#End-to-End Tests - Native Database Setup
2+
3+
This document describes how to set up MySQL and PostgreSQL for running end-to-end tests in environments without Docker, particularly when using an HTTP proxy.
4+
5+
##Overview
6+
7+
The end-to-end tests support three methods for connecting to databases:
8+
9+
1.**Environment Variables**: Set`POSTGRESQL_SERVER_URI` and`MYSQL_SERVER_URI` directly
10+
2.**Docker**: Automatically starts containers via the docker package
11+
3.**Native Installation**: Starts existing database services on Linux
12+
13+
##Installing Databases with HTTP Proxy
14+
15+
In environments where DNS doesn't work directly but an HTTP proxy is available (e.g., some CI environments), you need to configure apt to use the proxy before installing packages.
16+
17+
###Configure apt Proxy
18+
19+
```bash
20+
# Check if HTTP_PROXY is set
21+
echo$HTTP_PROXY
22+
23+
# Configure apt to use the proxy
24+
sudo tee /etc/apt/apt.conf.d/99proxy<<EOF
25+
Acquire::http::Proxy "$HTTP_PROXY";
26+
Acquire::https::Proxy "$HTTPS_PROXY";
27+
EOF
28+
29+
# Update package lists
30+
sudo apt-get update -qq
31+
```
32+
33+
###Install PostgreSQL
34+
35+
```bash
36+
# Install PostgreSQL
37+
sudo DEBIAN_FRONTEND=noninteractive apt-get install -y postgresql postgresql-contrib
38+
39+
# Start the service
40+
sudo service postgresql start
41+
42+
# Set password for postgres user
43+
sudo -u postgres psql -c"ALTER USER postgres PASSWORD 'postgres';"
44+
45+
# Configure pg_hba.conf for password authentication
46+
# Find the hba_file location:
47+
sudo -u postgres psql -t -c"SHOW hba_file;"
48+
49+
# Add md5 authentication for localhost (add to the beginning of pg_hba.conf):
50+
# host all all 127.0.0.1/32 md5
51+
52+
# Reload PostgreSQL
53+
sudo service postgresql reload
54+
```
55+
56+
###Install MySQL
57+
58+
```bash
59+
# Pre-configure MySQL root password
60+
echo"mysql-server mysql-server/root_password password mysecretpassword"| sudo debconf-set-selections
61+
echo"mysql-server mysql-server/root_password_again password mysecretpassword"| sudo debconf-set-selections
62+
63+
# Install MySQL
64+
sudo DEBIAN_FRONTEND=noninteractive apt-get install -y mysql-server
65+
66+
# Start the service
67+
sudo service mysql start
68+
69+
# Verify connection
70+
mysql -uroot -pmysecretpassword -e"SELECT 1;"
71+
```
72+
73+
##Expected Database Credentials
74+
75+
The native database support expects the following credentials:
76+
77+
###PostgreSQL
78+
-**URI**:`postgres://postgres:postgres@localhost:5432/postgres?sslmode=disable`
79+
-**User**:`postgres`
80+
-**Password**:`postgres`
81+
-**Port**:`5432`
82+
83+
###MySQL
84+
-**URI**:`root:mysecretpassword@tcp(localhost:3306)/mysql?multiStatements=true&parseTime=true`
85+
-**User**:`root`
86+
-**Password**:`mysecretpassword`
87+
-**Port**:`3306`
88+
89+
##Running Tests
90+
91+
```bash
92+
# Run end-to-end tests
93+
gotest -v -run TestReplay -timeout 20m ./internal/endtoend/...
94+
95+
# With verbose logging
96+
gotest -v -run TestReplay -timeout 20m ./internal/endtoend/...2>&1| tee test.log
97+
```
98+
99+
##Troubleshooting
100+
101+
###apt-get times out or fails
102+
- Ensure HTTP proxy is configured in`/etc/apt/apt.conf.d/99proxy`
103+
- Check that the proxy URL is correct:`echo $HTTP_PROXY`
104+
- Try running`sudo apt-get update` first to verify connectivity
105+
106+
###MySQL connection refused
107+
- Check if MySQL is running:`sudo service mysql status`
108+
- Verify the password:`mysql -uroot -pmysecretpassword -e "SELECT 1;"`
109+
- Check if MySQL is listening on TCP:`netstat -tlnp | grep 3306`
110+
111+
###PostgreSQL authentication failed
112+
- Verify pg_hba.conf has md5 authentication for localhost
113+
- Check password:`PGPASSWORD=postgres psql -h localhost -U postgres -c "SELECT 1;"`
114+
- Reload PostgreSQL after config changes:`sudo service postgresql reload`
115+
116+
###DNS resolution fails
117+
This is expected in some environments. Configure apt proxy as shown above.

‎internal/endtoend/endtoend_test.go‎

Lines changed: 56 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@ import (
1818
"github.com/sqlc-dev/sqlc/internal/config"
1919
"github.com/sqlc-dev/sqlc/internal/opts"
2020
"github.com/sqlc-dev/sqlc/internal/sqltest/docker"
21+
"github.com/sqlc-dev/sqlc/internal/sqltest/native"
2122
)
2223

2324
funclineEndings() cmp.Option {
@@ -113,23 +114,63 @@ func TestReplay(t *testing.T) {
113114
ctx:=context.Background()
114115

115116
varmysqlURI,postgresURIstring
116-
iferr:=docker.Installed();err==nil {
117-
{
118-
host,err:=docker.StartPostgreSQLServer(ctx)
119-
iferr!=nil {
120-
t.Fatalf("starting postgresql failed: %s",err)
117+
118+
// First, check environment variables
119+
ifuri:=os.Getenv("POSTGRESQL_SERVER_URI");uri!="" {
120+
postgresURI=uri
121+
}
122+
ifuri:=os.Getenv("MYSQL_SERVER_URI");uri!="" {
123+
mysqlURI=uri
124+
}
125+
126+
// Try Docker for any missing databases
127+
ifpostgresURI==""||mysqlURI=="" {
128+
iferr:=docker.Installed();err==nil {
129+
ifpostgresURI=="" {
130+
host,err:=docker.StartPostgreSQLServer(ctx)
131+
iferr!=nil {
132+
t.Logf("docker postgresql startup failed: %s",err)
133+
}else {
134+
postgresURI=host
135+
}
136+
}
137+
ifmysqlURI=="" {
138+
host,err:=docker.StartMySQLServer(ctx)
139+
iferr!=nil {
140+
t.Logf("docker mysql startup failed: %s",err)
141+
}else {
142+
mysqlURI=host
143+
}
121144
}
122-
postgresURI=host
123145
}
124-
{
125-
host,err:=docker.StartMySQLServer(ctx)
126-
iferr!=nil {
127-
t.Fatalf("starting mysql failed: %s",err)
146+
}
147+
148+
// Try native installation for any missing databases (Linux only)
149+
ifpostgresURI==""||mysqlURI=="" {
150+
iferr:=native.Supported();err==nil {
151+
ifpostgresURI=="" {
152+
host,err:=native.StartPostgreSQLServer(ctx)
153+
iferr!=nil {
154+
t.Logf("native postgresql startup failed: %s",err)
155+
}else {
156+
postgresURI=host
157+
}
158+
}
159+
ifmysqlURI=="" {
160+
host,err:=native.StartMySQLServer(ctx)
161+
iferr!=nil {
162+
t.Logf("native mysql startup failed: %s",err)
163+
}else {
164+
mysqlURI=host
165+
}
128166
}
129-
mysqlURI=host
130167
}
131168
}
132169

170+
// Log which databases are available
171+
t.Logf("PostgreSQL available: %v (URI: %s)",postgresURI!="",postgresURI)
172+
t.Logf("MySQL available: %v (URI: %s)",mysqlURI!="",mysqlURI)
173+
133174
contexts:=map[string]textContext{
134175
"base": {
135176
Mutate:func(t*testing.T,pathstring)func(*config.Config) {returnfunc(c*config.Config) {} },
@@ -138,19 +179,20 @@ func TestReplay(t *testing.T) {
138179
"managed-db": {
139180
Mutate:func(t*testing.T,pathstring)func(*config.Config) {
140181
returnfunc(c*config.Config) {
182+
// Add all servers - tests will fail if database isn't available
141183
c.Servers= []config.Server{
142184
{
143185
Name:"postgres",
144186
Engine:config.EnginePostgreSQL,
145187
URI:postgresURI,
146188
},
147-
148189
{
149190
Name:"mysql",
150191
Engine:config.EngineMySQL,
151192
URI:mysqlURI,
152193
},
153194
}
195+
154196
fori:=rangec.SQL {
155197
switchc.SQL[i].Engine {
156198
caseconfig.EnginePostgreSQL:
@@ -172,8 +214,8 @@ func TestReplay(t *testing.T) {
172214
}
173215
},
174216
Enabled:func()bool {
175-
err:=docker.Installed()
176-
returnerr==nil
217+
// Enabled if at least one database URI is available
218+
returnpostgresURI!=""||mysqlURI!=""
177219
},
178220
},
179221
}

‎internal/sqltest/local/mysql.go‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@ import (
1414
migrate"github.com/sqlc-dev/sqlc/internal/migrations"
1515
"github.com/sqlc-dev/sqlc/internal/sql/sqlpath"
1616
"github.com/sqlc-dev/sqlc/internal/sqltest/docker"
17+
"github.com/sqlc-dev/sqlc/internal/sqltest/native"
1718
)
1819

1920
varmysqlSync sync.Once
@@ -31,8 +32,15 @@ func MySQL(t *testing.T, migrations []string) string {
3132
t.Fatal(err)
3233
}
3334
dburi=u
35+
}elseifierr:=native.Supported();ierr==nil {
36+
// Fall back to native installation when Docker is not available
37+
u,err:=native.StartMySQLServer(ctx)
38+
iferr!=nil {
39+
t.Fatal(err)
40+
}
41+
dburi=u
3442
}else {
35-
t.Skip("MYSQL_SERVER_URI is empty")
43+
t.Skip("MYSQL_SERVER_URI is empty and neither Docker nor native installation is available")
3644
}
3745
}
3846

‎internal/sqltest/local/postgres.go‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ import (
1616
"github.com/sqlc-dev/sqlc/internal/pgx/poolcache"
1717
"github.com/sqlc-dev/sqlc/internal/sql/sqlpath"
1818
"github.com/sqlc-dev/sqlc/internal/sqltest/docker"
19+
"github.com/sqlc-dev/sqlc/internal/sqltest/native"
1920
)
2021

2122
varflight singleflight.Group
@@ -41,8 +42,15 @@ func postgreSQL(t *testing.T, migrations []string, rw bool) string {
4142
t.Fatal(err)
4243
}
4344
dburi=u
45+
}elseifierr:=native.Supported();ierr==nil {
46+
// Fall back to native installation when Docker is not available
47+
u,err:=native.StartPostgreSQLServer(ctx)
48+
iferr!=nil {
49+
t.Fatal(err)
50+
}
51+
dburi=u
4452
}else {
45-
t.Skip("POSTGRESQL_SERVER_URI is empty")
53+
t.Skip("POSTGRESQL_SERVER_URI is empty and neither Docker nor native installation is available")
4654
}
4755
}
4856

‎internal/sqltest/native/enabled.go‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
package native
2+
3+
import (
4+
"fmt"
5+
"os/exec"
6+
"runtime"
7+
)
8+
9+
// Supported returns nil if native database installation is supported on this platform.
10+
// Currently only Linux (Ubuntu/Debian) is supported.
11+
funcSupported()error {
12+
ifruntime.GOOS!="linux" {
13+
returnfmt.Errorf("native database installation only supported on linux, got %s",runtime.GOOS)
14+
}
15+
// Check if apt-get is available (Debian/Ubuntu)
16+
if_,err:=exec.LookPath("apt-get");err!=nil {
17+
returnfmt.Errorf("apt-get not found: %w",err)
18+
}
19+
returnnil
20+
}

0 commit comments

Comments
 (0)
[8]ページ先頭
©2009-2025 Movatter.jp