|
| 1 | +#Database Development Patterns |
| 2 | + |
| 3 | +##Database Work Overview |
| 4 | + |
| 5 | +###Database Generation Process |
| 6 | + |
| 7 | +1. Modify SQL files in`coderd/database/queries/` |
| 8 | +2. Run`make gen` |
| 9 | +3. If errors about audit table, update`enterprise/audit/table.go` |
| 10 | +4. Run`make gen` again |
| 11 | +5. Run`make lint` to catch any remaining issues |
| 12 | + |
| 13 | +##Migration Guidelines |
| 14 | + |
| 15 | +###Creating Migration Files |
| 16 | + |
| 17 | +**Location**:`coderd/database/migrations/` |
| 18 | +**Format**:`{number}_{description}.{up|down}.sql` |
| 19 | + |
| 20 | +- Number must be unique and sequential |
| 21 | +- Always include both up and down migrations |
| 22 | + |
| 23 | +###Helper Scripts |
| 24 | + |
| 25 | +| Script| Purpose| |
| 26 | +|--------|---------| |
| 27 | +|`./coderd/database/migrations/create_migration.sh "migration name"`| Creates new migration files| |
| 28 | +|`./coderd/database/migrations/fix_migration_numbers.sh`| Renumbers migrations to avoid conflicts| |
| 29 | +|`./coderd/database/migrations/create_fixture.sh "fixture name"`| Creates test fixtures for migrations| |
| 30 | + |
| 31 | +###Database Query Organization |
| 32 | + |
| 33 | +-**MUST DO**: Any changes to database - adding queries, modifying queries should be done in the`coderd/database/queries/*.sql` files |
| 34 | +-**MUST DO**: Queries are grouped in files relating to context - e.g.`prebuilds.sql`,`users.sql`,`oauth2.sql` |
| 35 | +- After making changes to any`coderd/database/queries/*.sql` files you must run`make gen` to generate respective ORM changes |
| 36 | + |
| 37 | +##Handling Nullable Fields |
| 38 | + |
| 39 | +Use`sql.NullString`,`sql.NullBool`, etc. for optional database fields: |
| 40 | + |
| 41 | +```go |
| 42 | +CodeChallenge: sql.NullString{ |
| 43 | +String: params.codeChallenge, |
| 44 | +Valid: params.codeChallenge !="", |
| 45 | +} |
| 46 | +``` |
| 47 | + |
| 48 | +Set`.Valid = true` when providing values. |
| 49 | + |
| 50 | +##Audit Table Updates |
| 51 | + |
| 52 | +If adding fields to auditable types: |
| 53 | + |
| 54 | +1. Update`enterprise/audit/table.go` |
| 55 | +2. Add each new field with appropriate action: |
| 56 | +-`ActionTrack`: Field should be tracked in audit logs |
| 57 | +-`ActionIgnore`: Field should be ignored in audit logs |
| 58 | +-`ActionSecret`: Field contains sensitive data |
| 59 | +3. Run`make gen` to verify no audit errors |
| 60 | + |
| 61 | +##In-Memory Database (dbmem) Updates |
| 62 | + |
| 63 | +###Critical Requirements |
| 64 | + |
| 65 | +When adding new fields to database structs: |
| 66 | + |
| 67 | +-**CRITICAL**: Update`coderd/database/dbmem/dbmem.go` in-memory implementations |
| 68 | +- The`Insert*` functions must include ALL new fields, not just basic ones |
| 69 | +- Common issue: Tests pass with real database but fail with in-memory database due to missing field mappings |
| 70 | +- Always verify in-memory database functions match the real database schema after migrations |
| 71 | + |
| 72 | +###Example Pattern |
| 73 | + |
| 74 | +```go |
| 75 | +// In dbmem.go - ensure ALL fields are included |
| 76 | +code:= database.OAuth2ProviderAppCode{ |
| 77 | +ID: arg.ID, |
| 78 | +CreatedAt: arg.CreatedAt, |
| 79 | +// ... existing fields ... |
| 80 | +ResourceUri: arg.ResourceUri,// New field |
| 81 | +CodeChallenge: arg.CodeChallenge,// New field |
| 82 | +CodeChallengeMethod: arg.CodeChallengeMethod,// New field |
| 83 | +} |
| 84 | +``` |
| 85 | + |
| 86 | +##Database Architecture |
| 87 | + |
| 88 | +###Core Components |
| 89 | + |
| 90 | +-**PostgreSQL 13+** recommended for production |
| 91 | +-**Migrations** managed with`migrate` |
| 92 | +-**Database authorization** through`dbauthz` package |
| 93 | + |
| 94 | +###Authorization Patterns |
| 95 | + |
| 96 | +```go |
| 97 | +// Public endpoints needing system access (OAuth2 registration) |
| 98 | +app,err:= api.Database.GetOAuth2ProviderAppByClientID(dbauthz.AsSystemRestricted(ctx), clientID) |
| 99 | + |
| 100 | +// Authenticated endpoints with user context |
| 101 | +app,err:= api.Database.GetOAuth2ProviderAppByClientID(ctx, clientID) |
| 102 | + |
| 103 | +// System operations in middleware |
| 104 | +roles,err:= db.GetAuthorizationUserRoles(dbauthz.AsSystemRestricted(ctx), userID) |
| 105 | +``` |
| 106 | + |
| 107 | +##Common Database Issues |
| 108 | + |
| 109 | +###Migration Issues |
| 110 | + |
| 111 | +1.**Migration conflicts**: Use`fix_migration_numbers.sh` to renumber |
| 112 | +2.**Missing down migration**: Always create both up and down files |
| 113 | +3.**Schema inconsistencies**: Verify against existing schema |
| 114 | + |
| 115 | +###Field Handling Issues |
| 116 | + |
| 117 | +1.**Nullable field errors**: Use`sql.Null*` types consistently |
| 118 | +2.**Missing audit entries**: Update`enterprise/audit/table.go` |
| 119 | +3.**dbmem inconsistencies**: Ensure in-memory implementations match schema |
| 120 | + |
| 121 | +###Query Issues |
| 122 | + |
| 123 | +1.**Query organization**: Group related queries in appropriate files |
| 124 | +2.**Generated code errors**: Run`make gen` after query changes |
| 125 | +3.**Performance issues**: Add appropriate indexes in migrations |
| 126 | + |
| 127 | +##Database Testing |
| 128 | + |
| 129 | +###Test Database Setup |
| 130 | + |
| 131 | +```go |
| 132 | +funcTestDatabaseFunction(t *testing.T) { |
| 133 | +db:= dbtestutil.NewDB(t) |
| 134 | + |
| 135 | +// Test with real database |
| 136 | +result,err:= db.GetSomething(ctx, param) |
| 137 | + require.NoError(t, err) |
| 138 | + require.Equal(t, expected, result) |
| 139 | +} |
| 140 | +``` |
| 141 | + |
| 142 | +###In-Memory Testing |
| 143 | + |
| 144 | +```go |
| 145 | +funcTestInMemoryDatabase(t *testing.T) { |
| 146 | +db:= dbmem.New() |
| 147 | + |
| 148 | +// Test with in-memory database |
| 149 | +result,err:= db.GetSomething(ctx, param) |
| 150 | + require.NoError(t, err) |
| 151 | + require.Equal(t, expected, result) |
| 152 | +} |
| 153 | +``` |
| 154 | + |
| 155 | +##Best Practices |
| 156 | + |
| 157 | +###Schema Design |
| 158 | + |
| 159 | +1.**Use appropriate data types**: VARCHAR for strings, TIMESTAMP for times |
| 160 | +2.**Add constraints**: NOT NULL, UNIQUE, FOREIGN KEY as appropriate |
| 161 | +3.**Create indexes**: For frequently queried columns |
| 162 | +4.**Consider performance**: Normalize appropriately but avoid over-normalization |
| 163 | + |
| 164 | +###Query Writing |
| 165 | + |
| 166 | +1.**Use parameterized queries**: Prevent SQL injection |
| 167 | +2.**Handle errors appropriately**: Check for specific error types |
| 168 | +3.**Use transactions**: For related operations that must succeed together |
| 169 | +4.**Optimize queries**: Use EXPLAIN to understand query performance |
| 170 | + |
| 171 | +###Migration Writing |
| 172 | + |
| 173 | +1.**Make migrations reversible**: Always include down migration |
| 174 | +2.**Test migrations**: On copy of production data if possible |
| 175 | +3.**Keep migrations small**: One logical change per migration |
| 176 | +4.**Document complex changes**: Add comments explaining rationale |
| 177 | + |
| 178 | +##Advanced Patterns |
| 179 | + |
| 180 | +###Complex Queries |
| 181 | + |
| 182 | +```sql |
| 183 | +-- Example: Complex join with aggregation |
| 184 | +SELECT |
| 185 | +u.id, |
| 186 | +u.username, |
| 187 | +COUNT(w.id)as workspace_count |
| 188 | +FROM users u |
| 189 | +LEFT JOIN workspaces wONu.id=w.owner_id |
| 190 | +WHEREu.created_at> $1 |
| 191 | +GROUP BYu.id,u.username |
| 192 | +ORDER BY workspace_countDESC; |
| 193 | +``` |
| 194 | + |
| 195 | +###Conditional Queries |
| 196 | + |
| 197 | +```sql |
| 198 | +-- Example: Dynamic filtering |
| 199 | +SELECT*FROM oauth2_provider_apps |
| 200 | +WHERE |
| 201 | + ($1::text ISNULLOR name ILIKE'%'|| $1||'%') |
| 202 | +AND ($2::uuid ISNULLOR organization_id= $2) |
| 203 | +ORDER BY created_atDESC; |
| 204 | +``` |
| 205 | + |
| 206 | +###Audit Patterns |
| 207 | + |
| 208 | +```go |
| 209 | +// Example: Auditable database operation |
| 210 | +func(q *sqlQuerier)UpdateUser(ctxcontext.Context,argUpdateUserParams) (User,error) { |
| 211 | +// Implementation here |
| 212 | + |
| 213 | +// Audit the change |
| 214 | +ifauditor:= audit.FromContext(ctx); auditor !=nil { |
| 215 | + auditor.Record(audit.UserUpdate{ |
| 216 | + UserID: arg.ID, |
| 217 | + Old: oldUser, |
| 218 | + New: newUser, |
| 219 | + }) |
| 220 | + } |
| 221 | + |
| 222 | +return newUser,nil |
| 223 | +} |
| 224 | +``` |
| 225 | + |
| 226 | +##Debugging Database Issues |
| 227 | + |
| 228 | +###Common Debug Commands |
| 229 | + |
| 230 | +```bash |
| 231 | +# Check database connection |
| 232 | +make test-postgres |
| 233 | + |
| 234 | +# Run specific database tests |
| 235 | +gotest ./coderd/database/... -run TestSpecificFunction |
| 236 | + |
| 237 | +# Check query generation |
| 238 | +make gen |
| 239 | + |
| 240 | +# Verify audit table |
| 241 | +make lint |
| 242 | +``` |
| 243 | + |
| 244 | +###Debug Techniques |
| 245 | + |
| 246 | +1.**Enable query logging**: Set appropriate log levels |
| 247 | +2.**Use database tools**: pgAdmin, psql for direct inspection |
| 248 | +3.**Check constraints**: UNIQUE, FOREIGN KEY violations |
| 249 | +4.**Analyze performance**: Use EXPLAIN ANALYZE for slow queries |
| 250 | + |
| 251 | +###Troubleshooting Checklist |
| 252 | + |
| 253 | +-[ ] Migration files exist (both up and down) |
| 254 | +-[ ]`make gen` run after query changes |
| 255 | +-[ ] Audit table updated for new fields |
| 256 | +-[ ] In-memory database implementations updated |
| 257 | +-[ ] Nullable fields use`sql.Null*` types |
| 258 | +-[ ] Authorization context appropriate for endpoint type |