- Notifications
You must be signed in to change notification settings - Fork1.1k
feat: persist prebuild definitions on template import#16951
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
Merged
Uh oh!
There was an error while loading.Please reload this page.
Merged
Changes from1 commit
Commits
Show all changes
88 commits Select commitHold shift + click to select a range
5ecc277 add prebuilds system user database changes and associated changes
SasSwartbc5f4f4 optionally prevent system users from counting to user count
dannykopping48c5372 appease the linter
dannykoppingb16d126 add unit test for system user behaviour
dannykopping2c25542 reverting RBAC changes; not relevant here
dannykopping8e491d8 removing unnecessary changes
dannykopping514fdbf exclude system user db tests from non-linux OSs
dannykopping390a1fd feat: add migrations and queries to support prebuilds
SasSwart07e9613 Merge remote-tracking branch 'origin/main' into prebuilds-db
SasSwarta07c2b2 feat: persist prebuild definitions on template import
SasSwart300e80f add prebuilds system user database changes and associated changes
SasSwartb788237 optionally prevent system users from counting to user count
dannykopping8122595 appease the linter
dannykoppingbfb7c28 add unit test for system user behaviour
dannykopping6639167 reverting RBAC changes; not relevant here
dannykopping769ae1d removing unnecessary changes
dannykoppinge7e9c27 exclude system user db tests from non-linux OSs
dannykopping3936047 Rename prebuild system user reference
SasSwart8bdcafb ensure that users.IsSystem is not nullable
SasSwart412d198 feat: add migrations and queries to support prebuilds
SasSwart51773ec Simplify workspace_latest_build view
dannykopping23773c2 Revert test change
dannykoppingbc3ff44 make gen
dannykoppingbaa3076 refactor: add comments to SQL queries
evgeniy-scherbinaed14fb3 test: added get-presets-backoff test
evgeniy-scherbina3cc74fb refactor: add comment to SQL query
evgeniy-scherbinafc32154 refactor: add comments + improve tests
evgeniy-scherbinad7b4ec4 fix: bug in SQL
evgeniy-scherbinae8b53f7 test: minor changes to the test
evgeniy-scherbina9df6554 refactor: remove job_status from SQL query
evgeniy-scherbinaccc309e refactor: embed preset_prebuilds table into presets table
evgeniy-scherbinaee1f16a refactor: rename sql table
evgeniy-scherbinad040ddd refactor: remove unnecessary JOIN
evgeniy-scherbina83a6722 refactor: remove unnecessary JOIN
evgeniy-scherbinacd70710 refactor: use INNER JOIN for consistency
evgeniy-scherbina0f3bda0 Merge remote-tracking branch 'origin/prebuilds-db' into jjs/insert-pr…
SasSwarta7c7cd2 make lint
SasSwart97cc4ff refactor: simplify GetPresetsBackoff SQL Query
evgeniy-scherbina4d59039 Revert "refactor: simplify GetPresetsBackoff SQL Query"
evgeniy-scherbina205d6af refactor: improve GetPresetsBackoff query
evgeniy-scherbinae489e1b Merge remote-tracking branch 'origin/main' into prebuilds-db
evgeniy-scherbina1b29686 Merge remote-tracking branch 'origin/main' into prebuilds-db
evgeniy-scherbina20470e4 fix: bump migration numbers
evgeniy-scherbina6fc1889 Merge remote-tracking branch 'origin/main' into jjs/insert-prebuilds
SasSwart7b9c8ce test: remove deprecated test
evgeniy-scherbinae189a0b fix: fix linter
evgeniy-scherbina692c0e5 fix: fix 000310_prebuilds.down migration
evgeniy-scherbinaf747db0 fix: fix fixture migration
evgeniy-scherbina3166a42 fix: fix get-presets-backoff test
evgeniy-scherbinaaa6b490 fix: fix linter
evgeniy-scherbinabc4e7d2 fix: fix linter
evgeniy-scherbinaf167b92 correctly select for the latest built with a preset in latest_prebuil…
SasSwart8fd34ab Merge remote-tracking branch 'origin/main' into prebuilds-db
SasSwart7a8ec49 Properly label and filter metrics for prebuilds
SasSwarta64d661 test: fix db tests
evgeniy-scherbina865998b Merge branch 'prebuilds-db' into jjs/insert-prebuilds
SasSwartc787cd2 test: added tests for workspaces with multiple agents
evgeniy-scherbinabd38603 refactor: avoid code duplication
evgeniy-scherbina097f9c3 clarify query clause
SasSwart4cfdd6f tidy up dbauthz_test.go
SasSwart4a34d52 refactor: remove * usage from prebuilds.sql queries
evgeniy-scherbina8d9cd45 refactor: remove * usage from prebuilds views
evgeniy-scherbinaf870d7e refactor: join wlb with pj
evgeniy-scherbina18ad931 refactor: Rename SQL query
evgeniy-scherbina4667171 Added comments for SQL query
evgeniy-scherbinaa26c094 refactor: fix down migration
evgeniy-scherbinabf4ab53 make lint
SasSwarta84b1bb Merge remote-tracking branch 'origin/main' into jjs/insert-prebuilds
SasSwart6ed4121 Merge remote-tracking branch 'origin/main' into prebuilds-db
SasSwarte8b1502 Merge branch 'prebuilds-db' into jjs/insert-prebuilds
SasSwart2312f41 renumber migrations
SasSwart4540a55 add tests for prebuilds in the provisionerdserver
SasSwart5c41ba9 Merge branch 'prebuilds-db' into jjs/insert-prebuilds
SasSwartd09b757 fix indent
SasSwart61e86f6 make more use of dbgen
SasSwart1419df0 update dbmemt
SasSwart6589221 Add tests
SasSwartb15b97a Merge origin/main
SasSwartdd656a7 Merge remote-tracking branch 'origin/main' into jjs/insert-prebuilds
SasSwart508b244 fix tests
SasSwarte1f585d remove duplicate migrations fixture
SasSwart7d4f1b9 clean up go.mod slightly
SasSwart43f82b9 update dependency on terraform-provider-coder
SasSwart5d8de71 simplify query
SasSwarta87933a Correct the prebuilds type
SasSwart798cfa1 fix prebuild decoding
SasSwartdc87f45 make fmt
SasSwart7c17fcd update our dependency on terraform-provider-coder
SasSwartFile filter
Filter by extension
Conversations
Failed to load comments.
Loading
Uh oh!
There was an error while loading.Please reload this page.
Jump to
Jump to file
Failed to load files.
Loading
Uh oh!
There was an error while loading.Please reload this page.
Diff view
Diff view
Merge origin/main
- Loading branch information
Uh oh!
There was an error while loading.Please reload this page.
commitb15b97a922653d8ad17deeae5e0a48cc8c8a8f14
There are no files selected for viewing
122 changes: 122 additions & 0 deletions.cursorrules
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,122 @@ | ||
| # Cursor Rules | ||
| This project is called "Coder" - an application for managing remote development environments. | ||
| Coder provides a platform for creating, managing, and using remote development environments (also known as Cloud Development Environments or CDEs). It leverages Terraform to define and provision these environments, which are referred to as "workspaces" within the project. The system is designed to be extensible, secure, and provide developers with a seamless remote development experience. | ||
| # Core Architecture | ||
| The heart of Coder is a control plane that orchestrates the creation and management of workspaces. This control plane interacts with separate Provisioner processes over gRPC to handle workspace builds. The Provisioners consume workspace definitions and use Terraform to create the actual infrastructure. | ||
| The CLI package serves dual purposes - it can be used to launch the control plane itself and also provides client functionality for users to interact with an existing control plane instance. All user-facing frontend code is developed in TypeScript using React and lives in the `site/` directory. | ||
| The database layer uses PostgreSQL with SQLC for generating type-safe database code. Database migrations are carefully managed to ensure both forward and backward compatibility through paired `.up.sql` and `.down.sql` files. | ||
| # API Design | ||
| Coder's API architecture combines REST and gRPC approaches. The REST API is defined in `coderd/coderd.go` and uses Chi for HTTP routing. This provides the primary interface for the frontend and external integrations. | ||
| Internal communication with Provisioners occurs over gRPC, with service definitions maintained in `.proto` files. This separation allows for efficient binary communication with the components responsible for infrastructure management while providing a standard REST interface for human-facing applications. | ||
| # Network Architecture | ||
| Coder implements a secure networking layer based on Tailscale's Wireguard implementation. The `tailnet` package provides connectivity between workspace agents and clients through DERP (Designated Encrypted Relay for Packets) servers when direct connections aren't possible. This creates a secure overlay network allowing access to workspaces regardless of network topology, firewalls, or NAT configurations. | ||
| ## Tailnet and DERP System | ||
| The networking system has three key components: | ||
| 1. **Tailnet**: An overlay network implemented in the `tailnet` package that provides secure, end-to-end encrypted connections between clients, the Coder server, and workspace agents. | ||
| 2. **DERP Servers**: These relay traffic when direct connections aren't possible. Coder provides several options: | ||
| - A built-in DERP server that runs on the Coder control plane | ||
| - Integration with Tailscale's global DERP infrastructure | ||
| - Support for custom DERP servers for lower latency or offline deployments | ||
| 3. **Direct Connections**: When possible, the system establishes peer-to-peer connections between clients and workspaces using STUN for NAT traversal. This requires both endpoints to send UDP traffic on ephemeral ports. | ||
| ## Workspace Proxies | ||
| Workspace proxies (in the Enterprise edition) provide regional relay points for browser-based connections, reducing latency for geo-distributed teams. Key characteristics: | ||
| - Deployed as independent servers that authenticate with the Coder control plane | ||
| - Relay connections for SSH, workspace apps, port forwarding, and web terminals | ||
| - Do not make direct database connections | ||
| - Managed through the `coder wsproxy` commands | ||
| - Implemented primarily in the `enterprise/wsproxy/` package | ||
| # Agent System | ||
| The workspace agent runs within each provisioned workspace and provides core functionality including: | ||
| - SSH access to workspaces via the `agentssh` package | ||
| - Port forwarding | ||
| - Terminal connectivity via the `pty` package for pseudo-terminal support | ||
| - Application serving | ||
| - Healthcheck monitoring | ||
| - Resource usage reporting | ||
| Agents communicate with the control plane using the tailnet system and authenticate using secure tokens. | ||
| # Workspace Applications | ||
| Workspace applications (or "apps") provide browser-based access to services running within workspaces. The system supports: | ||
| - HTTP(S) and WebSocket connections | ||
| - Path-based or subdomain-based access URLs | ||
| - Health checks to monitor application availability | ||
| - Different sharing levels (owner-only, authenticated users, or public) | ||
| - Custom icons and display settings | ||
| The implementation is primarily in the `coderd/workspaceapps/` directory with components for URL generation, proxying connections, and managing application state. | ||
| # Implementation Details | ||
| The project structure separates frontend and backend concerns. React components and pages are organized in the `site/src/` directory, with Jest used for testing. The backend is primarily written in Go, with a strong emphasis on error handling patterns and test coverage. | ||
| Database interactions are carefully managed through migrations in `coderd/database/migrations/` and queries in `coderd/database/queries/`. All new queries require proper database authorization (dbauthz) implementation to ensure that only users with appropriate permissions can access specific resources. | ||
| # Authorization System | ||
| The database authorization (dbauthz) system enforces fine-grained access control across all database operations. It uses role-based access control (RBAC) to validate user permissions before executing database operations. The `dbauthz` package wraps the database store and performs authorization checks before returning data. All database operations must pass through this layer to ensure security. | ||
| # Testing Framework | ||
| The codebase has a comprehensive testing approach with several key components: | ||
| 1. **Parallel Testing**: All tests must use `t.Parallel()` to run concurrently, which improves test suite performance and helps identify race conditions. | ||
| 2. **coderdtest Package**: This package in `coderd/coderdtest/` provides utilities for creating test instances of the Coder server, setting up test users and workspaces, and mocking external components. | ||
| 3. **Integration Tests**: Tests often span multiple components to verify system behavior, such as template creation, workspace provisioning, and agent connectivity. | ||
| 4. **Enterprise Testing**: Enterprise features have dedicated test utilities in the `coderdenttest` package. | ||
| # Open Source and Enterprise Components | ||
| The repository contains both open source and enterprise components: | ||
| - Enterprise code lives primarily in the `enterprise/` directory | ||
| - Enterprise features focus on governance, scalability (high availability), and advanced deployment options like workspace proxies | ||
| - The boundary between open source and enterprise is managed through a licensing system | ||
| - The same core codebase supports both editions, with enterprise features conditionally enabled | ||
| # Development Philosophy | ||
| Coder emphasizes clear error handling, with specific patterns required: | ||
| - Concise error messages that avoid phrases like "failed to" | ||
| - Wrapping errors with `%w` to maintain error chains | ||
| - Using sentinel errors with the "err" prefix (e.g., `errNotFound`) | ||
| All tests should run in parallel using `t.Parallel()` to ensure efficient testing and expose potential race conditions. The codebase is rigorously linted with golangci-lint to maintain consistent code quality. | ||
| Git contributions follow a standard format with commit messages structured as `type: <message>`, where type is one of `feat`, `fix`, or `chore`. | ||
| # Development Workflow | ||
| Development can be initiated using `scripts/develop.sh` to start the application after making changes. Database schema updates should be performed through the migration system using `create_migration.sh <name>` to generate migration files, with each `.up.sql` migration paired with a corresponding `.down.sql` that properly reverts all changes. | ||
| If the development database gets into a bad state, it can be completely reset by removing the PostgreSQL data directory with `rm -rf .coderv2/postgres`. This will destroy all data in the development database, requiring you to recreate any test users, templates, or workspaces after restarting the application. | ||
| Code generation for the database layer uses `coderd/database/generate.sh`, and developers should refer to `sqlc.yaml` for the appropriate style and patterns to follow when creating new queries or tables. | ||
| The focus should always be on maintaining security through proper database authorization, clean error handling, and comprehensive test coverage to ensure the platform remains robust and reliable. |
1 change: 1 addition & 0 deletions.github/.linkspector.yml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
11 changes: 7 additions & 4 deletions.github/workflows/ci.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
3 changes: 1 addition & 2 deletions.golangci.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
5 changes: 4 additions & 1 deletion.vscode/settings.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
12 changes: 12 additions & 0 deletionsagent/ls.go
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
11 changes: 6 additions & 5 deletionsagent/ls_internal_test.go
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
17 changes: 17 additions & 0 deletionsarchive/fs/tar.go
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,17 @@ | ||
| package archivefs | ||
| import ( | ||
| "archive/tar" | ||
| "io" | ||
| "io/fs" | ||
| "github.com/spf13/afero" | ||
| "github.com/spf13/afero/tarfs" | ||
| ) | ||
| func FromTarReader(r io.Reader) fs.FS { | ||
| tr := tar.NewReader(r) | ||
| tfs := tarfs.New(tr) | ||
| rofs := afero.NewReadOnlyFs(tfs) | ||
| return afero.NewIOFS(rofs) | ||
| } |
8 changes: 3 additions & 5 deletionscli/clitest/golden.go
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
27 changes: 14 additions & 13 deletionscli/create_test.go
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
1 change: 1 addition & 0 deletionscli/exp.go
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
Oops, something went wrong.
Uh oh!
There was an error while loading.Please reload this page.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.