Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

feat: use proto streams to increase maximum module files payload#18268

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Draft
Emyrk wants to merge34 commits intomain
base:main
Choose a base branch
Loading
fromstevenmasley/4mb

Conversation

Emyrk
Copy link
Member

@EmyrkEmyrk commentedJun 6, 2025
edited
Loading

Increase maximum module files payload using protobuf streams

Problem

Coder has a4MB protobuf message size limit that prevents templates with large Terraform modules from being processed. When dynamic parameters require module files from~/.terraform/modules that exceed this limit, template creation fails.

Solution

This PR implements protobuf streaming to handle large module files by:

  1. Streaming large payloads: When module files exceed the 4MB limit, they're streamed in chunks using a new UploadFile RPC method
  2. Database storage: Streamed files are stored in the database and referenced by hash for deduplication
  3. Backward compatibility: Small module files continue using the existing direct payload method

Key Changes

  • New protobuf streams: Added UploadFile RPC and DataUpload/ChunkPiece message types
  • Smart fallback: Automatically detects oversized payloads and switches to streaming
  • File deduplication: Module files are deduplicated per template version using content hashes
  • Testing: Added test coverage for the streaming upload functionality.

Architecture

  • Coderd↔️ Provisionerd: Uses new streaming RPC when payload exceeds limits.

    // UploadFile streams files to be inserted into the database.
    // The file upload_type should be used to determine how to handle the file.
    rpcUploadFile(streamUploadFileRequest)returns (Empty);

  • Provisionerd↔️ Provisioner: Extends existing stream with new message types for in-memory communication

This change enables templates with arbitrarily large Terraform modules while maintaining performance through deduplication and preserving the existing fast path for smaller payloads.

TODO:

  • Bump protobuf versions
  • Implement backwards compatible?
    • Do not upload large files to old coderds
  • Verify forward compatible
  • Add e2e test with a large module

Things that I wish I could change

  • We have 2 proto files, and they have overlapping types. IdeallyDataUpload andChunk would be defined in 1 place and have their types reusable in both contexts 😢.

@EmyrkEmyrk changed the titleStevenmasley/4mbfeat: increase maximum module files payload using proto streamsJun 6, 2025
@EmyrkEmyrk changed the titlefeat: increase maximum module files payload using proto streamsfeat: use proto streams to increase maximum module files payloadJun 9, 2025
Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment
Reviewers

@spikecurtisspikecurtisAwaiting requested review from spikecurtisspikecurtis will be requested when the pull request is marked ready for reviewspikecurtis is a code owner

@johnstcnjohnstcnAwaiting requested review from johnstcnjohnstcn will be requested when the pull request is marked ready for reviewjohnstcn is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

@EmyrkEmyrk

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

1 participant
@Emyrk

[8]ページ先頭

©2009-2025 Movatter.jp