This issue manifests when users have multiple templates which rely on the same files, for example see:#17442
In thefiles table we have aconstraint to enforce that there can only be one entry perhash, created_by combo. When runningterraform apply to update templates, this can mean that the file upload can fail forsome of the templates as they hit a race condition all trying to insert at the same time.
The fix here is to detect presence of the conflict error and run anotherGetFileByHashAndCreator. This should happen infrequently enough to not cause significant extra load on the DB. If in the future we notice that it does, we could change the underlying SQL for file insertion to run a CAS like call via theON CONFLICT syntax.
Note that the test added here will fail without the change from the first commit. I also tested this manually viadeploy.sh in my own workspace.
Apply diff example without code changes
terraform apply --parallelism=10var.coder_url Coder deployment URL Enter a value: ***coderd_template.test_templates["template-a"]: Refreshing state... [id=5d457fb1-4697-4567-97e9-856627f12b1a]coderd_template.test_templates["template-b"]: Refreshing state... [id=898f2d5f-f22b-4741-abbb-ac5c3e36eb5f]coderd_template.test_templates["template-c"]: Refreshing state... [id=acd53214-8bbd-402d-ab71-83fc9b5c0822]Terraform used the selected providers to generate the following execution plan. Resource actions areindicated with the following symbols: ~ update in-placeTerraform will perform the following actions: # coderd_template.test_templates["template-a"] will be updated in-place ~ resource "coderd_template" "test_templates" { ~ display_name = "template-a" -> (known after apply) id = "5d457fb1-4697-4567-97e9-856627f12b1a" ~ max_port_share_level = "public" -> (known after apply) name = "template-a" ~ organization_id = "8ff5f083-aec6-4405-8df7-a16d9362c8f4" -> (known after apply) ~ versions = [ ~ { ~ directory_hash = "0de194f2590374f4d9f579712ff6eed00a71bf60a37c5df40d03a9640ecc5a00" -> "29ff9a13dd5f3fbf1c2c5cf1a33ac650492c3d266f77e65bb569f9bfa711ac3d" ~ id = "7161d56a-1de3-43f1-afe0-f559e70712aa" -> (known after apply) ~ name = "1.0.5" -> "1.0.7" # (4 unchanged attributes hidden) }, ] # (14 unchanged attributes hidden) } # coderd_template.test_templates["template-b"] will be updated in-place ~ resource "coderd_template" "test_templates" { ~ display_name = "template-b" -> (known after apply) id = "898f2d5f-f22b-4741-abbb-ac5c3e36eb5f" ~ max_port_share_level = "public" -> (known after apply) name = "template-b" ~ organization_id = "8ff5f083-aec6-4405-8df7-a16d9362c8f4" -> (known after apply) ~ versions = [ ~ { ~ directory_hash = "22e2692a6d58da8010c8e48d2e4a1965cad34aac0a10a718cdb935d0ea625c14" -> "29ff9a13dd5f3fbf1c2c5cf1a33ac650492c3d266f77e65bb569f9bfa711ac3d" ~ id = "c431e295-1026-4ca4-a467-0d875eb89f4f" -> (known after apply) ~ name = "1.0.6" -> "1.0.7" # (4 unchanged attributes hidden) }, ] # (14 unchanged attributes hidden) } # coderd_template.test_templates["template-c"] will be updated in-place ~ resource "coderd_template" "test_templates" { ~ display_name = "template-c" -> (known after apply) id = "acd53214-8bbd-402d-ab71-83fc9b5c0822" ~ max_port_share_level = "public" -> (known after apply) name = "template-c" ~ organization_id = "8ff5f083-aec6-4405-8df7-a16d9362c8f4" -> (known after apply) ~ versions = [ ~ { ~ directory_hash = "22e2692a6d58da8010c8e48d2e4a1965cad34aac0a10a718cdb935d0ea625c14" -> "29ff9a13dd5f3fbf1c2c5cf1a33ac650492c3d266f77e65bb569f9bfa711ac3d" ~ id = "304215dc-bbde-4d41-8a5a-c9dd0926a96b" -> (known after apply) ~ name = "1.0.6" -> "1.0.7" # (4 unchanged attributes hidden) }, ] # (14 unchanged attributes hidden) }Plan: 0 to add, 3 to change, 0 to destroy.
Error from apply
Plan: 0 to add, 3 to change, 0 to destroy.Do you want to perform these actions? Terraform will perform the actions described above. Only 'yes' will be accepted to approve. Enter a value: yescoderd_template.test_templates["template-b"]: Modifying... [id=898f2d5f-f22b-4741-abbb-ac5c3e36eb5f]coderd_template.test_templates["template-a"]: Modifying... [id=5d457fb1-4697-4567-97e9-856627f12b1a]coderd_template.test_templates["template-c"]: Modifying... [id=acd53214-8bbd-402d-ab71-83fc9b5c0822]coderd_template.test_templates["template-a"]: Modifications complete after 4s [id=5d457fb1-4697-4567-97e9-856627f12b1a]coderd_template.test_templates["template-b"]: Modifications complete after 6s [id=898f2d5f-f22b-4741-abbb-ac5c3e36eb5f]╷│ Error: Provisioner Error│ │ with coderd_template.test_templates["template-c"],│ on main.tf line 19, in resource "coderd_template" "test_templates":│ 19: resource "coderd_template" "test_templates" {│ │ failed to upload directory: POST ***/api/v2/files: unexpected│ status code 500: Internal error saving file.│ Error: pq: duplicate key value violates unique constraint "files_hash_created_by_key"│
Running the apply again succeeds
Terraform used the selected providers to generate the following execution plan. Resource actions areindicated with the following symbols: ~ update in-placeTerraform will perform the following actions: # coderd_template.test_templates["template-c"] will be updated in-place ~ resource "coderd_template" "test_templates" { ~ display_name = "template-c" -> (known after apply) id = "acd53214-8bbd-402d-ab71-83fc9b5c0822" ~ max_port_share_level = "public" -> (known after apply) name = "template-c" ~ organization_id = "8ff5f083-aec6-4405-8df7-a16d9362c8f4" -> (known after apply) ~ versions = [ ~ { ~ directory_hash = "22e2692a6d58da8010c8e48d2e4a1965cad34aac0a10a718cdb935d0ea625c14" -> "29ff9a13dd5f3fbf1c2c5cf1a33ac650492c3d266f77e65bb569f9bfa711ac3d" ~ id = "304215dc-bbde-4d41-8a5a-c9dd0926a96b" -> (known after apply) ~ name = "1.0.6" -> "1.0.7" # (4 unchanged attributes hidden) }, ] # (14 unchanged attributes hidden) }Plan: 0 to add, 1 to change, 0 to destroy.Do you want to perform these actions? Terraform will perform the actions described above. Only 'yes' will be accepted to approve. Enter a value: yescoderd_template.test_templates["template-c"]: Modifying... [id=acd53214-8bbd-402d-ab71-83fc9b5c0822]coderd_template.test_templates["template-c"]: Modifications complete after 3s [id=acd53214-8bbd-402d-ab71-83fc9b5c0822]Apply complete! Resources: 0 added, 1 changed, 0 destroyed.
Apply diff example with code changes
coderd_template.test_templates["template-c"]: Refreshing state... [id=acd53214-8bbd-402d-ab71-83fc9b5c0822]coderd_template.test_templates["template-b"]: Refreshing state... [id=898f2d5f-f22b-4741-abbb-ac5c3e36eb5f]coderd_template.test_templates["template-a"]: Refreshing state... [id=5d457fb1-4697-4567-97e9-856627f12b1a]Terraform used the selected providers to generate the following execution plan. Resource actions areindicated with the following symbols: ~ update in-placeTerraform will perform the following actions: # coderd_template.test_templates["template-a"] will be updated in-place ~ resource "coderd_template" "test_templates" { ~ display_name = "template-a" -> (known after apply) id = "5d457fb1-4697-4567-97e9-856627f12b1a" ~ max_port_share_level = "public" -> (known after apply) name = "template-a" ~ organization_id = "8ff5f083-aec6-4405-8df7-a16d9362c8f4" -> (known after apply) ~ versions = [ ~ { ~ directory_hash = "29ff9a13dd5f3fbf1c2c5cf1a33ac650492c3d266f77e65bb569f9bfa711ac3d" -> "d4a509e4af5f96f5b3970098233d742105f75c997aafa8a6da2ee4953c864a54" ~ id = "e130f32f-2c3e-4332-818f-fd6de980128f" -> (known after apply) ~ name = "1.0.7" -> "1.0.8" # (4 unchanged attributes hidden) }, ] # (14 unchanged attributes hidden) } # coderd_template.test_templates["template-b"] will be updated in-place ~ resource "coderd_template" "test_templates" { ~ display_name = "template-b" -> (known after apply) id = "898f2d5f-f22b-4741-abbb-ac5c3e36eb5f" ~ max_port_share_level = "public" -> (known after apply) name = "template-b" ~ organization_id = "8ff5f083-aec6-4405-8df7-a16d9362c8f4" -> (known after apply) ~ versions = [ ~ { ~ directory_hash = "29ff9a13dd5f3fbf1c2c5cf1a33ac650492c3d266f77e65bb569f9bfa711ac3d" -> "d4a509e4af5f96f5b3970098233d742105f75c997aafa8a6da2ee4953c864a54" ~ id = "67207027-a3a0-4679-834e-1f8ac69ad31e" -> (known after apply) ~ name = "1.0.7" -> "1.0.8" # (4 unchanged attributes hidden) }, ] # (14 unchanged attributes hidden) } # coderd_template.test_templates["template-c"] will be updated in-place ~ resource "coderd_template" "test_templates" { ~ display_name = "template-c" -> (known after apply) id = "acd53214-8bbd-402d-ab71-83fc9b5c0822" ~ max_port_share_level = "public" -> (known after apply) name = "template-c" ~ organization_id = "8ff5f083-aec6-4405-8df7-a16d9362c8f4" -> (known after apply) ~ versions = [ ~ { ~ directory_hash = "29ff9a13dd5f3fbf1c2c5cf1a33ac650492c3d266f77e65bb569f9bfa711ac3d" -> "d4a509e4af5f96f5b3970098233d742105f75c997aafa8a6da2ee4953c864a54" ~ id = "8771b486-16a2-47d8-bcc0-08d7aeae5e15" -> (known after apply) ~ name = "1.0.7" -> "1.0.8" # (4 unchanged attributes hidden) }, ] # (14 unchanged attributes hidden) }Plan: 0 to add, 3 to change, 0 to destroy.Do you want to perform these actions? Terraform will perform the actions described above. Only 'yes' will be accepted to approve.
apply works on the first try
Plan: 0 to add, 3 to change, 0 to destroy.Do you want to perform these actions? Terraform will perform the actions described above. Only 'yes' will be accepted to approve. Enter a value: yescoderd_template.test_templates["template-a"]: Modifying... [id=5d457fb1-4697-4567-97e9-856627f12b1a]coderd_template.test_templates["template-c"]: Modifying... [id=acd53214-8bbd-402d-ab71-83fc9b5c0822]coderd_template.test_templates["template-b"]: Modifying... [id=898f2d5f-f22b-4741-abbb-ac5c3e36eb5f]coderd_template.test_templates["template-a"]: Modifications complete after 2s [id=5d457fb1-4697-4567-97e9-856627f12b1a]coderd_template.test_templates["template-c"]: Modifications complete after 5s [id=acd53214-8bbd-402d-ab71-83fc9b5c0822]coderd_template.test_templates["template-b"]: Modifications complete after 8s [id=898f2d5f-f22b-4741-abbb-ac5c3e36eb5f]
The template files are as follows:
Top level main.tf
terraform { required_providers { coderd = { source = "coder/coderd" } }}provider "coderd" { url = var.coder_url}variable "coder_url" { description = "Coder deployment URL" type = string}# Create 3 templates that use identical filesresource "coderd_template" "test_templates" { for_each = { "template-a" = "Template A" "template-b" = "Template B" "template-c" = "Template C" } name = each.key description = each.value versions = [{ name = "1.0.0" directory = "./template" # Same directory = same files = same hash active = true tf_vars = [ { name = "template_name" value = each.key # Only difference between templates } ] }]}
template/main.tf
terraform { required_providers { coder = { source = "coder/coder" } }}variable "template_name" { description = "Name of the template" type = string default = "test"}data "coder_workspace" "me" {}resource "coder_agent" "main" { arch = "amd64" os = "linux"}resource "null_resource" "example" { provisioner "local-exec" { command = "echo 'Template name is: ${var.template_name}'" }}
template/README.md
# Workspace TemplateThis is a test template for reproducing the file upload race condition.## Features- Basic workspace setup- Agent configuration- Multiple files to increase upload size
To test, simply modify the template files (adding a comment in each is all you need, plus updating the version #) and thenterraform apply with the--parallelism flag.
This issue manifests when users have multiple templates which rely on the same files, for example see:#17442
In the
filestable we have aconstraint to enforce that there can only be one entry perhash, created_bycombo. When runningterraform applyto update templates, this can mean that the file upload can fail forsome of the templates as they hit a race condition all trying to insert at the same time.The fix here is to detect presence of the conflict error and run another
GetFileByHashAndCreator. This should happen infrequently enough to not cause significant extra load on the DB. If in the future we notice that it does, we could change the underlying SQL for file insertion to run a CAS like call via theON CONFLICTsyntax.Note that the test added here will fail without the change from the first commit. I also tested this manually via
deploy.shin my own workspace.Apply diff example without code changes
Error from apply
Running the apply again succeeds
Apply diff example with code changes
apply works on the first try
The template files are as follows:
Top level main.tf
template/main.tf
template/README.md
To test, simply modify the template files (adding a comment in each is all you need, plus updating the version #) and then
terraform applywith the--parallelismflag.