@kylecarbs and I were debugging a gnarly postgres issue over the weekend, and unfortunately it looks like it is still coming up occassionally:https://github.com/coder/coder/runs/5014420662?check_suite_focus=true#step:8:35 - so thought this might be a good testing Monday task.

Intermittently, the test would fail with something like a401 - invalid e-mail, or a409 - initial user already created. This was quite surprising, because the tests are designed to spin up their own, isolated database.

We tried a few things to debug this...

Attempt 1: Log out the generated port numbers when running the docker image.

Based on the errors, it seemed like one test must be connecting to another test's database - that would explain why we'd get these conflicts! However, logging out the port number that came from docker always gave a unique number... and we couldn't find evidence of one database connecting to another.

Attempt 2: Store the database in unique, temporary folder.

@kylecarbs and I found that the there was avolume for the postgres data... so@kylecarbs implemented mounting the volume to a unique, per-test temporary folder in#89

It sounded really promising... but unfortunately we hit the issue again!

Attempt 3... this PR

After we hit the failure again, we noticed in thedocker ps logs something quite strange:

When the docker image is run - it creates two port bindings, an IPv4 and an IPv6 one. Theseshould be the same - but surprisingly, they can sometimes be different. It isn't deterministic, and seems to be more common when there are multiple containers running. Importantly,they can overlap as in the above image.

Turns out, it seems this is a docker bug:moby/moby#42442 - which may be fixed in newer versions.

To work around this bug, we have to manipulate the port bindings (like you would with-p) at the command line. We can do this withdocker/dockertest, but it means we have to get a free port ahead of time to know which port to map.

With that fix in - thedocker ps is a little more sane:

...and hopefully means we can safely run the containers in parallel again.