- Notifications
You must be signed in to change notification settings - Fork905
fix: move pubsub publishing out of database transactions to avoid conn exhaustion#17648
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
Uh oh!
There was an error while loading.Please reload this page.
Conversation
…n starvationSigned-off-by: Danny Kopping <dannykopping@gmail.com>
6efe39f
toba2f90a
CompareSigned-off-by: Danny Kopping <dannykopping@gmail.com>
Signed-off-by: Danny Kopping <dannykopping@gmail.com>
clock: clock, | ||
registerer: registerer, | ||
done: make(chan struct{}, 1), | ||
provisionNotifyCh: make(chan database.ProvisionerJob, 10), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Thumb-suck; I want to protect against temporary network/database blips but also don't want to accumulate too many messages.
Uh oh!
There was an error while loading.Please reload this page.
…ot affect behaviourSigned-off-by: Danny Kopping <dannykopping@gmail.com>
Signed-off-by: Danny Kopping <dannykopping@gmail.com>
func (*brokenPublisher) Publish(event string, _ []byte) error { | ||
// I'm explicitly _not_ checking for EventJobPosted (coderd/database/provisionerjobs/provisionerjobs.go) since that | ||
// requires too much knowledge of the underlying implementation. | ||
return xerrors.Errorf("refusing to publish %q", event) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Should we fail slowly instead?
e.g.<-time.After(testutil.IntervalFast)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Addressed in9ddad0b
evgeniy-scherbina commentedMay 2, 2025 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
@dannykopping is it possible to use another approach, and allow calling func (p*PGPubsub)PublishWithinTx(eventstring,message []byte,tx database.Store)error {tx.ExecContext(context.Background(),`select pg_notify(`+pq.QuoteLiteral(event)+`, $1)`,message)} Implementations which doesn't support txs may implement it: func (p*InMemPubSub)PublishWithinTx(eventstring,message []byte,tx database.Store)error {_=tx// ignoredreturnp.PublishTx(event,message)} But it will require some changes in Because we impose restrictions on ourselves, every time we need to use Publish within tx - we have to come up with workaround like in this PR. |
Uh oh!
There was an error while loading.Please reload this page.
Signed-off-by: Danny Kopping <dannykopping@gmail.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
LGTM
a646478
intomainUh oh!
There was an error while loading.Please reload this page.
@evgeniy-scherbina I'll defer to@spikecurtis about your above question. However, I would imagine that we wouldn't want to overwhelm the API with knowledge of in- or out-of-transaction considerations. There are also some problems around 2PC ("A transaction that has executed NOTIFY cannot be prepared for two-phase commit."), and in general it just makes pubsub a bit less deterministic. |
Uh oh!
There was an error while loading.Please reload this page.
Database transactions hold onto connections, and
pubsub.Publish
tries to acquire a connection of its own. If the latter is called within a transaction, this can lead to connection exhaustion.I plan two follow-ups to this PR:
https://github.com/coder/coder/blob/main/cli/server.go#L2360-L2376
We will then be able to write tests showing how connection exhaustion occurs.
pubsub.Publish
from being called within a transaction.