Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

fix: move pubsub publishing out of database transactions to avoid conn exhaustion#17648

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Merged
dannykopping merged 7 commits intomainfromdk/notify-outside-tx
May 5, 2025

Conversation

dannykopping
Copy link
Contributor

@dannykoppingdannykopping commentedMay 1, 2025
edited
Loading

Database transactions hold onto connections, andpubsub.Publish tries to acquire a connection of its own. If the latter is called within a transaction, this can lead to connection exhaustion.

I plan two follow-ups to this PR:

  1. Make connection counts tuneable

https://github.com/coder/coder/blob/main/cli/server.go#L2360-L2376

We will then be able to write tests showing how connection exhaustion occurs.

  1. Write a linter/ruleguard to preventpubsub.Publish from being called within a transaction.

…n starvationSigned-off-by: Danny Kopping <dannykopping@gmail.com>
Signed-off-by: Danny Kopping <dannykopping@gmail.com>
Signed-off-by: Danny Kopping <dannykopping@gmail.com>
@dannykoppingdannykopping marked this pull request as ready for reviewMay 2, 2025 09:56
clock: clock,
registerer: registerer,
done: make(chan struct{}, 1),
provisionNotifyCh: make(chan database.ProvisionerJob, 10),
Copy link
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Thumb-suck; I want to protect against temporary network/database blips but also don't want to accumulate too many messages.

johnstcn reacted with thumbs up emoji
…ot affect behaviourSigned-off-by: Danny Kopping <dannykopping@gmail.com>
Signed-off-by: Danny Kopping <dannykopping@gmail.com>
func (*brokenPublisher) Publish(event string, _ []byte) error {
// I'm explicitly _not_ checking for EventJobPosted (coderd/database/provisionerjobs/provisionerjobs.go) since that
// requires too much knowledge of the underlying implementation.
return xerrors.Errorf("refusing to publish %q", event)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Should we fail slowly instead?

e.g.<-time.After(testutil.IntervalFast)

Copy link
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Addressed in9ddad0b

@evgeniy-scherbina
Copy link
Contributor

evgeniy-scherbina commentedMay 2, 2025
edited
Loading

@dannykopping is it possible to use another approach, and allow callingPublish from within existing transaction?
For ex.: add smth like this:

func (p*PGPubsub)PublishWithinTx(eventstring,message []byte,tx database.Store)error {tx.ExecContext(context.Background(),`select pg_notify(`+pq.QuoteLiteral(event)+`, $1)`,message)}

Implementations which doesn't support txs may implement it:

func (p*InMemPubSub)PublishWithinTx(eventstring,message []byte,tx database.Store)error {_=tx// ignoredreturnp.PublishTx(event,message)}

But it will require some changes indatabase.Store andPGPubSub


Because we impose restrictions on ourselves, every time we need to use Publish within tx - we have to come up with workaround like in this PR.

Signed-off-by: Danny Kopping <dannykopping@gmail.com>
Copy link
Contributor

@spikecurtisspikecurtis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

LGTM

@dannykoppingdannykopping merged commita646478 intomainMay 5, 2025
29 checks passed
@dannykoppingdannykopping deleted the dk/notify-outside-tx branchMay 5, 2025 09:54
@github-actionsgithub-actionsbot locked and limited conversation to collaboratorsMay 5, 2025
@dannykopping
Copy link
ContributorAuthor

@dannykopping is it possible to use another approach, and allow callingPublish from within existing transaction? For ex.: add smth like this:

func (p*PGPubsub)PublishWithinTx(eventstring,message []byte,tx database.Store)error {tx.ExecContext(context.Background(),`select pg_notify(`+pq.QuoteLiteral(event)+`, $1)`,message)}

Implementations which doesn't support txs may implement it:

func (p*InMemPubSub)PublishWithinTx(eventstring,message []byte,tx database.Store)error {_=tx// ignoredreturnp.PublishTx(event,message)}

But it will require some changes indatabase.Store andPGPubSub

Because we impose restrictions on ourselves, every time we need to use Publish within tx - we have to come up with workaround like in this PR.

@evgeniy-scherbina I'll defer to@spikecurtis about your above question. However, I would imagine that we wouldn't want to overwhelm the API with knowledge of in- or out-of-transaction considerations.NOTIFYs within transactions (docs) have some semantics which may not always be desireable.

There are also some problems around 2PC ("A transaction that has executed NOTIFY cannot be prepared for two-phase commit."), and in general it just makes pubsub a bit less deterministic.

Sign up for freeto subscribe to this conversation on GitHub. Already have an account?Sign in.
Reviewers

@johnstcnjohnstcnjohnstcn left review comments

@spikecurtisspikecurtisspikecurtis approved these changes

@SasSwartSasSwartAwaiting requested review from SasSwart

Assignees

@dannykoppingdannykopping

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

4 participants
@dannykopping@evgeniy-scherbina@johnstcn@spikecurtis

[8]ページ先頭

©2009-2025 Movatter.jp