- Notifications
You must be signed in to change notification settings - Fork3.9k
feat(core): add redis lock during insertion of event in event table during initial attempt of outgoing webhook delivery#7579
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
Uh oh!
There was an error while loading.Please reload this page.
Conversation
semanticdiff-combot commentedMar 20, 2025 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
Changed Files
|
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
merchant_id.get_string_repr(), | ||
unique_locking_key | ||
); | ||
let redis_lock_expiry_seconds = state.conf().lock_settings.redis_lock_expiry_seconds; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
@jarnura Are we okay with using the same TTL as that used for API locking?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Have a separate locking expiry time
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
a511363
to7d21415
Compared9aaea6
toe7774d9
Compare.attach_printable("The redis value which acquired the lock is not equal to the redis value requesting for releasing the lock") | ||
} | ||
} | ||
Err(error) => Err(error).change_context(errors::ApiErrorResponse::InternalServerError), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Please attach context messages usingattach_printable()
/attach_printable_lazy()
whenever raising internal server errors.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Other than that, looks good to me!
Uh oh!
There was an error while loading.Please reload this page.
…uring initial attempt of outgoing webhook delivery
eed84fe
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
Type of Change
Description
The current implementation of insertion of an event in
event
table does not handle the case where multiple requests with the same idempotent_event_id might be processed concurrently, leading to multiple unique constraint violation errors in the database.This was reported by a merchant who was facing high volumes of unique constraint violation errors.
NOTE: this unique constraint log error issue is only applicable for the
initial
attempt of outgoing webhook delivery, not for retry attemptsWhy does this happen?
idempotent_event_id
might reach the database at the same time.Solution (Using Redis Lock)
Before querying the database, acquire a lock in Redis for the givenidempotent_event_id.
If there are multiple processes with sameidempotent_event_id only one process will hold the lock, others will leave early.
If a lock is acquired:
Additional Changes
Motivation and Context
How did you test it?
Cannot be directly tested.
Checklist
cargo +nightly fmt --all
cargo clippy