- Notifications
You must be signed in to change notification settings - Fork7
feat: implement retry mechanism for log processing#136
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
Uh oh!
There was an error while loading.Please reload this page.
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
What happens when the log never succeeds? Maybe because an agent token is no longer valid. Judging by the code, logstream will attempt to send it to coder Forever
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
l.logger.Error(ctx,"max retries exceeded", | ||
slog.F("retryCount",rs.retryCount), | ||
slog.F("maxRetries",l.maxRetries)) | ||
rs.exhausted=true |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
This will be kept in memory forever now right?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
It will be kept in memory until the pod is deleted.
https://github.com/coder/coder-logstream-kube/blob/kacpersaw/feat-log-queue/logger.go#L562
…efault of 15 retries for log send failures.
a5bd376
intomainUh oh!
There was an error while loading.Please reload this page.
Closes#119
This PR adds error handling and retry functionality to the log streaming by adding exponential backoff retry logic for failed log operations.
Changes
PostLogSource
andAgent dRPC client
failures to handle connection issuesWith this PR, users will see more reliable logs from pod startup in their Coder workspace, including pod events before container startup that were previously not visible.