Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

Make exportertimeout encompass retries/backoffs, add jitter to backoffs, cleanup code a bit#4564

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Merged
xrmx merged 38 commits intoopen-telemetry:mainfromDylanRussell:retry2
Jun 13, 2025

Conversation

@DylanRussell
Copy link
Contributor

@DylanRussellDylanRussell commentedApr 30, 2025•
edited
Loading

Description

Maketimeout encompass retries and backoffs, rather than being applied per HTTP request or gRPC RPC.

Added a +/- 20% jitter to each backoff (both gRPC/HTTP).

Cleanup up the exporter code some. I got rid of a pointless 32 second sleep we would do after our last retry attempt before failing.

fixes:#3309,#4043,#2663

#4183 -- similar to this PR and what's discussed in#4043, but I implemented it in as minimal a way as I could..

Fixes # (issue)

Type of change

Please delete options that are not relevant.

  • [ x] New feature (non-breaking change which adds functionality)

How Has This Been Tested?

Lots of unit tests.

Does This PR Require a Contrib Repo Change?

  • Yes. - Link to PR:
  • [x ] No.

Checklist:

  • [ x] Followed the style guidelines of this project
  • [ x] Changelogs have been updated
  • [x ] Unit tests have been added
  • [x ] Documentation has been updated

Copy link
Member

@emdnetoemdneto left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Running a basic example of sending span/metric to a non existent collector through grpc:

Before (timeout not respected)
$ OTEL_EXPORTER_OTLP_TIMEOUT=5 uv run repro.py2025-05-09 01:19:57 INFO [test] Hello world2025-05-09 01:19:57 WARNING [opentelemetry.exporter.otlp.proto.grpc.exporter] Transient error StatusCode.UNAVAILABLE encountered while exporting metrics to localhost:4317, retrying in 1s.2025-05-09 01:19:58 WARNING [opentelemetry.exporter.otlp.proto.grpc.exporter] Transient error StatusCode.UNAVAILABLE encountered while exporting metrics to localhost:4317, retrying in 2s.2025-05-09 01:20:00 WARNING [opentelemetry.exporter.otlp.proto.grpc.exporter] Transient error StatusCode.UNAVAILABLE encountered while exporting metrics to localhost:4317, retrying in 4s.^[[B2025-05-09 01:20:02 WARNING [opentelemetry.exporter.otlp.proto.grpc.exporter] Transient error StatusCode.UNAVAILABLE encountered while exporting traces to localhost:4317, retrying in 1s.2025-05-09 01:20:03 WARNING [opentelemetry.exporter.otlp.proto.grpc.exporter] Transient error StatusCode.UNAVAILABLE encountered while exporting traces to localhost:4317, retrying in 2s.2025-05-09 01:20:04 WARNING [opentelemetry.exporter.otlp.proto.grpc.exporter] Transient error StatusCode.UNAVAILABLE encountered while exporting metrics to localhost:4317, retrying in 8s.2025-05-09 01:20:05 WARNING [opentelemetry.exporter.otlp.proto.grpc.exporter] Transient error StatusCode.UNAVAILABLE encountered while exporting traces to localhost:4317, retrying in 4s.2025-05-09 01:20:09 WARNING [opentelemetry.exporter.otlp.proto.grpc.exporter] Transient error StatusCode.UNAVAILABLE encountered while exporting traces to localhost:4317, retrying in 8s.2025-05-09 01:20:12 WARNING [opentelemetry.exporter.otlp.proto.grpc.exporter] Transient error StatusCode.UNAVAILABLE encountered while exporting metrics to localhost:4317, retrying in 16s.
Now (timeout is respected)
$ OTEL_EXPORTER_OTLP_TIMEOUT=5 uv run repro.py2025-05-09 01:22:43 INFO [test] Hello world2025-05-09 01:22:48 ERROR [opentelemetry.exporter.otlp.proto.grpc.exporter] Failed to export metrics to localhost:4317, error code: StatusCode.DEADLINE_EXCEEDED2025-05-09 01:22:53 ERROR [opentelemetry.exporter.otlp.proto.grpc.exporter] Failed to export traces to localhost:4317, error code: StatusCode.DEADLINE_EXCEEDED

When exporting metrics/traces in the same program I noticed that:
Is this expected?

$ OTEL_EXPORTER_OTLP_TRACES_TIMEOUT=5 uv run repro.py2025-05-09 15:37:54 INFO [test] Hello world2025-05-09 15:38:04 ERROR [opentelemetry.exporter.otlp.proto.grpc.exporter] Failed to export traces to localhost:4317, error code: StatusCode.DEADLINE_EXCEEDED2025-05-09 15:38:04 ERROR [opentelemetry.exporter.otlp.proto.grpc.exporter] Failed to export metrics to localhost:4317, error code: StatusCode.DEADLINE_EXCEEDED

Copy link
Member

@aabmassaabmass left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

@aabmass
Copy link
Member

I updated theLogExporter andSpanExporter interfaces to includetimeout_millis (MetricExporter already has it), this isn't a breaking change because@abstractmethod just checks that classes implementin have a method with a particular name, it doesn't check method parameters at all.. It does cause a pylint error, but I think that's a good thing.

I didn't see this in the code, if you updated can you just update the description as well?

@DylanRussell
Copy link
ContributorAuthor

Updated the description

@DylanRussellDylanRussell changed the titleAdd a timeout param to all OTLP grpc / http export calls -- fixed merge conflictsSwitch gRPC exporters to use official gRPC retry config. Maketimeout encompass retries/backoffsMay 22, 2025
@DylanRussell
Copy link
ContributorAuthor

@emdneto any additional comments on this since I made changes to remove the retry config ? Otherwise I think this can be merged

…try/exporter/otlp/proto/grpc/exporter.pyCo-authored-by: Emídio Neto <9735060+emdneto@users.noreply.github.com>
Copy link
Member

@emdnetoemdneto left a comment•
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

@DylanRussell running griffe I got:

griffe check opentelemetry-exporter-otlp-proto-http -a main -s exporterexporter/opentelemetry-exporter-otlp-proto-http/tests/test_proto_span_exporter.py:0: TestOTLPSpanExporter.test_exponential_backoff: Public object was removedexporter/opentelemetry-exporter-otlp-proto-http/tests/test_proto_log_exporter.py:0: TestOTLPHTTPLogExporter.test_exponential_backoff: Public object was removedexporter/opentelemetry-exporter-otlp-proto-http/tests/metrics/test_otlp_metrics_exporter.py:0: TestOTLPMetricExporter.test_exponential_backoff: Public object was removedexporter/opentelemetry-exporter-otlp-proto-http/src/opentelemetry/exporter/otlp/proto/http/metric_exporter/__init__.py:205: OTLPMetricExporter.export(timeout_millis): Parameter default was changed: 10000 -> None

@aabmass
Copy link
Member

Good to go@emdneto ?

@emdnetoemdneto self-requested a reviewJune 10, 2025 21:56
@xrmxxrmxenabled auto-merge (squash)June 13, 2025 07:29
@xrmxxrmx merged commit6ed676a intoopen-telemetry:mainJun 13, 2025
351 of 353 checks passed
@github-project-automationgithub-project-automationbot moved this fromReady for review toDone in@xrmx's Python PR digestJun 13, 2025
mxiamxia pushed a commit to aws-observability/aws-otel-python-instrumentation that referenced this pull requestOct 30, 2025
…58b0 (#524)This PR updates the upstream OpenTelemetry Python dependency to itsSeptember 2025 release; Upgrading from version 1.33.1/0.54b1 to1.37.0/0.58b0.It also resolves several conflicts between the following OTel PRs andexisting ADOT patches:starlette: Remove maximum version constraintopen-telemetry/opentelemetry-python-contrib#3456Make a BatchProcessor class which both BatchSpanRecordProcessor andBatchLogRecordProcessor can useopen-telemetry/opentelemetry-python#4562Make exporter timeout encompass retries/backoffs, add jitter tobackoffs, cleanup code a bitopen-telemetry/opentelemetry-python#4564Update BatchSpanProcessor to use new BatchProcessor classopen-telemetry/opentelemetry-python#4580Fix issue where deadlock can occur over logging._lockopen-telemetry/opentelemetry-python#4636Tests Performedtox -e linttox -e spellchecktox -e 3.9-test-aws-opentelemetry-distrotox -e 3.10-test-aws-opentelemetry-distrotox -e 3.11-test-aws-opentelemetry-distrotox -e 3.12-test-aws-opentelemetry-distrotox -e 3.13-test-aws-opentelemetry-distroSmoke/contract tests: ./gradlewappsignals-tests:contract-tests:contractTestsBy submitting this pull request, I confirm that you can use, modify,copy, and redistribute this contribution, under the terms of yourchoice.
Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment

Reviewers

@xrmxxrmxxrmx approved these changes

@aabmassaabmassaabmass approved these changes

@emdnetoemdnetoemdneto approved these changes

Assignees

No one assigned

Labels

None yet

Projects

Status: Done

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Exporters shutdown takes longer then a minute when failing to send metrics/traces

4 participants

@DylanRussell@aabmass@xrmx@emdneto
[8]ページ先頭
©2009-2025 Movatter.jp