Posted onMay 8, 2023

otel-cli - push otel traces with ease

#opentelemetry #bash

The problem

otel-cli is a command-line interface tool that simplifies OpenTelemetry instrumentation and enables developers to quickly get started with distributed tracing. It supports a wide range of programming languages and frameworks, and provides a variety of exporters, including Jaeger, Zipkin, and Prometheus, among others.

One of the main advantages of usingotel-cli is its simplicity. It provides a simple and intuitive interface that makes it easy for developers to get started with tracing, and provides real-time feedback on the traces generated by your application, making it easy to identify and diagnose issues in real-time.

Another advantage ofotel-cli is its flexibility. It supports multiple exporters, giving developers the flexibility to send traces to a wide range of monitoring and observability tools. Additionally, it is an open-source tool, which means that it is available for free and can be used and modified by anyone.

Installation

On mac

brew tap equinix-labs/otel-clibrew install otel-cli

On other systems
Get it from thereleases link

Configuration

❯ docker run --name=otel-col -p 4318:4318 -p 4317:4317  -d otel/opentelemetry-collector638e85e867aa4b69b356b5053991ce097d946a0aa9ecfa7568c867113209a307otel-demo/python-http on  main [!⇡] via 🐍 v3.9.7 on ☁️❯ docker psCONTAINER ID   IMAGE                          COMMAND                  CREATED         STATUS        PORTS                                                                 NAMES638e85e867aa   otel/opentelemetry-collector   "/otelcol --config /…"   2 seconds ago   Up 1 second   0.0.0.0:4317-4318->4317-4318/tcp, 55678-55679/tcp                     otel-col

Workflow

Simple operation

# pushing in grpc mode❯ OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 otel-cli exec --name=test-001 "sleep 1" --verbose --protocol grpc❯ docker logs otel-col 2>&1 | grep -C5 test-001InstrumentationScope otel-cli/execSpan #0    Trace ID       : 874ff9770b8f13f3276e1837b377e861    Parent ID      :    ID             : 36e4f9b9a33b4278    Name           : test-001    Kind           : Client    Start time     : 2023-05-08 13:28:39.799345 +0000 UTC    End time       : 2023-05-08 13:28:40.816506833 +0000 UTC    Status code    : Ok    Status message :# pushing in http mode (default)❯ OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 otel-cli exec --name=test-002 "sleep 1" --verbose❯ docker logs otel-col 2>&1 | grep -C5 test-002InstrumentationScope otel-cli/execSpan #0    Trace ID       : 5eff81fbe83ab6bd2027d00a289093f8    Parent ID      :    ID             : 85ae91f4d46f73c3    Name           : test-002    Kind           : Client    Start time     : 2023-05-08 13:30:00.372481 +0000 UTC    End time       : 2023-05-08 13:30:01.382757833 +0000 UTC    Status code    : Ok    Status message :

Note the below failure modes

# pushing to grpc endpoint in http mode❯ OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 otel-cli exec --name=test-001 "sleep 1" --verbose2023/05/08 18:58:01 OpenTelemetry error: traces export: Post "http://localhost:4317/v1/traces": net/http: HTTP/1.x transport connection broken: malformed HTTP response "\x00\x00\x06\x04\x00\x00\x00\x00\x00\x00\x05\x00\x00@\x00"# pushing to http endpoint in grpc mode❯ OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 otel-cli exec --name=test-001 "sleep 1" --verbose --protocol grpc2023/05/08 18:59:46 OpenTelemetry error: traces export: context deadline exceeded: rpc error: code = Unavailable desc = connection error: desc = "error reading server preface: http2: frame too large"

Child spans

❯ OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 otel-cli exec --name=test-demo-parent --kind producer "OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 otel-cli exec --name=test-demo-child --kind consumer sleep 1" --verbose❯ docker logs otel-col 2>&1 | egrep -C5 "test-demo"InstrumentationScope otel-cli/execSpan #0    Trace ID       : e383c3e41d049f3b11a85dad316643b7    Parent ID      : c359f233b3ad20ee    ID             : 9a70f7873e23b236    Name           : test-demo-child    Kind           : Consumer    Start time     : 2023-05-08 13:40:42.674921 +0000 UTC    End time       : 2023-05-08 13:40:43.684543334 +0000 UTC    Status code    : Ok    Status message :--InstrumentationScope otel-cli/execSpan #0    Trace ID       : e383c3e41d049f3b11a85dad316643b7    Parent ID      :    ID             : c359f233b3ad20ee    Name           : test-demo-parent    Kind           : Producer    Start time     : 2023-05-08 13:40:42.659909 +0000 UTC    End time       : 2023-05-08 13:40:43.721519167 +0000 UTC    Status code    : Ok    Status message :Attributes:     -> command: Str(OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 otel-cli exec --name=test-demo-child --kind consumer sleep 1)

With attributes

❯ OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 otel-cli exec --name=test-attr -a foo=bar -a pi=3.14 "sleep 1" --verbose❯ docker logs otel-col 2>&1 | egrep -C10 "test-attr"Resource SchemaURL:Resource attributes:     -> service.name: Str(otel-cli)ScopeSpans #0ScopeSpans SchemaURL:InstrumentationScope otel-cli/execSpan #0    Trace ID       : dca4cf94916d31c29c160a23a8418024    Parent ID      :    ID             : 5a4fe44b46241c90    Name           : test-attr    Kind           : Client    Start time     : 2023-05-08 13:43:08.812387 +0000 UTC    End time       : 2023-05-08 13:43:09.82861175 +0000 UTC    Status code    : Ok    Status message :Attributes:     -> foo: Str(bar)     -> pi: Double(3.14)     -> command: Str(sleep 1)    {"kind": "exporter", "data_type": "traces", "name": "logging"}

With status update (exit status is considered)

❯ OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 otel-cli exec --name=test-status  --fail "ls /nothing" --verbosels: /nothing: No such file or directory❯ docker logs otel-col 2>&1 | egrep -C12 "test-status"2023-05-08T13:44:49.513Z    info    TracesExporter  {"kind": "exporter", "data_type": "traces", "name": "logging", "resource spans": 1, "spans": 1}2023-05-08T13:44:49.513Z    info    ResourceSpans #0Resource SchemaURL:Resource attributes:     -> service.name: Str(otel-cli)ScopeSpans #0ScopeSpans SchemaURL:InstrumentationScope otel-cli/execSpan #0    Trace ID       : 183374319aa00c0672b0b9921a1cf862    Parent ID      :    ID             : 6e47444744c67a35    Name           : test-status    Kind           : Client    Start time     : 2023-05-08 13:44:49.346037 +0000 UTC    End time       : 2023-05-08 13:44:49.3641295 +0000 UTC    Status code    : Error    Status message : command failed: exit status 1Attributes:     -> command: Str(ls /nothing)Events:SpanEvent #0     -> Name: command failed     -> Timestamp: 2023-05-08 13:44:49.364126 +0000 UTC     -> DroppedAttributesCount: 0

Trace propagation with a file based trace-parent carrier

❯ OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318❯ carrier=$(mktemp)❯ otel-cli span -s test-span -k server -n "traceparent-demo" --tp-print --tp-carrier $carrier# trace id: f0563604541c82962e9b2434f4522e86#  span id: 17bfae9cd27127ddTRACEPARENT=00-f0563604541c82962e9b2434f4522e86-17bfae9cd27127dd-01❯ otel-cli exec --service test-span -n "traceparent-demo-client" --kind client --tp-carrier $carrier "sleep 1"❯ docker logs otel-col 2>&1 | egrep -C5 "traceparent-demo"InstrumentationScope otel-cli/spanSpan #0    Trace ID       : f0563604541c82962e9b2434f4522e86    Parent ID      :    ID             : 17bfae9cd27127dd    Name           : traceparent-demo    Kind           : Server    Start time     : 2023-05-08 13:48:31.241478 +0000 UTC    End time       : 2023-05-08 13:48:31.241782 +0000 UTC    Status code    : Unset    Status message :--InstrumentationScope otel-cli/execSpan #0    Trace ID       : f0563604541c82962e9b2434f4522e86    Parent ID      : 17bfae9cd27127dd    ID             : 47674b2d5cc516ea    Name           : traceparent-demo-client    Kind           : Client    Start time     : 2023-05-08 13:50:28.751103 +0000 UTC    End time       : 2023-05-08 13:50:29.776050709 +0000 UTC    Status code    : Ok    Status message :

Spans in the background

❯ sockdir=$(mktemp -d)❯ otel-cli span background -s "span-example" -n test-bg-span --sockdir $sockdir --timeout 60s &[1] 49185✦ ❯ otel-cli span event --name "cool thing" --attrs "foo=bar" --sockdir $sockdir✦ ❯ otel-cli span event --name "another cool thing" --attrs "foo=bar,pi=3.14" --sockdir $sockdir✦ ❯ otel-cli span event --name "final cool thing" --attrs "foo=bar,pi=3.14" --sockdir $sockdir✦ ❯ otel-cli span end --sockdir $sockdir[1]  + 49185 done       otel-cli span background -s "span-example" -n test-bg-span --sockdir $sockdir❯ docker logs otel-col 2>&1 | egrep -C5 "span-example|cool thing"Resource SchemaURL:Resource attributes:     -> service.name: Str(span-example)ScopeSpans #0ScopeSpans SchemaURL:InstrumentationScope otel-cli/spanSpan #0    Trace ID       : 944487fedd6ea482908fbb26f83f0329--    End time       : 2023-05-08 14:18:31.290052 +0000 UTC    Status code    : Unset    Status message :Events:SpanEvent #0     -> Name: cool thing     -> Timestamp: 2023-05-08 14:17:53.720251 +0000 UTC     -> DroppedAttributesCount: 0     -> Attributes::          -> foo: Str(bar)SpanEvent #1     -> Name: another cool thing     -> Timestamp: 2023-05-08 14:18:06.132737 +0000 UTC     -> DroppedAttributesCount: 0     -> Attributes::          -> foo: Str(bar)          -> pi: Double(3.14)SpanEvent #2     -> Name: final cool thing     -> Timestamp: 2023-05-08 14:18:15.368533 +0000 UTC     -> DroppedAttributesCount: 0     -> Attributes::          -> foo: Str(bar)          -> pi: Double(3.14)

Official git repo