那些遇到过的问题

无法使用 grafana tempo 连接 opentelemetry 收集器

Dmi*_*nov 5 grafana open-telemetry grafana-tempo

我无法让酒店收集器工作，我是开放遥测技术的新手，所以感觉我在某个地方犯了一个愚蠢的错误

这是我的 python 示例脚本，它应该创建一个示例跟踪，该跟踪应该被导出，由 opentelemetry-collector 拾取并推送到 grafana tempo 后端

from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import (
    OTLPSpanExporter,
)
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import (
    BatchSpanProcessor,
    ConsoleSpanExporter,
)

span_exporter = OTLPSpanExporter(
   # endpoint="http://tempo.monitoring:3100"
   endpoint="10.120.4.111"
   # endpoint="http://10.120.7.235:4317"
)
provider = TracerProvider()
processor = BatchSpanProcessor(ConsoleSpanExporter())
span_processor = BatchSpanProcessor(span_exporter)
provider.add_span_processor(processor)
provider.add_span_processor(span_processor)
trace.set_tracer_provider(provider)


tracer = trace.get_tracer(__name__)

with tracer.start_as_current_span("foo"):
    with tracer.start_as_current_span("bar"):
        with tracer.start_as_current_span("baz"):
            print("Hello world from OpenTelemetry Python!")
Run Code Online (Sandbox Code Playgroud)

这是我运行时的输出：

description should only be set when status_code is set to StatusCode.ERROR
Hello world from OpenTelemetry Python!
{
    "name": "baz",
    "context": {
        "trace_id": "0xbc86870da9156c784c340ca16042bf6b",
        "span_id": "0xdd38d089ff192445",
        "trace_state": "[]"
    },
    "kind": "SpanKind.INTERNAL",
    "parent_id": "1c559a143ee166cd",
    "start_time": "2021-10-08T22:56:27.973528Z",
    "end_time": "2021-10-08T22:56:27.973569Z",
    "status": {
        "status_code": "UNSET"
    },
    "attributes": {},
    "events": [],
    "links": [],
    "resource": {
        "telemetry.sdk.language": "python",
        "telemetry.sdk.name": "opentelemetry",
        "telemetry.sdk.version": "1.0.0",
        "service.name": "unknown_service"
    }
}
{
    "name": "bar",
    "context": {
        "trace_id": "0xbc86870da9156c784c340ca16042bf6b",
        "span_id": "0x1c559a143ee166cd",
        "trace_state": "[]"
    },
    "kind": "SpanKind.INTERNAL",
    "parent_id": "0a5bd46eef0c046a",
    "start_time": "2021-10-08T22:56:27.973497Z",
    "end_time": "2021-10-08T22:56:27.973611Z",
    "status": {
        "status_code": "UNSET"
    },
    "attributes": {},
    "events": [],
    "links": [],
    "resource": {
        "telemetry.sdk.language": "python",
        "telemetry.sdk.name": "opentelemetry",
        "telemetry.sdk.version": "1.0.0",
        "service.name": "unknown_service"
    }
}
{
    "name": "foo",
    "context": {
        "trace_id": "0xbc86870da9156c784c340ca16042bf6b",
        "span_id": "0x0a5bd46eef0c046a",
        "trace_state": "[]"
    },
    "kind": "SpanKind.INTERNAL",
    "parent_id": null,
    "start_time": "2021-10-08T22:56:27.973419Z",
    "end_time": "2021-10-08T22:56:27.973622Z",
    "status": {
        "status_code": "UNSET"
    },
    "attributes": {},
    "events": [],
    "links": [],
    "resource": {
        "telemetry.sdk.language": "python",
        "telemetry.sdk.name": "opentelemetry",
        "telemetry.sdk.version": "1.0.0",
        "service.name": "unknown_service"
    }
}
Run Code Online (Sandbox Code Playgroud)

然而，在 grafana tempo 中找不到这个痕迹：

description should only be set when status_code is set to StatusCode.ERROR
Hello world from OpenTelemetry Python!
{
    "name": "baz",
    "context": {
        "trace_id": "0xbc86870da9156c784c340ca16042bf6b",
        "span_id": "0xdd38d089ff192445",
        "trace_state": "[]"
    },
    "kind": "SpanKind.INTERNAL",
    "parent_id": "1c559a143ee166cd",
    "start_time": "2021-10-08T22:56:27.973528Z",
    "end_time": "2021-10-08T22:56:27.973569Z",
    "status": {
        "status_code": "UNSET"
    },
    "attributes": {},
    "events": [],
    "links": [],
    "resource": {
        "telemetry.sdk.language": "python",
        "telemetry.sdk.name": "opentelemetry",
        "telemetry.sdk.version": "1.0.0",
        "service.name": "unknown_service"
    }
}
{
    "name": "bar",
    "context": {
        "trace_id": "0xbc86870da9156c784c340ca16042bf6b",
        "span_id": "0x1c559a143ee166cd",
        "trace_state": "[]"
    },
    "kind": "SpanKind.INTERNAL",
    "parent_id": "0a5bd46eef0c046a",
    "start_time": "2021-10-08T22:56:27.973497Z",
    "end_time": "2021-10-08T22:56:27.973611Z",
    "status": {
        "status_code": "UNSET"
    },
    "attributes": {},
    "events": [],
    "links": [],
    "resource": {
        "telemetry.sdk.language": "python",
        "telemetry.sdk.name": "opentelemetry",
        "telemetry.sdk.version": "1.0.0",
        "service.name": "unknown_service"
    }
}
{
    "name": "foo",
    "context": {
        "trace_id": "0xbc86870da9156c784c340ca16042bf6b",
        "span_id": "0x0a5bd46eef0c046a",
        "trace_state": "[]"
    },
    "kind": "SpanKind.INTERNAL",
    "parent_id": null,
    "start_time": "2021-10-08T22:56:27.973419Z",
    "end_time": "2021-10-08T22:56:27.973622Z",
    "status": {
        "status_code": "UNSET"
    },
    "attributes": {},
    "events": [],
    "links": [],
    "resource": {
        "telemetry.sdk.language": "python",
        "telemetry.sdk.name": "opentelemetry",
        "telemetry.sdk.version": "1.0.0",
        "service.name": "unknown_service"
    }
}
Run Code Online (Sandbox Code Playgroud)

opentelemtry-collector pod 没有显示任何有价值的内容

monitoring/opentelemetry-collector-agent-fhcd2[opentelemetry-collector]: 2021-10-08T22:58:16.162Z   INFO    loggingexporter/logging_exporter.go:375 MetricsExporter {"#metrics": 15}
monitoring/opentelemetry-collector-agent-bbsk8[opentelemetry-collector]: 2021-10-08T22:58:21.490Z   INFO    loggingexporter/logging_exporter.go:375 MetricsExporter {"#metrics": 15}
monitoring/opentelemetry-collector-agent-z8jlp[opentelemetry-collector]: 2021-10-08T22:58:22.314Z   INFO    loggingexporter/logging_exporter.go:375 MetricsExporter {"#metrics": 15}
Run Code Online (Sandbox Code Playgroud)

grafana节奏也“麻木”

monitoring/tempo-0[tempo]: level=info ts=2021-10-08T22:40:33.403096615Z caller=frontend.go:114 tenant=single-tenant method=GET traceID=470d349a2aa1a574 url=/api/traces/742a5bfccbfb8df24251a19c85600818 duration=1.040366ms response_size=0 status=404
monitoring/tempo-0[tempo]: level=info ts=2021-10-08T22:45:11.168260856Z caller=frontend.go:114 tenant=single-tenant method=GET traceID=281b800cb325d227 url=/api/traces/d2da9c5a4638245a8b7d8807934778bc duration=1.969761ms response_size=0 status=404
monitoring/tempo-0[tempo]: level=info ts=2021-10-08T22:57:32.889567771Z caller=frontend.go:114 tenant=single-tenant method=GET traceID=2f1eee7bb623353d url=/api/traces/bc86870da9156c784c340ca16042bf6b duration=1.123504ms response_size=0 status=404
host 10.120.4.11 has ports opened
[root@utils /]# curl -X POST 10.120.4.111:4317
curl: (1) Received HTTP/0.9 when not allowed
opentelemetry is running through this helm chart: https://open-telemetry.github.io/opentelemetry-helm-charts
Run Code Online (Sandbox Code Playgroud)

opentelemetry helm 图表的配置

config:
  exporters:
    logging: 
      loglevel: info
    otlp:
      endpoint: tempo.monitoring:4317
  receivers:
    otlp:
      grpc:
      http:
  service:
    pipelines:
      traces:
        receivers:
          - otlp
          - jaeger
          - zipkin
        processors:
          - batch
          - memory_limiter
        exporters:
          - logging
          - otlp

nodeSelector:
  apps: "true"            
standaloneCollector:
  enabled: false
Run Code Online (Sandbox Code Playgroud)

所以我尝试了 config.exporters.otlp.endpoint 的不同值 - 没有效果。

有任何想法吗？

小智 3

我遇到过类似的问题，我有来自 opentelemetry-collector 的跟踪，但无法在 Grafana Tempo 中导出它们。

我认为您的问题是接收器的配置似乎无效（但这只是猜测）。

我认为应该是这样的：

config:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
        http:
          endpoint: 0.0.0.0:4318
Run Code Online (Sandbox Code Playgroud)

一旦您在 opentelemetry-collector 中获得了跟踪，您应该会获得以下类型的日志：

2022-10-21T14:38:22.641Z        info    TracesExporter  {"kind": "exporter", "data_type": "traces", "name": "logging", "#spans": 5}
2022-10-21T14:38:32.658Z        info    TracesExporter  {"kind": "exporter", "data_type": "traces", "name": "logging", "#spans": 30}
Run Code Online (Sandbox Code Playgroud)

这是我使用社区 Helm 图表在 Kubernetes 上的工作配置：

头盔图表：

opentelemetry-collector 值：

config:
  exporters:
    logging:
      loglevel: info
    otlp:
      # Doc: https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/otlpexporter
      endpoint: tempo-distributed-distributor.monitoring.svc.cluster.local:4317
      tls:
        # Disabled TLS for this example
        # Doc : https://github.com/open-telemetry/opentelemetry-collector/tree/main/config/configtls
        insecure: true
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
        http:
          endpoint: 0.0.0.0:4318
  service:
    pipelines:
      traces:
        exporters:
          - logging
          - otlp
        processors:
          - memory_limiter
          - batch
        receivers:
          - otlp
Run Code Online (Sandbox Code Playgroud)

速度分布值：

distributor:
  config:
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: "0.0.0.0:4317"
          http:
            endpoint: "0.0.0.0:4318"
Run Code Online (Sandbox Code Playgroud)

希望这会帮助其他人！

归档时间：

查看次数：

5790 次

最近记录：

3 年 前
相关归档
在 Grafana 上使用 InfluxDB 计算每秒请求数 8
组合来自 2 个不同指标的标签值：Grafana 8
如何在 Grafana + InfluxDB 中使用键/值仪表板变量？ 6
如何在 Grafana 的时间序列中绘制水平线 6
如何在Grafana表（或singlestat）中显示InfluxDB测量的最新时间戳？ 5
OpenTelemetry Python - 如何将新的跨度实例化为给定trace_id的子跨度 5
计算grafana中两个查询之间的差异 3
将 Grafana 与 BigQuery 数据源结合使用 2
使用Telegraf(使用InfluxDB)在Grafana中使用网络图表是平的 1
使用 prometheus 监控 docker - 不同容器的 cpu 使用情况看起来相同 1
难疑归档
如何列出提交中的所有文件？ 2619
如何在Python中将字符串解析为float或int？ 2108
什么是TypeScript,为什么我会用它代替JavaScript？ 1637
使用其名称(字符串)调用模块的函数 1580
静态只读与const 1349
如何使用Sublime Text 2重新格式化HTML代码？ 1282
jQuery获取特定的选项标签文本 1211
如何获得最近提交的Git分支列表？ 1197
有条件地申请课程的最佳方式是什么？ 1172
type()和isinstance()之间有什么区别？ 1163