Fibre monitoring

This page covers observability in Fibre v10.4.0-mocha. Complete the server setup first. Add the flags below to your existing start command, keeping your home and node addresses. All observability flags apply to every subcommand.

Logging

fibre start --log-level debug --log-format json

Tracing and metrics

Fibre sends traces and metrics over OTLP/HTTP to an OpenTelemetry collector, such as Grafana Alloy or the OTel Collector. The collector must accept both signals and forward them to your tracing and metrics backends. The celestia-app repository ships a reference collector configuration  and a Docker Compose stack  with Prometheus, Tempo, and Grafana that you can start from. The stack publishes Prometheus on host port 9090, which the setup guide uses for application gRPC. When running it on the validator host, change that mapping in docker-compose.yml, for example to 9091:9090.

fibre start --otel-endpoint http://localhost:4318

Use the collector’s base URL. Fibre appends /v1/traces and /v1/metrics itself. For example, https://collector.example.com/otel sends data to /otel/v1/traces and /otel/v1/metrics.

Grafana dashboard

Import the release’s Fibre dashboard JSON  into Grafana. The dashboard expects a Prometheus data source with UID prometheus; use that UID or update the dashboard to match your data source. Client panels need telemetry from Fibre clients as well as the server. The same directory has a Fibre runtime dashboard  for Go runtime metrics such as memory and goroutines.

To verify the pipeline, start Fibre with --otel-endpoint set, open the Fibre dashboard, and confirm the server panels populate once uploads or downloads arrive. If they stay empty, work through the troubleshooting table.

Community dashboards

Use these community-maintained dashboards to monitor Fibre on Mocha:

See Tensile’s methodology  for how observations and verdicts are calculated.

Metric reference

Traces

Fibre samples 10% of root spans and respects upstream sampling decisions, using ParentBased(TraceIDRatioBased(0.1)). W3C TraceContext and Baggage propagation carry trace context across gRPC and HTTP calls.

Each trace includes service.name=fibre, service.version, and service.instance.id (the hostname).

Metrics

Metrics are exported periodically over OTLP. Duration histograms include a success attribute, so their counts can be used to calculate error rates. Metric exemplars link observations to traces; in Grafana, select an exemplar to open its trace.

The tables below list the original OTLP metric names. Your collector or backend may convert dots to underscores and add unit suffixes.

Server metrics

fibre.server.sign.duration measures how long Fibre waits for payment promise endorsements. It is not the metric behind the 10 ms signing latency requirement; that is celestia-app’s cometbft_privval_signing_latency_*, see signing latency on the setup page.

Client metrics

These metrics come from instrumented Fibre clients. Collect client telemetry alongside server telemetry to measure end-to-end uploads and downloads.

Profiling with pprof

Enable the HTTP profiling server when you need to inspect CPU use, memory, goroutines, or contention:

fibre start --pprof

The default address is localhost:6060. To use a different local port:

fibre start --pprof=localhost:7070

Mutex and block profiling start automatically with the pprof server, using mutex fraction 5 and block rate 1.

Continuous profiling with Pyroscope

Fibre can push profiles to a Pyroscope server. When tracing and Pyroscope are both enabled, goroutine labels include span IDs so Grafana can link traces to profiles. Profiles have version and hostname tags for filtering.

fibre start --pyroscope-endpoint http://localhost:4040

For an endpoint that requires authentication:

fibre start \
  --pyroscope-endpoint https://<pyroscope_host> \
  --pyroscope-basic-auth-user <user_id> \
  --pyroscope-basic-auth-password <api_key>

Troubleshooting

See the release’s observability reference  for more details.