Skip to main content

Observability enhancements

Centrifugo PRO provides enhanced observability, as when the business grows it's crucial to have deep insight into the system.

Client name resolution in metrics​

Centrifugo PRO has some enhancements to exposed metrics.

It's possible to understand how many clients from different environments are currently connected to your Centrifugo — i.e. from a browser, from Android, iOS devices. This is possible because our SDKs pass the name of the SDK to a server and provide a way to redefine it.

Names of clients you are using in SDKs must be registered in Centrifugo configuration. This is done to avoid cardinality issues in Prometheus.

config.json
{
  "prometheus": {
    "enabled": true,
    "additional_client_names": [
      "my-name1",
      "my-name2"
    ]
  }
}

Centrifugo PRO is already aware of some names used by our official SDKs, so out of the box you will get segmentation by those.

Channel namespace resolution for metrics​

Centrifugo PRO supports channel namespace resolution for many metrics related to channel. One application could be for setups with many namespaces, to understand which namespaces consume more bandwidth, or which namespace generates more frames or errors. Or the number of inflight subscriptions with channel namespace resolution!

To enable:

config.json
{
  "prometheus": {
    "enabled": true,
    "channel_namespace_resolution": true
  }
}

Centrifugo PRO requires a separate flag to enable channel namespace resolution for metrics because it may have some overhead (in most cases negligible though).

Transport accept protocol resolution​

Centrifugo PRO can expose the accept protocol used by client's transport in metric labels. This allows you to understand which protocols clients are using to establish connections - for example, distinguishing between WebSocket connections that were established via HTTP/1.1 versus HTTP/2 or HTTP/3, or tracking HTTP-streaming and SSE connections by their underlying HTTP protocol version.

To enable:

config.json
{
  "prometheus": {
    "enabled": true,
    "expose_transport_accept_protocol": true
  }
}

When enabled, the following metrics will include the accept_protocol label:

  • centrifugo_client_connections_accepted - counter of accepted connections
  • centrifugo_client_connections_inflight - gauge of current connections

The accept_protocol label can have the following values:

  • h1 - HTTP/1.1
  • h2 - HTTP/2
  • h3 - HTTP/3

This helps in understanding the protocol distribution across your infrastructure and can be useful for performance analysis and infrastructure planning.

Client labels as Prometheus dimensions​

Centrifugo PRO can export selected client labels as additional Prometheus dimensions on per-client metrics. Combined with labels set from JWT or the connect proxy, this gives per-tier, per-region, per-app-version breakdowns of connection-level metrics without operating multiple Centrifugo deployments.

To enable, list the label keys to export under prometheus.client_labels:

config.json
{
  "prometheus": {
    "enabled": true,
    "client_labels": ["region", "tier", "app_version"]
  }
}

Exported dimension names are prefixed with app_ to guarantee they cannot collide with built-in metric labels — for example, client_labels: ["region", "tier"] becomes the Prometheus dimensions app_region and app_tier. Your application code reads the unprefixed keys via labels.region / labels.tier (e.g., in CEL expressions or proxy requests) — the prefix is applied only on the metric export path.

When a configured key is missing on a particular client, the empty string is used as the dimension value (so all metric series stay shape-consistent).

Cardinality

Every unique combination of exported label values creates a new Prometheus time series. Keep the value set bounded and small — region (5–20 values), tier (3–5 values), app version (dozens, not millions). Do not export user IDs, session IDs, request IDs, or any unbounded input. Combined with built-in labels like transport and op, even a few high-cardinality keys can multiply your time series count rapidly.

The same caveat applies to the analytics labels column — different storage, same cardinality concern.

When client_labels is empty (the default), no label dimensions are exported and there is zero overhead on the metric emission path.

OpenTelemetry metrics export​

New in Centrifugo PRO v6.8.1

Centrifugo PRO can export its metrics to an OpenTelemetry-compatible backend (Grafana Cloud, GCP Cloud Operations, Datadog, AWS CloudWatch via OTLP, OTel Collector, etc.) without running a Prometheus sidecar. Internally Centrifugo continues to use Prometheus instrumentation, then a bridge translates the metrics registry into OTLP and pushes them via the OTel SDK.

To enable, both flags are needed — the OpenTelemetry section turns the subsystem on, and metrics: true activates the metrics pipeline (in addition to traces):

config.json
{
  "opentelemetry": {
    "enabled": true,
    "metrics": true
  }
}

Endpoint, headers, and protocol are configured via the standard OTEL_EXPORTER_OTLP_* environment variables — the same ones that drive trace export:

OTEL_EXPORTER_OTLP_ENDPOINT="https://otlp.example.com" \
OTEL_EXPORTER_OTLP_HEADERS="api-key=..." \
OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf" \
./centrifugo

OTEL_EXPORTER_OTLP_PROTOCOL accepts http/protobuf (default) or grpc.

Exported metrics carry standard OTel resource attributes: service.name is centrifugo (override with OTEL_SERVICE_NAME), attributes from OTEL_RESOURCE_ATTRIBUTES are merged in (taking precedence over Centrifugo defaults), and since Centrifugo PRO v6.8.3 service.instance.id defaults to the unique Centrifugo node ID (regenerated on each process start).

If your backend is Google Cloud, set opentelemetry.google_cloud_adc_auth to push metrics straight to telemetry.googleapis.com without a sidecar — see Export to Google Cloud (ADC) below. This is a base OpenTelemetry option available in Centrifugo OSS, where it authenticates trace export; in Centrifugo PRO the same single setting also covers the metrics pipeline.

Export to Google Cloud (ADC)​

New in Centrifugo PRO v6.8.2

Google Cloud's OTLP endpoint (telemetry.googleapis.com) requires every request to carry a valid OAuth2 access token. The standard OTLP exporter does not attach one, so pushing metrics straight to Google Cloud fails as unauthenticated unless you run a sidecar collector to inject credentials — which defeats the point of moving off a Prometheus sidecar.

Set opentelemetry.google_cloud_adc_auth to true to make Centrifugo authenticate the exporter with Google Cloud Application Default Credentials (ADC). Then metrics (and traces) can be pushed directly to telemetry.googleapis.com without a sidecar:

config.json
{
  "opentelemetry": {
    "enabled": true,
    "metrics": true,
    "google_cloud_adc_auth": true,
    "resource_detectors": ["gcp"]
  }
}
OTEL_EXPORTER_OTLP_ENDPOINT="https://telemetry.googleapis.com" \
OTEL_EXPORTER_OTLP_PROTOCOL="grpc" \
OTEL_RESOURCE_ATTRIBUTES="gcp.project_id=YOUR_PROJECT_ID" \
./centrifugo

resource_detectors: ["gcp"] is optional but recommended — see Cloud resource detectors.

The option works with both exporter protocols — over grpc the ADC token is attached as a per-RPC credential, over http/protobuf via an OAuth2 HTTP client transport. In both cases the token is minted lazily on first export and then cached and refreshed automatically. google_cloud_adc_auth is a base OpenTelemetry option (shared with tracing), so a single setting covers both pipelines.

tip

Set the target project via OTEL_RESOURCE_ATTRIBUTES="gcp.project_id=...". Do not put it in OTEL_EXPORTER_OTLP_HEADERS as x-goog-user-project — Google warns that this can produce duplicate values and fail requests.

Each instance must report a unique identity

Google Cloud derives the instance label of the time series identity from the service.instance.id resource attribute and requires points of each series to arrive in order. When several Centrifugo instances report under the same service.instance.id — or without one at all — Google Cloud rejects their interleaved points as out-of-order and metrics may collapse to zero. Since Centrifugo PRO v6.8.3 each process reports its unique node ID as service.instance.id, so autoscaled deployments get distinct time series automatically. To use your own identity, set OTEL_RESOURCE_ATTRIBUTES="service.instance.id=..." — the environment value takes precedence.

ADC must be resolvable in the runtime environment — automatic on GKE/GCE/Cloud Run via the attached service account, or locally via GOOGLE_APPLICATION_CREDENTIALS / gcloud auth application-default login.

Cloud resource detectors​

New in Centrifugo PRO v6.8.3

opentelemetry.resource_detectors lists cloud platforms whose resource attributes Centrifugo detects from the platform metadata service at startup and attaches to exported metrics and traces:

config.json
{
  "opentelemetry": {
    "enabled": true,
    "metrics": true,
    "resource_detectors": ["gcp"]
  }
}

Supported values:

  • gcp — cloud.region / cloud.availability_zone, k8s.cluster.name on GKE, faas.* service info on Cloud Run.
  • aws — cloud.region / cloud.availability_zone, host info on EC2, container and task info on ECS.

For Google Cloud these attributes fill the location and cluster labels of the time series identity — without them all metrics land in location global. Detection outside the listed platform is a no-op, and detected values can be overridden via OTEL_RESOURCE_ATTRIBUTES. If Centrifugo runs on the platform but metadata lookups fail (server not ready yet, blocked by network policy), it exits with an error at startup rather than exporting telemetry under an incomplete identity.

Pair with native histograms for full fidelity​

By default, Prometheus Histograms are translated to OTel fixed-bucket histograms. To get the high-fidelity ExponentialHistogram representation that most OTel-native backends prefer, enable native histograms alongside:

config.json
{
  "prometheus": {
    "enabled": true,
    "native_histograms": true
  },
  "opentelemetry": {
    "enabled": true,
    "metrics": true
  }
}

This is the recommended configuration for new OTel-only deployments. Prometheus scraping is not required when the OTLP push pipeline is the source of truth — you can leave prometheus.enabled: false if there are no Prometheus consumers (the bridge still works against the in-process registry).

Summary instruments are deprecated​

Deprecation notice

All Prometheus Summary instruments in Centrifugo are deprecated as of v6.8.1 and will be removed in Centrifugo v7. Use the _histogram companions instead — they expose the same data in a form that aggregates correctly across nodes (histogram_quantile()) and translates cleanly to OpenTelemetry.

With prometheus.native_histograms: true recommended above, Centrifugo stops exposing all Prometheus Summary instruments — every duration/distribution metric is carried by its _histogram companion (which uses native exponential schema when the flag is on). The OTel pipeline therefore carries only Histograms and ExponentialHistograms — clean ingest at OTel-native backends.

If you leave prometheus.native_histograms off but still enable opentelemetry.metrics, Centrifugo's Summary metrics will be translated to OTel's legacy Summary data point shape, which most OTel-native backends treat as second-class data or drop at ingest. Use the corresponding _histogram companion metrics in your OTel dashboards in that mode, or — strongly recommended — enable native histograms and get the clean pipeline.

The pro-only Summary metrics deprecated by the same migration are: centrifugo_push_job_duration_seconds, centrifugo_clickhouse_analytics_flush_duration_seconds, centrifugo_clickhouse_analytics_batch_size, and centrifugo_shared_poll_relay_backend_duration_seconds. Each has an _histogram companion (added in v6.8.1) that becomes the canonical instrument when native histograms is enabled.

Why Summaries are being removed: in a clustered Centrifugo deployment (multiple nodes), Summary's pre-computed quantile estimates cannot be aggregated across instances — there's no mathematically valid way to combine per-node p99s into a fleet-wide p99. Histograms solve this by aggregating bucket counts across nodes, then computing percentiles with histogram_quantile(). For any multi-node deployment the Summary quantile data is, at best, misleading.

PRO metrics reference​

Beyond the enhanced labels described above, Centrifugo PRO exposes its own metrics. They follow the same conventions as the OSS metrics reference: the centrifugo_ namespace, Histograms that switch to native schema when prometheus.native_histograms is on, and deprecated Summaries that disappear in that mode.

All of them are visualized by the PRO · … rows of the official Grafana dashboard.

One family of PRO-only metrics is described in the OSS reference rather than here, because it sits next to closely related OSS metrics: the *_redis_node_grouped_* Redis Cluster metrics, which only the PRO node-grouped sharded PUB/SUB path can fill. They are marked as PRO there and are not registered at all in Centrifugo OSS.

Note that the map broker and the PostgreSQL broker are not PRO features — both ship in Centrifugo OSS, so their metrics (centrifugo_map_broker_*, centrifugo_broker_postgres_*) belong to the OSS reference. What PRO adds on top is described in map subscriptions — per-namespace map brokers and sharded PUB/SUB among them.

Push notifications​

centrifugo_push_notification_count​

  • Type: Counter
  • Labels: provider, recipient_type, platform, success, err_code
  • Description: Count of push notifications sent, split by provider (fcm, apns, hms), recipient type, platform, whether the provider accepted it, and the provider error code when it did not.
  • Usage: Build a delivery success ratio from success="true" over the total. Codes such as unregistered are normal device-token churn; authentication errors are not.

centrifugo_push_scheduled_request_count​

  • Type: Counter
  • Labels: recipient_type
  • Description: Count of push notification requests accepted for later execution.
  • Usage: Shows how much of your push volume is scheduled rather than immediate.

centrifugo_push_queue_consuming_lag​

  • Type: Gauge
  • Labels: provider, queue
  • Description: Number of push jobs waiting to be consumed — pending entries of the Redis stream for the consumer group, or rows in the push_jobs table whose run_at is already due. Despite the name this is a job count, not a duration, and it is a Gauge — take its current value, do not wrap it in rate().
  • Usage: Sustained growth means push workers cannot keep up with the send rate.

centrifugo_push_consuming_inflight_jobs​

  • Type: Gauge
  • Labels: provider, queue
  • Description: Number of push jobs currently being processed.
  • Usage: Shows worker utilization per queue.

centrifugo_push_job_duration_seconds​

Deprecated

This Summary is deprecated and will be removed in Centrifugo v7. Use centrifugo_push_job_duration_seconds_histogram. Not exposed when prometheus.native_histograms is enabled.

  • Type: Summary
  • Labels: provider, recipient_type
  • Description: Duration of a push processing job.

centrifugo_push_job_duration_seconds_histogram​

  • Type: Histogram. Uses native (sparse, exponential) schema when prometheus.native_histograms is enabled.
  • Labels: provider, recipient_type
  • Description: Same data as the Summary above, in histogram_quantile()- and OpenTelemetry-friendly form.
  • Usage: Prefer this metric for percentile queries. Its _count also gives you the push job rate.

Analytics (ClickHouse)​

centrifugo_clickhouse_analytics_flush_duration_seconds​

Deprecated

Deprecated Summary — use centrifugo_clickhouse_analytics_flush_duration_seconds_histogram. Not exposed when native histograms are enabled.

  • Type: Summary
  • Labels: type, retries, result
  • Description: Duration of a ClickHouse data flush.

centrifugo_clickhouse_analytics_flush_duration_seconds_histogram​

  • Type: Histogram. Uses native schema when native histograms are enabled.
  • Labels: type, retries, result
  • Description: Time to write one batch to ClickHouse, by data type, retry count and outcome.
  • Usage: Rising flush latency is the leading indicator of analytics drops — the buffer fills while writes are slow.

centrifugo_clickhouse_analytics_batch_size​

Deprecated

Deprecated Summary — use centrifugo_clickhouse_analytics_batch_size_histogram. Not exposed when native histograms are enabled.

  • Type: Summary
  • Labels: type
  • Description: ClickHouse flush batch size distribution.

centrifugo_clickhouse_analytics_batch_size_histogram​

  • Type: Histogram. Uses native schema when native histograms are enabled.
  • Labels: type
  • Description: Rows per flush, by data type.
  • Usage: Batches pinned at the configured maximum mean Centrifugo is flushing as fast as it can — the next thing to give is the buffer.

centrifugo_clickhouse_analytics_drop_count​

  • Type: Counter
  • Labels: type
  • Description: Number of analytics events discarded because the buffer was full.
  • Usage: Direct analytics data loss. Alert on any sustained non-zero rate.

PostgreSQL connection pool​

These metrics describe the pgx connection pool Centrifugo PRO uses for PostgreSQL-backed features. The pool label separates pools when more than one is configured.

centrifugo_database_pool_max_conns​

  • Type: Gauge
  • Labels: pool
  • Description: Maximum size of the PostgreSQL connection pool.

centrifugo_database_pool_total_conns​

  • Type: Gauge
  • Labels: pool
  • Description: Current number of connections in the pool (idle + acquired + constructing).

centrifugo_database_pool_idle_conns​

  • Type: Gauge
  • Labels: pool
  • Description: Current number of idle connections.

centrifugo_database_pool_acquired_conns​

  • Type: Gauge
  • Labels: pool
  • Description: Connections currently checked out of the pool.
  • Usage: Riding at pool_max_conns means the pool is saturated — the queries below start queueing.

centrifugo_database_pool_acquires_total​

  • Type: Counter
  • Labels: pool
  • Description: Total successful connection acquires.

centrifugo_database_pool_empty_acquires_total​

  • Type: Counter
  • Labels: pool
  • Description: Acquires that had to wait for a connection to become free.
  • Usage: The clearest saturation signal for the pool — a growing rate here means raising max_conns (or reducing query time) will help.

centrifugo_database_pool_canceled_acquires_total​

  • Type: Counter
  • Labels: pool
  • Description: Acquires abandoned by context cancellation while waiting.
  • Usage: These are requests that gave up — usually visible to clients as errors or timeouts.

centrifugo_database_pool_acquire_wait_seconds_total​

  • Type: Counter
  • Labels: pool
  • Description: Total time spent waiting on connection acquires.
  • Usage: Divide its rate by the rate of pool_acquires_total to get the mean acquire wait time.

Bus, rate limiting and channel state​

centrifugo_bus_messages_processed_total​

  • Type: Counter
  • Labels: name
  • Description: Total number of messages processed by a PRO bus consumer.

centrifugo_bus_errors_total​

  • Type: Counter
  • Labels: name
  • Description: Total number of errors while processing bus messages.

centrifugo_rate_limit_hits_over_limit​

  • Type: Counter
  • Description: Number of requests rejected by PRO rate limiting.
  • Usage: Expect a non-zero baseline when limits are tuned tightly; alert on step changes rather than on any non-zero value.

centrifugo_channel_state_events_queue_consuming_lag_milliseconds​

  • Type: Gauge
  • Labels: name, partition
  • Description: Consuming lag of the channel state events queue, per partition, in milliseconds.
  • Usage: A single lagging partition usually points at an unbalanced key distribution rather than at overall throughput.

Shared poll relay​

Metrics of the PRO shared poll relay, which offloads polling to a dedicated relay instead of polling per node.

centrifugo_shared_poll_relay_poll_cycles_total​

  • Type: Counter
  • Labels: namespace
  • Description: Total number of completed relay poll cycles.

centrifugo_shared_poll_relay_backend_requests_total​

  • Type: Counter
  • Labels: proxy_name
  • Description: Total number of backend refresh requests issued by the relay.

centrifugo_shared_poll_relay_backend_errors_total​

  • Type: Counter
  • Labels: proxy_name
  • Description: Total number of failed backend refresh calls.

centrifugo_shared_poll_relay_backend_duration_seconds​

Deprecated

Deprecated Summary — use centrifugo_shared_poll_relay_backend_duration_seconds_histogram. Not exposed when native histograms are enabled.

  • Type: Summary
  • Labels: proxy_name
  • Description: Duration of backend refresh calls.

centrifugo_shared_poll_relay_backend_duration_seconds_histogram​

  • Type: Histogram. Uses native schema when native histograms are enabled.
  • Labels: proxy_name
  • Description: Backend refresh latency in histogram_quantile()-friendly form.

centrifugo_shared_poll_relay_cycle_duration_seconds​

  • Type: Histogram. Uses native schema when native histograms are enabled.
  • Labels: namespace
  • Description: Full poll cycle wall time.

centrifugo_shared_poll_relay_cycle_work_duration_seconds​

  • Type: Histogram. Uses native schema when native histograms are enabled.
  • Labels: namespace
  • Description: Poll cycle work time, minus the spread delay.
  • Usage: Compare against your configured poll interval — approaching it means the relay cannot keep up.

centrifugo_shared_poll_relay_sem_wait_duration_seconds​

  • Type: Histogram. Uses native schema when native histograms are enabled.
  • Labels: proxy_name
  • Description: Semaphore wait duration — the contention indicator for relay concurrency limits.

centrifugo_shared_poll_relay_active_channels​

  • Type: Gauge
  • Description: Number of channels currently polled by the relay.

centrifugo_shared_poll_relay_tracked_keys​

  • Type: Gauge
  • Labels: namespace
  • Description: Number of keys tracked by the relay, per namespace.

centrifugo_shared_poll_relay_items_count​

  • Type: Counter
  • Labels: namespace, result
  • Description: Items returned per cycle, by result — changed, unchanged or removed.
  • Usage: A dominant unchanged share means the poll interval is shorter than it needs to be.

Sentry integration​

Centrifugo PRO comes with an integration with Sentry. Just a couple of lines in the configuration:

{
  ...
  "sentry": {
    "enabled": true,
    "dsn": "your-project-public-dsn"
  }
}

– and you will see Centrifugo PRO errors collected by your self-hosted or cloud Sentry installation.

Sentry options​

sentry.enabled​

Boolean flag to enable Sentry integration.

sentry.dsn​

Sentry DSN to use for error reporting.

sentry.environment​

Environment name to set for Sentry events.

sentry.sample_rate​

Sample rate to set for Sentry events. By default, all events are sent to Sentry. You can set a sample rate to send only a fraction of events to Sentry. For example, to send 1/10 of events set this to 0.1.