Server observability
To provide a better server observability Centrifugo supports reporting metrics in Prometheus format and can automatically export metrics to Graphite.
Metrics
Prometheus metrics
To enable Prometheus endpoint start Centrifugo with prometheus
option on:
{
...
"prometheus": true
}
This will enable /metrics
endpoint so the Centrifugo instance can be monitored by your Prometheus server.
Graphite metrics
To enable automatic export to Graphite (via TCP):
{
...
"graphite": true,
"graphite_host": "localhost",
"graphite_port": 2003
}
By default, stats will be aggregated over 10 seconds intervals inside Centrifugo and then pushed to Graphite over TCP connection.
If you need to change this aggregation interval use the graphite_interval
option (in seconds, default 10
).
Grafana dashboard
Check out Centrifugo official Grafana dashboard for Prometheus storage. You can import that dashboard to your Grafana, point to Prometheus storage – and enjoy visualized metrics.
Traces
OpenTelemetry
At this point Centrifugo can export traces for HTTP and GRPC server API requests in OpenTelemetry format.
To enable:
{
...
"opentelemetry": true,
"opentelemetry_api": true
}
OpenTelemetry must be explicitly turned on to avoid tracing overhead when it's not needed.
To configure OpenTelemetry export behaviour we are relying on OpenTelemetry environment vars supporting only HTTP export endpoints for now.
So a simple example to run Centrifugo with server API tracing would be running Jaeger with COLLECTOR_OTLP_ENABLED
:
docker run --rm -it --name jaeger \
-e COLLECTOR_OTLP_ENABLED=true \
-p 16686:16686 \
-p 4318:4318 \
jaegertracing/all-in-one:latest
Then start Centrifugo:
OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4318" CENTRIFUGO_OPENTELEMETRY=1 CENTRIFUGO_OPENTELEMETRY_API=1 ./centrifugo
Send some API requests - and open http://localhost:16686 to see traces in Jaeger UI.
By default, Centrifugo exports traces in http/protobuf
format. If you want to use GRPC exporter then it's possible to turn it on by setting environment variable OTEL_EXPORTER_OTLP_PROTOCOL
to grpc
(GRPC exporter format supported since Centrifugo v5.0.3).
Logs
Logging may be configured using log_level
option. It may have the following values:
none
trace
debug
info
(default)warn
error
We generally do not recommend anything below info to be used in production.
By default Centrifugo logs to STDOUT. Usually this is what you need when running servers on modern infrastructures. Logging into file may be configured using log_file
option.