OTLP metrics

Added in v0.6.0-ewhauser.2. For traces, logs, and general telemetry configuration, see the upstream telemetry guide.

Enable metrics

export CELLD_OTEL=http://collector:4318
export OTEL_RESOURCE_ATTRIBUTES=celld.fleet=development
export OTEL_METRIC_EXPORT_INTERVAL=60000

Start celld with these environment variables and your normal fleet arguments. Metrics are enabled by default with an OTLP collector; set OTEL_METRICS_EXPORTER=none to disable them while retaining traces and logs. The collector base URL supplies the endpoint; celld appends /v1/metrics.

Metrics

celld exports metrics only to an OTLP collector (CELLD_OTEL=<url>). The bucket sink carries traces and logs, but not metrics. celld exports one set of metrics per interval (OTEL_METRIC_EXPORT_INTERVAL, 60 seconds by default).

No data point carries a cell, script, or request attribute. Each series belongs to a node, so the number of series grows with the number of nodes and not with the number of cells. Use OTEL_RESOURCE_ATTRIBUTES to name the fleet, so a dashboard can group its nodes.

Node load

These metrics are gauges. celld reads them from the same snapshot that /state serves, so a chart and /state agree. celld omits a gauge whose /state field is absent or null, because an unknown value is not a zero.

metric	/state source
celld.node.owned_cells	owned_cells
celld.node.resident_cells	node_load.resident_cells
celld.node.occupied	occupied
celld.node.restoring	restoring
celld.node.capacity_waiting	capacity_waiting
celld.node.activation_waiting	activation_waiting
celld.node.host_websockets	node_load.host_websockets
celld.node.rss_bytes	node_load.rss_bytes
celld.node.in_use_bytes	node_load.in_use_bytes
celld.node.cgroup_working_set_bytes	node_load.cgroup_working_set_bytes
celld.node.cpu_utilization	node_load.cpu_percent_x100 / 10000, a ratio of one core
celld.node.open_fds, celld.node.fd_limit	node_load
celld.node.pressured, celld.node.memory_headroom, celld.node.draining	node_load, as 0 or 1
celld.node.shed_cells	node_load.shed_cells
celld.node.isolates, celld.node.isolate_cells	the live and cells counts of the cell pools in deployment.isolates.cells, summed with the cell pools of the generations the node still drains

Cell distributions

These metrics are delta exponential histograms. Each point carries its min, max, sum, and count. A backend can merge the points of all the nodes into fleet-wide percentiles, with no bucket bounds to choose. Datadog maps them to distributions. celld omits a histogram when an interval has no observations.

celld.cell.cpu_time (seconds) has one observation per cell that ran JavaScript in the interval. The observation is the thread CPU time of that cell’s turns. It is exact, and a facet’s CPU counts toward its root cell. It is a distribution over active cells, not over requests or turns. When metrics are on, celld reads the thread CPU clock at each end of every cell turn. It does this even when no CPU limit is set. Stateless Worker requests are not in this metric.

celld.cell.heap_bytes (bytes) has one observation per resident cell. The observation is its isolate’s V8 heap, physical plus external, divided by the number of cells that share that isolate. Up to 32 cells can share an isolate, and V8 cannot cheaply attribute memory to one of them, so this value is an average:

• It is exact for an isolate that holds one cell. When a heavy cell shares an isolate with light cells, the metric understates the maximum.
• It counts only the V8 heap. It does not count memory on the Rust side, such as storage buffers and sockets.

The heap sample never waits for an isolate that is running a turn. celld skips that isolate and counts it in celld.cell.heap_samples_skipped.

Delivery

Metrics use the retry and backoff rules of the other OTLP signals. The exporter retries one export and holds one more export in a queue. celld drops an export that arrives while both places are taken. It counts the drop in celld.metrics.shed, and it reports that count in the next export that goes out.