Telemetry

Every GPU on every cluster reports once a second. The feed is the same one that drives the public site, the console, the capacity reports and the API. There is no second, rounder number kept for the brochure.

Preview

During preview the public site shows simulated telemetry, generated from a daily seed so the 24 hour chart is stable across reloads. It is labelled preview wherever it appears. Once C1 access opens, the feed switches to the agent described below without any change to the schema.

What is sampled

Per GPU, once a second: utilization in percent, memory used and memory total in GB, temperature in C, power draw in W. Per cluster, once a second: average utilization, number of running jobs, and 30 day uptime in percent. Job start and job end events are emitted as they happen.

Schema

Each message on the feed is one JSON object of one of three types.

{ "type": "sample", "data": {
  "cluster": "C1", "region": "us-west", "ts": 1757894400000,
  "utilAvg": 87.3, "uptime": 99.94, "jobsRunning": 4,
  "gpus": [
    { "id": "gpu0", "util": 91.0, "memUsed": 63.2, "memTotal": 80, "tempC": 66.5, "powerW": 588 }
  ]
} }

{ "type": "job_start", "jobId": "job_01j8x2", "gpuId": "gpu3", "ts": 1757894401000 }

{ "type": "job_end", "jobId": "job_01j8x2", "gpuId": "gpu3", "ts": 1757901600000 }

Timestamps are Unix milliseconds. Utilization is the SM utilization reported by the driver.

The agent

A small Node process runs on each cluster host. Once a second it calls nvidia-smi --query-gpu with a fixed field list, parses the CSV, builds a sample and pushes it over a websocket to the telemetry service. If the websocket drops, the agent buffers up to 5 minutes of samples and reconnects with exponential backoff. The agent does not read anything from inside containers.

Reading it

GET /v1/clusters/c1/telemetry?window=5m returns samples for the last 5 minutes. Windows up to 24 hours return one sample per minute, aggregated as the mean. Windows over 24 hours are served from the capacity report store at one sample per 5 minutes.

For streaming, connect to wss://telemetry.racksystems.cloud/c1 with your API key as a bearer header. You receive the same messages the site receives, without aggregation.

Retention

Per second samples are kept for 7 days. Per minute aggregates are kept for 13 months. Job events are kept for as long as the job record, which is 90 days in the console and 13 months in the capacity report store.

Your jobs in the feed

The public feed shows utilization per GPU but does not label which team runs on it. Inside the console and the API, samples for GPUs your job holds carry your job id. Other teams' job ids are not visible to you.