Scheduler

The scheduler decides which job runs on which GPU and when. It has one queue per cluster and three plans. This page explains the placement rules so you can predict what happens to a job.

Plans and priority

Reserved jobs come first. They run on GPUs your team holds and start within seconds of submission. Nobody else can be placed on those GPUs while the reservation is active.

On demand jobs come second. They take free GPUs that are not reserved. On C1 that is whatever is left after reservations. If no GPU is free, the job waits in the queue in submission order. Expected wait is shown on the capacity endpoint as a per plan estimate.

Batch jobs come last. They run on any GPU that is idle right now, reserved or not. When the owner of a reserved GPU submits a job, the batch job on that GPU gets SIGTERM and 60 seconds to checkpoint, then SIGKILL. Batch jobs also yield to on demand jobs.

Placement

A job that asks for N GPUs gets N GPUs on the same NVLink fabric. On C1 that is always the same node. The scheduler prefers GPUs that are adjacent on the fabric, so a 2 GPU job gets a pair on the same switch when one is free.

The scheduler does not split a job across time. If 4 GPUs are not free at once, a 4 GPU on demand job waits until they are, even if 3 are free now. Batch jobs are the same. Nothing starts partially.

Queue order

Within a plan, jobs are ordered by submission time. There is no priority field, no bidding and no per team weighting. A team that submits 50 batch jobs at once will run them in order, and other teams' batch jobs queue behind them. If that becomes a problem we will add a per team fair share. It has not been a problem on one rack.

Preemption

Only batch jobs are preempted. The notice is 60 seconds, delivered as SIGTERM to PID 1 in the container. The job's status becomes preempted. With retry_on_preempt: true it requeues at the front of the batch queue and restarts from the image, not from memory. Write checkpoints to /shared.

On demand and reserved jobs are never preempted. They end when the command exits or max_minutes is reached.

Maintenance

Host maintenance drains the cluster. New jobs are not placed, running reserved and on demand jobs are allowed to finish up to their max_minutes, batch jobs are preempted. Windows are announced 7 days ahead and listed in the status history.

Reading the queue

GET /v1/capacity returns free GPUs per plan and the estimated wait in minutes for a 1 GPU job of each plan. GET /v1/jobs/{id} returns the position in queue for a queued job. Both are described in the API reference.